Hi Dhiren,
I think you have pretty much answered your own question. If the USB ground is affecting the measurement, then you have some sort of instability with noise relative to ground. Remember that "ground" is a reference point but is also a part of the system. Ground bounce, or some other distburance to ground can not only affect the measurement but the integrity of the power system as well. If you route "ground" with traces, you can pick up noise on the ground as well as you can on any other input. Sometimes the noise can come from the external supply routing, such as a wire from the supply to a connector on the PCB. So, before you can solve the problem you have to identify the source of the problem. I've given you a couple of ideas of places to look. Another thing to consider is what is happening when you connect the USB ground. Does it make an earth ground connection that is missing from your power source?
Best regards,
Bob B