Ashwini,
You mentioned that you are using Fs/4 NCO mixer enabled. Could you clarify if you are using the fs/4 coarse mixer option? Or are you using the NCO programmed as fs/4 mixer.
I suspect the higher distortions is due to the NCO accumulator not initialized correctly.
You may refer to section 3.3 and 3.4 of the following app note to understand if you have the correct initialization sequence for the NCO block
http://www.ti.com/lit/an/slaa584/slaa584.pdf
If you can send me your register programming for the DAC, I can also review this for you.
Also, please let me know how you are terminating the DAC output. The max compliance voltage for each leg (Ioutp or Ioutn) is from -0.5V to +0.6V. Output voltage exceeding this spec may cause distortions.
-Kang