I don't think this is a THS8200 configuration issue.
I don't know enough about the 3730 but can you check to see if it has an output clock inversion control? If yes, try inverting the output clock.
Using an oscilloscope check that the clock to data transitions meet the THS8200 input setup/hold requirements.
Check the integrity of the VGA connector output H & V syncs. Make sure that the voltage levels are correct and that the rise/fall times are correct to the VESA standard.
Check the line to line jitter on the h-sync. It needs to be less that 1/3 of a pixel period to be VESA compliant. If not then the output clock from the 3730 is not stable enough.
Due to the fact that analog VGA does not send a clock signal with the video the receiving monitor must re-construct a local version of the clock from the h-syncs. If the h-syncs are not rock solid and at the correct levels/slew rates then the receiver cannot correctly re-generate the sampling clock.
Finally, on your monitor make sure you run the monitor's "auto adjust" algorithm to make sure it is correctly sampling the input at the right phase.
BR,
Steve