This looks like an AM mis-configuration, possibly in the frame buffer driver/configuration.
This could be many things still, but I don't think the THS is the issue.
When you say "windowed mode is OK" do you mean that even if you move the window completely to the right hand edge into the region which is currently corrupt that it displays OK? Can you ever display something correctly in this right hand region? What I am trying to determine is where in the rendering path the fault is.
Can you display a mainly white display then capture a line of active video on an oscilloscope, showing h-sync and each or red, green and blue? One color will be OK if you can't capture all color channels.
Make sure that the horizontal front and back porches are correct for the resolution you are trying to display.
Brightness variations in analog video are most often caused by the active video encroaching in to the sync region.
Do you get exactly the same results on different monitors? Sometimes different behavior can help determine the root cause.
BR,
Steve