What format video are you sending to the decoder?
768 lines is only generated for PAL and SECAM video sources.
Note also that the TVP5146M2 does not support square pixel, only the TVP5146.
You also seem to be touching a lot of registers that I don't think are necessary to change.
Try setting the video standard register to PAL so that all other registers are set automatically.
If you do want to set all registers then only the following should need to be changed...
AVID start pixel = 67h (NB write MSByte after the LSByte) (may be set automatically when setting square pixel in the output formatter register)
AVID stop pixel = 2B8h (NB write MSByte after the LSByte) (may be set automatically when setting square pixel in the output formatter register)
Sync control = Your setting invert the H & V polarity. Are you sure you want to do this?
Output Formatter 1 = 0xC1 = square sampling, extended range, 20 bit 422 discrete syncs
Output Formatter 2 = 0x11 to enable the data and clock outputs
Output Formatter 3 = 0xEE, AVID & FID enabled.
Output Formatter 4 = 0xAF, H & V output enabled as per 0x32
Output Formatter 5 = not needed since not using 10 bit mode hence Cx pins not configurable.
Output Formatter 6 = not needed since not using 10 bit mode hence Cx pins not configurable.
Clear Lost Lock Detect. Only needed if you are checking the lost lock status bits
AGC White Peak Processing. Unless you are sure you want to change this to disable luma peak processing I would leave this as the default.
Note, when changing any timing parameters it is usually necessary to force the video mode before changing timing registers since the TVP remembers timing parameters on a per video standard basis. For example, if the decoder is receiving a NTSC signal and you change the AVID start/stop, then feed a PAL signal the TVP will restore the previously set PAL parameters (likely the PAL defaults in this example).
BR,
Steve