Sebastiaan,
I tried modifying an ADS7953 tester board here to setup the reference the way you have it on your board. As expected, the conversion results came out bad. However, I am not able to get the last nibble stuck at 0xF. I currently don’t have a clear experiment to explain that behaviour.
Nonetheless, the reference on your board needs to be fixed. For the ADS7953, the LSB = Vref/(2^12-1) = 2.5/4095 = 611uV. With a resistor divider to derive the reference, the actual voltage that the ADC uses for its comparisons would vary based on small shifts in the load current and impact the output codes. It is recommended to drive the REF pin with a low noise reference like the REF5025, as mentioned in page 42 on the datasheet.
Regards,
Sandeep