Hi Alexander,
The ADC is extremely sensitive to voltage variations on its reference input, which incidentally, is not a high impedance port when the ADC is converting. In fact, the load during conversion is characterized by high frequency current pulses and a large bypass capacitor is typically required to supply these instantaneous load fluctuations. So as a first step I would like to recommend installing a 1uF-10uF ceramic capacitor close to the VREF input of the ADC. Please let me know if this doesn't change your results.
Best Regards,
Harsha