Hi Michel,
An asymmetrical reference voltage should not cause an offset problem because the input signal is compared to the relative voltage of the reference. A drift on the offset definitey will cause gain error drift as the reference voltage directly affects the ADC's full-scale code.
What kinds of components are connected to the ADC's analog inputs? Hoave to checked if the offset drift could be coming from an external source?
Regards,
Chris