Hello,
I think you are running into the noise floor of the LMP90080 by setting VREF so low. As VREF is reduced, the range of acceptable analog input voltages is also reduced. Reducing the value of VREF also reduces the size of the LSB. When the LSB size goes below the noise floor of the LMP90080, the noise will span an increasing number of codes and performance will degrade. For optimal performance, VREF should be the same as VA and sourced with a clean source that is bypassed with a ceramic capacitor value of 0.1 μF and a tantalum capacitor of 10 μF.
It is probably best to change to the LMP90100. Since you will then have 24 bits, you can set VREF to 5V. The part is pin-to-pin compatible with the LMP90080 so no change is needed to the PCB.
Mike