There is not limit as to when you need to get the data, just so that you get it all out before the next DRDY. At the next DRDY, the output register is updated with the new data.
Regarding your circuit:
There are some potential problems with the reference circuit for the device that may be causing some of the issues that you are seeing. It looks like the capacitance on the reference it a little high. Can't comment on specifics of what is can handle, but it seems like a lot.
There is also a lot of capacitance loading the reference buffer after R22 that may be causing some instability during operation. (You can reference the buffer circuit in Figure 39 of the datasheet for an example). You might try a quick experiment by removing the 2 large caps (C34, C35) and see if things change.
On the analog inputs, the filters that you have could also cause some problems. Generally, you don't want the common mode capacitors to be larger than the differential capacitor. Any mismatch in the common mode caps could translate to voltage error that would propagate through the circuit.