The issue is that by default, all unused inputs are floating and will drift to 0 volts. Since the inputs are AC coupled, if a signal is still present when the input is turned off, then on the AIC3101 side of the input AC coupling cap which is now biased at 0 volts, the signal will go above and bellow ground which will violate the input range.
If a signal is going to be present when an input is not routed to the ADC, you must set a weak common mode voltage at the unused inputs. On Page 0, Register 20 and 23 are listed as reserved, however, if you send a 0x7C to both Register 20 and 23, you will see that the inputs will be biased to mid-rail even when not routed.
The only values you should send to Register 20 and 23 are:
0x78 = Unused inputs will be Hi-Z
0x7C= Unused inputs will be weakly driven to mid-rail