Hello Dieter,
Thanks for your post. I was able to replicate your EVM results on my bench and I noticed a couple details worth explaining:
1. FFT: As you noticed, the input frequency of 51 kHz appears in the FFT as a mirrored signal power at 49 kHz. This is due to a phenomenon called "aliasing." In the frequency domain, the passband of the ADC is limited to 1/2 the sampling frequency (fS), according to Nyquist. In your case, you are sampling at 100 kSPS (100 kHz), so the output data will only show frequencies up to 50 kHz. Frequencies that are larger than the 1/2 fS can "fold" or alias into a lower frequency range.
The digital filter in the ADS1278 attenuates signals larger than 1/2 fS to prevent them from aliasing back into the passband of the ADC.
However, in this transition band from 0.45 to 0.55 (fIN normalized to fDATA), you see that the digital filter has not finished rolling off. At fIN = 51 kHz (fIN / fDATA = 0.51), it is reasonable to expect about -12 dB of attenuation, which you observed in your FFT.
2. VIN: The second concern is the amplitude of your input. I back-calculated your original signal amplitude to be about 4.2 VPP, is that correct? If so, this is too large for a VREF = 2.4 V. Remember that the usable input range is from - VREF to + VREF (Table 5 in the datasheet).
Calculation:
Full-Scale = VREF = 2.5 V; -1.5 dB = 20*log(VIN / 2.5 V) -> VIN = 2.1 VPP
After -12 dB of attenuation from the digital filter, I would expect VOUT = 0.5 VPP; however, your scope capture shows 1 VPP, so I presume you started with VIN = 4.2 VPP.
To summarize, fIN must be less than fDATA / 2 = 50 kHz and the amplitude of VIN must remain less than 2.5 VPP.
I hope this was helpful, please let me know if you have additional questions.
Regards,