Hi Jiri,
Can I assume that you have inner layers to your board? Can you send me the inner layer plots too? Were you able to pass any levels of the IEC61000-4-4 (like threat level 1, 2 or 3)? For the test, did the cabling all connect to X12, or were other points to other connectors tested as well?
I would try to isolate the power problem first. You may want to add some TVS diodes to the input side of the supply (U20) and the digital isolator (U16). You may need some protection on your input signal lines too. Devices like the TPD4E001 also provide enhanced ESD protection.
Best regards,
Bob B