Jin,
1. The transformer allows an easy interface between the DAC's differential current driver to most of the test equipment, which are mostly single-ended 50ohm load based. The balun before the transformer is meant to improve differential balancing of the output signal. We noticed that the even order harmonics improves by a couple dB at frequencies about 100MHz.
2. 4:1 transformer allows the higher output power of 4dBm, or 1Vpp into the 50ohm load. See datasheet description for output interface for detail.
3. You may use filter design tools available on the market such as Elsie http://tonnesoftware.com/elsie.html. The following app note talks about the filter design procedures:
http://www.ti.com/lit/an/slwa053b/slwa053b.pdf
You may either design differential filter between the differential DAC driver and the transformer. The source and load impedance will depend on the transformer type. You may also design the filter as single-ended filter after the transformer. The choice would depend mostly on customer preference. Attached TINA model for DAC + transformer can help you with Spice simulations for filter responses.
-Kang