PIX breaking fragmented packets over IPSec tunnel

[ltwomey]

I have a PIX 6.2(2) configured to establish an IPSec tunnel (3DES/SHA) to a Checkpoint firewall. Tunnel comes up, traffic flows properly most of the time. However, if a client on the PIX LAN generates a fragmented UDP packet, the PIX passes the first fragment over the VPN successfully but the second fragment appears at the far side as two separate (un-fragmented) UDP packets with "random" source and destination ports and much larger number of bytes than the original fragmented packet (on subsequent fragmented UDP packets the first fragment arrives at the far side but subsequent fragments never appear). Fragmented packets generated by the remote (non-PIX) site are handled correctly by the firewalls.

While testing this, I disabled the VPN, and the fragmented packets arrived successfully at the remote site, so this problem seems to be related to the encryption of the packets by the PIX. The PIX is not configured to drop fragmented packets. Dropping the VPN to use DES/MD5 had no effect.

Is anyone aware of any problems with handling fragmented packets in PIX v6.2(2), or is there a setting that I haven't managed to find that may affect this?

Thanks in advance,

Louis Twomey.

[afakhan]

Hi,

I couldn't find any known bug for this issue on PIX 6.2.2, where pix can't enrypt the udp fragments, and sends out in cleartext.

What abt lowering down MTU on those servers(or by PIX) sitting behind the PIX to avoid fragmentation as the work-around.

Thanks,

Afaq

[ltwomey]

Hi Afaq,

Thanks for the response. It turns out that the problem was the Checkpoint end of the VPN link. The PIX was successfully encrypting the packets, but the Checkpoint (NG FP1) was not decrypting the second (and subsequent) fragments correctly. Upgrading to Checkpoint NG FP2 solved the problem. This problem never arose on a Checkpoint NG FP1 to Checkpoint NG FP1 VPN link, which suggests that the Checkpoint does not like the fragmented packets as encrypted by the PIX - the fact that the problem does not occur in NG FP2 suggests that this was a bug in NG FP1 rather than a problem with the PIX.

Regards,

Louis.