Calculating MSS size for my PIX....

[abatson]

Recently discovered that the sysopt command has my MSS size set at 1380 bytes. This is done for VPN traffic, where after encapsulation, may be bigger than 1500 bytes & in need of fragmentation. If the DF bit is set, the traffic hits a brick wall. I have IPsec & GRE traffic going thru my PIX, so I need some help in learning exactly how large an MSS I can allow, before my VPN traffic dissapears? I know the sysopt command refers to the *data area* of the packet, so can someone quote me the header sizes for IPsec & GRE? I know IP headers are 40 bytes.

-Alex

[ehirsel]

Here is the info from pix 6.3 doc about the 1380 mss algorithm:

1380 data + 20 TCP + 20 IP + 24 AH + 24 ESP_CIPHER + 12 ESP_AUTH + 20 IP = 1500 bytes

From http://www.networksorcery.com/enp/protocol/gre.htm:

The gre headers can be 24 bytes max. (it depends upon the gre version).

The smallest ip header is 20 bytes, not 40. The tcp and udp headers have a min of 20 bytes.

If you do not use AH in any of your ipsec packets, you could account for gre in those 24 bytes, so the pix default mss of 1380 should work for you. If you do use AH then use an mss of 1380-24=1356.

Don't use AH if you perform NAT and/or PAT between the endpoints because the AH check will fail.

I hope this helps.