Not sure about AH (IMHO using ah-sha-hmac with ESP adds almost nothing to security). If the transform set is esp-aes esp-sha-hmac then the MAXIMUM overhead is calculated as follows:
ESP constant: SPI(4) + SN(4) + PADLength(1) + NextHeader(1) = 10
ESP-AuthData: always truncated to 12 Bytes
AES-CBC (RFC 3602): IV(16) + MaxPadding(15)
Overall MAXIMUM ESP(AES) overhead = 10 + 12 + 31 = 53. Obviously the padding cannot be odd, so use 52 as the MAXIMUM overhead.
(Also, so far as my understanding goes, the Padding for MTU=1500 should never be more than 2, but still use 30 for safety reasons...)
P-to-P GRE overhead is 4, P-to-MP GRE overhead is 8.
ESP in tunnel mode adds IP header which is usually 20 Bytes. Transport mode doesn't add new IP header.
To sum up: "ip mtu" on the point-to-point GRE interface with esp-aes esp-sha-hmac and IPSec tunnel mode should be set to 1500-20(IP-added-by-IPSec)-52(ESP)-4(p-to-p GRE)-20(IP-added-by-GRE)=1500-96=1404.
For transport mode "ip mtu" = 1424.
Using this figures you can calculate the "ip tcp adjust-mss" by yourself ;)
