MTU for site to site IPSEC tunnel (ESP tunnel mode no-nat-T)

laurin.beckhusen · ‎08-09-2016

Hello all,

Trying to understand some behavior and could use some help.

We have a Policy Based Site-to-Site tunnel between an ASA (active / standby) pair and a juniper srx 650. Tunnel is up and working.

However I noticed some strangeness when pinging across the tunnel with different packet sizes to create fragmentation.

First ping attempt.

C:\Users\user>ping -f -l 1400 ip.ip.ip.ip

Pinging ip.ip.ip.ip with 1400 bytes of data:
Reply from ip.ip.ip.ip: bytes=1400 time=6ms TTL=126
Reply from ip.ip.ip.ip: bytes=1400 time=7ms TTL=126
Reply from ip.ip.ip.ip: bytes=1400 time=6ms TTL=126
Reply from ip.ip.ip.ip: bytes=1400 time=6ms TTL=126

Ping statistics for ip.ip.ip.ip:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 6ms, Maximum = 7ms, Average = 6ms

C:\Users\user>ping -f -l 1402 ip.ip.ip.ip

Pinging ip.ip.ip.ip with 1402 bytes of data:
Reply from ip.ip.ip.ip: bytes=1402 time=10ms TTL=126
Reply from ip.ip.ip.ip: bytes=1402 time=7ms TTL=126
Reply from ip.ip.ip.ip: bytes=1402 time=7ms TTL=126
Reply from ip.ip.ip.ip: bytes=1402 time=8ms TTL=126

Ping statistics for ip.ip.ip.ip:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 7ms, Maximum = 10ms, Average = 8ms

Cool, so far so good, But i did another ping with the same size and it got fragmented:

C:\Users\laurin.beckhusen>ping -f -l 1400 ip.ip.ip.ip

Pinging ip.ip.ip.ip with 1400 bytes of data:
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.

Ping statistics for ip.ip.ip.ip:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

I waited 10 minutes or so and now they are going through again......!!

C:\Users\user>ping -f -l 1400 ip.ip.ip.ip

Pinging ip.ip.ip.ip with 1400 bytes of data:
Reply from ip.ip.ip.ip: bytes=1400 time=7ms TTL=126
Reply from ip.ip.ip.ip: bytes=1400 time=6ms TTL=126
Reply from ip.ip.ip.ip: bytes=1400 time=6ms TTL=126
Reply from ip.ip.ip.ip: bytes=1400 time=6ms TTL=126

I started looking at the config and noticed the MTU is 1500 for the inside interfaces and outside interfaces and there is no TCP_MSS and all interface MTU's are 1500. On the juniper side the interfaces are set to 1500.

So I am trying to understand why somtimes the larger packet goes through and sometimes it doesn't,,,

Based on my understanding 1398 of data from ping + 8 byte icmp header + 20 byte IP header = 1426. Looking at show crypto ipsec sa I see: path mtu 1500, ipsec overhead 74(44), media mtu 1500. So it looks good to me. 1426 Byte ip packet 74 Byte over head = 1500.

Reading a cisco article(http://www.cisco.com/c/en/us/support/docs/ip/generic-routing-encapsulation-gre/25885-pmtud-ipfrag.html) the most ipsec overhead could be is 58 (The added header(s) varies in length dependent on the IPsec configuration mode but they do not exceed ~58 bytes) BUT !!! Looking at the cisco calculator https://cway.cisco.com/tools/ipsec-overhead-calc/ it said the ipsec overhead is 64, so that got me thinking where did the other 6 Bytes go? Which one is correct?

Is there a command I could issue on my ASA to see if there is fragmentation? happening.

My concerns are making sure the end to end communication over the vpn tunnel is solid. Is there a sweet spot setting I could use? At my previous place they had TCP_MSS values of 1380 and they lowered the mtu, so that got me thinking I need to do that here and do some fine tuning...

Thanks for the help.

Laurin

Aditya Ganjoo · ‎08-09-2016

Hi Laurin,

It is pretty weird that this is happening intermittently.

However I would suggest you to reduce the MTU on the ASA to 1300 and test:

sysopt connection tcpmss 1300

Please check this link for more clarity:

http://www.cisco.com/c/en/us/td/docs/interfaces_modules/services_modules/vspa/configuration/guide/ivmsw_book/ivmvpnb.html

Regards,

Aditya

Please rate helpful posts and mark correct answers.

laurin.beckhusen · ‎08-09-2016

Hi Aditya,

Will that command only affect vpn traffic or all traffic?

-Laurin

Aditya Ganjoo · ‎08-09-2016

Hi Laurin,

This is a global setting and would impact all the traffic on the device.

Regards,

Aditya

Please rate helpful posts and mark correct answers.

laurin.beckhusen · ‎08-15-2016

Hi Aditya,

Thank you for the help. I haven't changed the MSS window or MTU as I want to gain some more understanding of what will happen. We host public services and internal users need access to services located through a site to site vpn tunnel, so I need to setup a time to test to see how it affects users if were to change the TCP window size.

BTW. I am confused how changing tcp window size would help with fragmented pings? Pings don't use tcp mss values?

I am thinking it will take a combination of MTU and tcp window size, but the changes would affect all traffic going through the device, ( we host remote access ssl vpn on the outside interface) so are there common strategies in setting up networks to handle this. Buy two ASA's and use on for vpn termination and the other for inbound hosted services... Sorry the question is turning into more.

Thanks for the help.

laurin.beckhusen · ‎08-09-2016

Update:

I started reading http://www.cisco.com/c/en/us/support/docs/security/asa-5500-x-series-next-generation-firewalls/82444-fragmentation.html and noticed the breakdown in the ipsec headers.

If we have nat-t disabled and it is a esp tunnel then am I correct in saying the over head is 56(esp) + 20(outside ip header tunnel mode). Which would then leave 1426 for the IP packet.

So it seems to me an icmp packet over 1398Bytes going over the tunnel with the DF bit set should fragment at 1398, but yet I can push it through at 1400....