MTU on Tunnel Interfaces

GRANT3779 · ‎10-22-2015

Reading the following statement from an article I was reading -

"In the cases where IPsec is being used, it is customary to set the MTU size on the tunnel interfaces to 1400 bytes and to set the TCP-MSS-adjust to 1360 bytes"

I my understanding of this correct -

Standard MTU size for Ethernet -1500bytes before ethernet header applies.

1360 bytes set for MSS. Once TCP header and IP Header added, will take size of packet to 1400bytes. This is then sent to the tunnel Interface and will have any GRE / IPSEC headers added.

Why do we set the size of the Tunnel Interface to 1400? Is this to try minimize any fragmentation when the frame is sent out the physical Interface?

Masoud Pourshabanian · ‎10-22-2015

Backbone interfaces may drop packets with size of more than 1500 bytes. Tunnel Interfaces add header to the original packet depending on type of the tunnel. As an example, GRE tunnel add 24 bytes header to the packet. As a result, packets with the size of more than 1500 may be dropped while traveling through tunnel. When you set MTU on a Tunnel interface, packets will be split before going through tunnel to prevent from being dropped.

Th size of 1400 is very conservative size. Please check this web site to calculate the proper MTU size.

https://cway.cisco.com/tools/ipsec-overhead-calc/ipsec-overhead-calc.html

Masoud

GRANT3779 · ‎10-24-2015

Hello Masoud,

You mention the following...

When you set MTU on a Tunnel interface, packets will be split before going through tunnel to prevent from being dropped

If packets are less in size than the configured mtu on the tunnel interface then I assume they won't be split by the tunnel int?

Masoud Pourshabanian · ‎10-24-2015

Yes, that is correct. If MTU discovery or TCP MSS works, host will send packets smaller than MTU set on the Tunnel interface so router does not split them.

Masoud

Joseph W. Doherty · ‎10-22-2015

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Why do we set the size of the Tunnel Interface to 1400? Is this to try minimize any fragmentation when the frame is sent out the physical Interface?

Yes, exactly, to try to minimize any fragmentation.

The 1400 allows for all the typical GRE/IPSec overhead, which is often actually closer to about 80 bytes, but the 1400's 100 bytes helps preclude going over by a few bytes.

BTW, typical IP and TCP headers consume 40 bytes but either might have options which will throw off the usual TCP adjust setting (i.e. it will be incorrect).

Masoud Pourshabanian · ‎10-22-2015

I just need to add that it is set to minimize any fragmentation after leaving the tunnel, not to minimize end to end fragmentation. In the tunnel with MTU of 1400, packet will be fragmented if there is no other fragmentation prevention methods implemented

Suppose packet comes from a computer with the MTU size of 1500. Packet arrives at Tunnel interface with MTU of 1400 is set. Packet must be fragmented to be able to travel through tunnel without any further fragmentation.

MTU needs to be implemented by end hosts to minimize fragmentation. I mean sender( computer in this case) needs to decrease MTU. It can be done manually or by using "tcp path-mtu-discovery".

Overhead of tunnel also is different depending of types of encryption and tunnel. If GRE tunnel with transport IPSEC are implemented, only 24 bytes will be added on Tunnel interface.

Masoud

Joseph W. Doherty · ‎10-22-2015

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Just to add to Masoud's addition . . .

I just need to add that it is set to minimize any fragmentation after leaving the tunnel, not to minimize end to end fragmentation. In the tunnel with MTU of 1400, packet will be fragmented if there is no other fragmentation prevention methods implemented

Often a tunnel provides the smallest end-to-end MTU, but not always.

Without client PMTUD, it would seem there's no advantage to setting a tunnel's MTU smaller, but especially with encryption, there's an issue whether encryption is before or after fragmentation, the latter can cause additional fragmentation. A nice Cisco tech note is: http://www.cisco.com/c/en/us/support/docs/ip/generic-routing-encapsulation-gre/25885-pmtud-ipfrag.html

BTW, an example of another prevention method is the OP's TCP adjust MSS (which only, though, applies to TCP).

MTU needs to be implemented by end hosts to minimize fragmentation. I mean sender( computer in this case) needs to decrease MTU. It can be done manually or by using "tcp path-mtu-discovery".

Also BTW, many hosts (but not all) enable PMTUD by default. If not enabled, hosts will often set off-local network MTU to 576, also to avoid fragmentation.

Lastly, be aware some security devices will block ICMP fragmentation needed messages, which tends to cause problems.

Overhead of tunnel also is different depending of types of encryption and tunnel. If GRE tunnel with transparent IPSEC are implemented, only 24 bytes will be added on Tunnel interface.

It does, and a good example would be using tunnel vs. transport mode with a GRE/IPSec tunnel. Again, the Cisco recommended 1400 allows for worst case, but it can be suboptimal.

By "transparent" IPSec, did you mean just a GRE tunnel? If so, there indeed is only 24 bytes of overhead, and so if you did want to be more optimal, you could set MTU to 1500 less 24, MSS less 24.

Masoud Pourshabanian · ‎10-22-2015

Hello Joseph

I totally agree with your second post and I rated your post. I am trying to say just adding MTU on Interface tunnel does not necessarily minimize fragmentation. It depends on other factors as well.

By transparent, I meant transport. Sorry for confusion. It was misspelling. Size of header onTunnel interface will be different depending on which features are ebabled for encryption and authentication and tunneling.

Also MTU should be selected properly. Throughput will be lower If MTU is less than what the end-to-end links can carry without fragmentation.

Please take a look at link below. it is a nice header calculator for IPSEC along with tunnel.

https://cway.cisco.com/tools/ipsec-overhead-calc/ipsec-overhead-calc.html

Masoud

Joseph W. Doherty · ‎10-22-2015

Thanks for the tool reference - nice!

GRANT3779 · ‎10-23-2015

Hi Masoud / Joseph,

Thanks a lot for the input / info, appreciate the time you take to answer my questions.

Masoud - you mention here -

"Suppose packet comes from a computer with the MTU size of 1500. Packet arrives at Tunnel interface with MTU of 1400 is set. Packet must be fragmented to be able to travel through tunnel without any further fragmentation"

For TCP - Isn't my TCP mss adjust command used to try and prevent this in the first place for the conversation? I assume the statement above would be more for UDP traffic?

Joseph - Are you saying setting a tunnel IP MTU has no advantage if done by itself? (e.g with no tcp mss adjust set OR client PMTUD available?

Thanks

Joseph W. Doherty · ‎10-23-2015

Joseph - Are you saying setting a tunnel IP MTU has no advantage if done by itself? (e.g with no tcp mss adjust set OR client PMTUD available?

No, just the opposite; or that's what I intended.

Masoud Pourshabanian · ‎10-23-2015

Hello,

Let me explain 4 cases. Suppose you have a client and a server a tunnel in between.

1- Neither MTU nor TCP MSS is implemented on tunnel : Tunnel adds header and packets get larger which results to packets drop by intermediate router. PMTUD is enabled by default on all devices. When a router drops a packets due to the large size, it sends an ICMP message to the sender. The problem is servers are protected by firewalls and firewalls are usually implemented stateful so the ICMP initiated outside traveling inside gets drop by firewall and server does not receive it. Thus, probability of working of PMTUD is not high.

2- Only MTU is implemented on tunnel. Router fragments packets larger than MTU and sends them through tunnel. PMTUD may be initiated, but it may not work due to the previous case. Back to your question. MTU 1400 only minimizes further fragmentation through tunnel. Packets get fragmented before leaving the tunnel.

3-TCP MSS is only implemented. Host usually sends the size of MTU while handshaking(SYN packets) . When TCP MSS is configured on tunnel, router listens to SYN packets and adjusts TCP MSS option. In this way, other host receives a syn packet including TCP MSS adjusted by router so host reduces the size of packets. In this way tunnel does not receive larger packet so fragmentation is minimized. it only works with TCP packet, so large UDP packets may be dropped.

4- Both are implemented. TCP MSS tries to adjust proper MTU size and IP MTU fragments any larger packets to prevent from being drop.

There is no grantee TCP MSS works. Packets may be dropped if proper MTU on tunnel is not set. In some cases you may even end up adjusting MTU on hosts to make sure fragmentation does not happen.

Lakshman Kanth · ‎08-25-2020

Hi,

In my case my router config mentioned below. R1 physical interface MTU is lower than the tunnel MTU. Does it impact the network performance?

Router 1 tunnel interface MTU is 1418, MSS is 1340 and Physical interface MTU is 1300

Router 2 tunnel interface MTU is 1418, MSS is 1340 and Physical interface MTU is 1500

-Lakshman

Giuseppe Larosa · ‎08-25-2020

Hello @Lakshman Kanth ,

>> R1 physical interface MTU is lower than the tunnel MTU. Does it impact the network performance?

yes in the direction R1 to R2 there will be fragmentation performed by R1

Even worse in R2 to R1 direction as packets with size greater then 1300 bytes will be fragmented on the penultimate router hop to R1 as the link MTU is only 1300 bytes.

To minimize fragmentation both routers R1 and R2 should use an MTU less then 1300 bytes like 1200 bytes.

The link with the lowest MTU on the path is the bottleneck

Hope to help

Giuseppe

Joseph W. Doherty · ‎08-25-2020

"Even worse in R2 to R1 direction as packets with size greater then 1300 bytes will be fragmented on the penultimate router hop to R1 as the link MTU is only 1300 bytes."

Just to add a note, why this is even worse, it's because there can be two fragmentations if going from R2 to R1. R2 will fragment, and the "penultimate router" may fragment R2's first fragment (but not the second, as it doesn't exceed MTU). I.e. R1 could receive 3 fragments, while R2 would only receive 2 fragments.