cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
33212
Views
5
Helpful
25
Replies

Need help with MTU issue (GRE + IPSec VPN tunnel)

Alen Danielyan
Level 1
Level 1

Hi,

I have some questions about the best strategy for MSS / MTU definition and PMTUD activation.

What I read:

Resolve IP Fragmentation, MTU, MSS, PMTUD Issues with GRE and IPSEC.pdf (I read this one twice )

MTU, TCP MSS, PMTUD MSS Adjust.doc

MTU Tuning for L2TP.pdf

Adjusting IP MTU, TCP MSS, and PMTUD on Windows Systems.pdf

Path MTU Discovery in Windows.mht

Step-by-step instructions for tuning TCP under Windows XP.mht

The idea is clear for me, the first source is a full ABC for the issue, but I still have some questions.

1. When using GRE+IPSec tunnels in what order are the following procedures done: fragmentation / encapsulation / encryption?

2. If I have DMVPN (GRE + IPSec) tunnels with my branches and I also have ACL (for incoming traffic) on the head office router WAN interface, which permits isakmp, esp and icmp traffic from the branch router only, and I enable PMTUD together with ip unreachables on that interface, what should I add in the ACL for PMTUD function to work normally? I mean should I add a filter to permit ICMP packets from the next hop? Or should I permit ICMP from any hop?

3. As I understand in case of GRE + IPSec (obviously transport mode) which works over ADSL line we have MSS = 1500 - 24 (GRE) - 38 (IPSec transport mode) - 20 (IP) - 20 (TCP) - 8 (for DSL) = 1390. Is this correct?

4. How to check the real max MTU between two Cisco routers connected via VPN (GRE + IPSec)?

I know I can use "ping -f" but, AFAIK, GRE will encapsulate the original packet and coolly fragment it, so I will not see the real max MTU (according to my calcs it would be ~ 1438.)

And my main question:

I have DM-VPN (as I understand it means GRE + IPSec in transport mode?) with my local regional branches.

My target is:

- to provide the best performance of the connections between hosts in the head office and each branch

- do not touch hosts settings

- provide invisibility for my routers in public networks.

(AFAIK, the latter means ACLs on our routers WAN interfaces for incoming traffic denying everything except traffic from our another respective router WAN interfaces plus "no ip unreachables". Am I right?)

Now what we have by default:

- in the offices we have Ethernet networks and Windows based PCs, which have PMTUD enabled, MTU = 1500, ICMP unreachables enabled. => All is ready for PMTUD (possibly except personal firewalls, which could block ICMP packets from a router and thus prevent PMTUD from functioning properly)

- on the offices Cisco routers: PMTUD is disabled by default, MTU on physical interfaces = 1500, on GRE tunnel = 1476, ip unreachables on WAN interfaces we are turning off (to provide invisibility).

Now, what is the best strategy?

My version:

for our routers LAN interfaces:

- set "ip tcp adjust-mss 1390"

- enable PMTUD and make sure "ip unreachables" option is activated (AFAIK, it is by default)

- leave MTU by default

for WAN interfaces:

- do nothing (which means PMTUD is disabled, MSS and MTU by default)

for tunnel interfaces:

- do nothing.

Please help me to clarify the question, because I can not handle the information I read, I am feeling giddy...

P.S. Why I am thinking of PMTUD together with MSS adjustment? - Because MSS works for TCP traffic only...

25 Replies 25

Hi Mohamed, thank you for your answers.

msobier123 wrote:

Hello Alen;

you wrote:

((( Please anyone, read my last post and answer the question. I just can  not calm down till get an answer.

Answer,

Fragmentation is not desired and recommended , fragmentation could lead to performance degradation on a network , it also not required for certain security concern. Fragmentation cause high load on routers as well as if a detination doesnt recieve all the fragmented packet (fragmented packet get lost on the middle), the packet needs to be resended again.

Now, coming to your previous question, the best common methods as follows:

1- The Tunnel interface should be at least set to 1400 , the attribute is not arbitary, its devided as follows:

     -- GRE Header  = 4 Byte

     -- IP Header      =  20 Byte

     -- TCP Header   =  20 Byte

     -- IPsec Header = 56 Byte

Total is 100 Byte substracting it from 1500 , as such the tunnel should be at least set with 1400.

2- The TCP maximum segment size MSS should be adjusted to 40 byte lower than than the 1400 byte of the GRE tunnel and should be set on the LAN segment.

Thank you, but I read the theory many times and everything is clear (except your minusing TCP header, I don't understand why should we minus TCP  header when calculating MTU?).

The main thing: your settings seem quite reasonable. Thank you.

Do you think activation (in addition to your above mentioned 2 settings ) of PMTUD on both tunnel and LAN interfaces can make things worse?

IMHO, it should not. It will just allow in the future to even decrease the MTU if needed and if PMTUD works correctly, but will not make any problems for current settings even if PMTUD doesn't work correctly. Am I right?

The question is, if I have already activated both: globally configured PMTUD and tunnel PMTUD, should I disable them?

msobier123 wrote:

Coming to your 1st question:

1- The packet will first be encrypted and then Encapsulated with a GRE and then fragmented if it need fragmentation, so the order is  Encryption > Encapsulation  > Fragmentation.


Clear, thank you.

(But in Cisco's Resolve IP Fragmentation, MTU, MSS, PMTUD Issues with GRE and  IPSEC.pdf it was an example (scenario 10) where the order was fragmentation / encapsulation /  encryption!? Why?)

msobier123 wrote:

2- You dont need this command, if the ICMP packet is too big, icmp packet will be fragmented if the Dont fragment bit is not set on the router. if the router for any reason has DF bit set, an icmp (destination host unreachable) message will be sent to the original host dictating packet needs to be fragmented but DF is set.

Well, this is clear, but first an intermediary router should send an ICMP packet to my router (as it is the starting point of tunnel and new, incapsulated packet source is his external interface ip). And my router will not "receive" it as it will be blocked by the ACL on ext. interface: block all, except ISAKMP, ESP and ICMP with source ip of another tunnel end router's ext. interface ip.

So I need to add the entry, if I want unreachables about "fragmentation needed" from intermediary routers to be passed and processed. Don't I?

msobier123 wrote:

3- Yes, it will do the same function and will set the DF bit on the icmp.

Thank you. This was very important question, as I was not sure I can get real MTU when testing through the IPSec encrypted GRE tunnel.

No, I know I can find max path MTU just by pinging with -f option from one LAN host - another LAN host!

msobier123 wrote:

4- you can either have IPsec tunnel profile or IPsec Crypto map based on your IPsec configuration.

I see this on my spokes:

crypto isakmp policy 1

encr aes 256

authentication pre-share

group 2

crypto isakmp key 6 xxx address 0.0.0.0 0.0.0.0

crypto isakmp invalid-spi-recovery

crypto isakmp keepalive 10 periodic

!

!

crypto ipsec transform-set dmvpnset esp-aes 256 esp-sha-hmac

!

crypto ipsec profile dmvpnprof

set transform-set dmvpnset

And I don't see anything that contains "ipsec tunnel" phrase.

=> As I understand I have IPSec in transport mode, which is "better" in case we are making encapsulation using GRE. Am I correct?

And MTU should be: 1500 - 24 (GRE) - 38 (max for IPSec transport mode) - 8  (for DSL) = 1430?

And this is in case line providers are not using any other encapsulations?

P.S. I'll rate your post later, I am not still satisfied and don't want the question to be marked as answered.

Hi Alen,

Please se my answers in blue:

Thank you, but I read the theory many times and everything is clear (except your minusing TCP header, I don't understand why should we minus TCP  header when calculating MTU?).


TCP based applications will add 20 byte of header to the original Packet size, thats why its subtracted from the total MTU.

The main thing: your settings seem quite reasonable. Thank you.

Do you think activation (in addition to your above mentioned 2 settings ) of PMTUD on both tunnel and LAN interfaces can make things worse?

I personally suggest seeting the MTU manually, I have experienced before for one client using PMTUD wich only decreases the IP header + GRE header on the tunnel. Here you need to decrease TCP header and IPsec header values.

IMHO, it should not. It will just allow in the future to even decrease the MTU if needed and if PMTUD works correctly, but will not make any problems for current settings even if PMTUD doesn't work correctly. Am I right?

Yes , since you have manually adjusted the MTU, you could have PMTUD enabled and it would not have affect unless the adjusted MTU removed.

The question is, if I have already activated both: globally configured PMTUD and tunnel PMTUD, should I disable them?

See above

(But in Cisco's Resolve IP Fragmentation, MTU, MSS, PMTUD Issues with GRE and  IPSEC.pdf it was an example (scenario 10) where the order was fragmentation / encapsulation /  encryption!? Why?)

Alen, I was refering to GRE in tunnel Mode, GRE Over IPsec !! Fragmentation will not happen if the packet is not encapsulated and the Encapsulation takes place after Encryption in GRE Over IPsec (Tunnel Mode). I am not sure about your document, try to resend it so we can have alook at it.

Well, this is clear, but first an intermediary router should send an ICMP packet to my router (as it is the starting point of tunnel and new, incapsulated packet source is his external interface ip). And my router will not "receive" it as it will be blocked by the ACL on ext. interface: block all, except ISAKMP, ESP and ICMP with source ip of another tunnel end router's ext. interface ip.

So I need to add the entry, if I want unreachables about "fragmentation needed" from intermediary routers to be passed and processed. Don't I?


Yes, if you have an ACL on ext, you need to permit it. and you need also to make sure (IP unreachable) at the interface level is enabled.

I see this on my spokes:

crypto isakmp policy 1

encr aes 256

authentication pre-share

group 2

crypto isakmp key 6 xxx address 0.0.0.0 0.0.0.0

crypto isakmp invalid-spi-recovery

crypto isakmp keepalive 10 periodic

!

!

crypto ipsec transform-set dmvpnset esp-aes 256 esp-sha-hmac

!

crypto ipsec profile dmvpnprof

set transform-set dmvpnset

And I don't see anything that contains "ipsec tunnel" phrase.

=> As I understand I have IPSec in transport mode, which is "better" in case we are making encapsulation using GRE. Am I correct?

And MTU should be: 1500 - 24 (GRE) - 38 (max for IPSec transport mode) - 8  (for DSL) = 1430?

And this is in case line providers are not using any other encapsulations?

You have two IPsec GRE modes:


1- Tunnel Mode: GRE over IPsec header ( GRE is not encrypted)

2- Tranport Mode: GRE inside IPsec header (GRE is encrypted).


From Security prespective, its better to have GRE inside IPsec which will also encrypt the GRE unlike in Tunnel Mode.


MTU should be as follows:


1- Dilaer Interface : 1492 ( subtracting 8 byte ppp header from the original 1500)

2- GRE Tunnel : 1392 (Subtracting 20 byte Ip header and 20 byte TCP header and 56 byte IPsec header , total is 100 leading 1492 - 100 = 1392)

3- TCP MSS: Maximum segment size should be set to 1352, because of the 40 byte IP and TCP headers. subtracting 1392 - 40 = 1352)

HTH

Mohamed

- TCP based applications will  add 20 byte of header to the original Packet size, thats why its  subtracted from the total MTU.

Wait a moment, we are talking about MTU, not MSS. MTU is either IP level - ip MTU, or L2\L1. When we calculate any of the mentioned MTUs we should not minus TCP header!

- Yes , since you have  manually adjusted the MTU, you could have PMTUD enabled and it would not  have affect unless the adjusted MTU removed.

Very well, clear.

- Alen, I was refering to GRE  in tunnel Mode, GRE Over IPsec !! Fragmentation will not happen if the  packet is not encapsulated and the Encapsulation takes place after  Encryption in GRE Over IPsec (Tunnel Mode). I am not sure about your  document, try to resend it so we can have alook at it.

I am also talking about GRE + IPSec. The source is this one (the most famous for MTU issue):

http://www.cisco.com/en/US/tech/tk827/tk369/technologies_white_paper09186a00800d6979.shtml

See scenario 10.

- Yes, if you have an ACL on  ext, you need to permit it. and you need also to make sure (IP  unreachable) at the interface level is enabled.

I know I'need to add an entry. My question was: "Do you think it is reasonable to add  “permit  icmp any any packet-too-big” in my ACLs for incoming traffic on  the  routers WAN interfaces"?

The matter is, by disabling ip unreachables (together with the above mentioned ACL) on ext. interfaces I make my routers invisible in public networks. As I understand “permit icmp any any packet-too-big” could make them visible. My question is: is it reasonable?

Would you do that?

- You  have two IPsec GRE modes:

1- Tunnel  Mode: GRE over IPsec header ( GRE is not encrypted)

2-  Tranport Mode: GRE inside IPsec header (GRE is encrypted).

From Security prespective,  its better to have GRE inside IPsec which will also encrypt the GRE  unlike in Tunnel Mode.

Sorry, but AFAIK, the main difference is in tunnel mode also initial source and destination addresses are replaced by new ones and are encaplsulated, not only IP data is.

So you can not say Transport mode is more secure. It could be said Tunnel mode is more secure, but when using GRE this is not the case, because GRE is making addresses removal itself and there is no need in double procedure.

Hope I could explain my point.

Addition:

"IPsec can be implemented in a host-to-host transport mode, as well as in a network tunnel mode.


Transport mode
In transport mode, only the payload (the data you transfer) of the IP packet is encrypted and/or authenticated. The routing is intact, since the IP header is neither modified nor encrypted; however, when the authentication header is used, the IP addresses cannot be translated, as this will invalidate the hash value. The transport and application layers are always secured by hash, so they cannot be modified in any way (for example by translating the port numbers). Transport mode is used for host-to-host communications.

A means to encapsulate IPsec messages for NAT traversal has been defined by RFC documents describing the NAT-T mechanism.

Tunnel mode

In tunnel mode, the entire packet (which need not be IP) is encrypted and/or authenticated. It is then encapsulated into a new IP packet with a new IP header. Tunnel mode is used to create Virtual Private Networks for network-to-network communications (e.g. between routers to link sites), host-to-network communications (e.g. remote user access), and host-to-host communications (e.g. private chat).

Tunnel mode supports NAT traversal."

MTU  should be as follows:

1- Dilaer  Interface : 1492 ( subtracting 8 byte ppp header from the original 1500)

2- GRE  Tunnel : 1392 (Subtracting 20 byte Ip header and 20 byte TCP header and  56 byte IPsec header , total is 100 leading 1492 - 100 = 1392)

3- TCP  MSS: Maximum segment size should be set to 1352, because of the 40 byte  IP and TCP headers. subtracting 1392 - 40 = 1352)

Again, I can not agree.

For path ip MTU it has to be 1500 (default ip MTU for Ethernet technology networks) - 24 (GRE) - 38 (max for IPSec transport mode) - 8  (for DSL) = 1430.

And MSS in that case = 1430 - 20 (TCP header) - 20 (IP header).

Thank you for the time you spent on this topic.!

Mohamed, please don't leave the topic.

P.S. A new issue: after I enabled PMTUD on my routers NTP failed to work! (My spoke routers synch from regional center routers via tunnels)

Yesterday I spent half of the day to solve the problem, until finally supposed it could be connected with PMTUD. Today I turned off PMTUD (globally configured one) and NTP is working again now.

So I'll keep it disabled.

rodmoore
Level 1
Level 1

What if someone were to take an opposite approach?  Lower the MTU on the crypto map interface (1300) and raise the mtu on the tunnel interface (1500). We will all agree that fragmentation will occur after GRE encapsulation and reassembly will occur on the other side the IPSEC tunnel.  So, some delay is introduced and more router CPU is used.  Is this a bad thing?  It can be if: you don't have CPU to spare or if the frag/reassembly delay affects the application (e.g. voice or video - but you can't adjust the MSS on these anyway as they use UDP).

What are the benefits?  This works for all traffic, not just TCP.  It's simple to configure, understand and maintain. The network becomes transparent to the end systems - no more wondering is this the application or the network when an app breaks.  It's the app.

I've read the same paper and why it doesn't explore this option, I have no idea.  Worse case, it sells bigger routers with more memory...

Mohamed Sobair
Level 7
Level 7

Alen,

The Global Path MTUD adjust tcp based and NTP uses UDP port 123, However, as I noted earlier, I strongly suggest you to manually set the IP MTU to 1400 on the tunnel interface. there are lots of application wont work when a packet gets fragmented or it will cause high delay as well.

HTH

Mohamed

While I still hope you will look at and comment my comments, I have new question:

As I remember even if I only set ip MTU on tunnel interfaces everything works fine. (I am talking about branch 2 head VPN connections)

Are there any reason to add ip MTU or MSS also on the routers LAN interfaces? What can it give else?

The negative impact would be all traffic (not only tunneled) going through the routers would have packets with small MTU. What else positive or negative we get?

P.S. About my NTP issue, here is the situation: I have branch routers LAN interfaces as ntp sources, LAN interface of the head office as ntp server and I had ip MTU 1400 on LAN interfaces on all of my routers. Then after I activated global PMTUD, I removed ip MTU 1400 from the head's router LAN interface (as Internet is also connected to the router and I did not want to reduce "Internet" packets).

And after some time (that is the main sad thing) ntp synch stopped working. After I finally understand what could be the reason and disabled PMTUD on the head router everything start working again.

But later I also disabled PMTUD on all branch routers either, just in case...

A new question: is sending "ip unreachable" (fragmentation needed but DF bit set) from the intermediary router whose MTU is less than the one of the recieved packet, a part of PMTUD mechanism or it is a standard behaviour regardless of PMTUD activation status?

Alen

The answer to the recent question is straightforward. The sending of the unreachable (fragmentation required but DF set) is a standard behavior. This is used by PMTUD, but its operation is quite independent of whether PMTUD is enabled or not.

(and in fact no one other than the local router knows whether PMTUD is enabled or not)

HTH

Rick

HTH

Rick

rburts wrote:

Alen

The answer to the recent question is straightforward. The sending of the unreachable (fragmentation required but DF set) is a standard behavior. This is used by PMTUD, but its operation is quite independent of whether PMTUD is enabled or not.

(and in fact no one other than the local router knows whether PMTUD is enabled or not)

HTH

Rick

Thank you, Richard.

Can you please also answer my last but one question?

As I remember even when I only set ip MTU on tunnel interfaces everything works fine. (I am talking about branch to head - hub to spoke VPN connections). Are there any reason to add ip MTU or MSS also on the routers LAN interfaces?

As I understand, the negative impact would be all traffic (not only tunneled) going through the routers would have packets with small MTU. What else positive or negative we get?

Thank you.

Richard, please answer.

Review Cisco Networking for a $25 gift card