WAN MTU change: impact on applications?

[mikep1230]

Has anyone made an in-production MTU change to their WAN?  If so, how have the applications been impacted?

Our DMVPN (enterprise site-to-site) MTU is currently set to 1400 bytes, and we're looking at dropping it by a few more.  I know that new connections established after the change will figure out the new MTU size using the usual MSS/PMTUD methods.  I'm being asked to estimate the impact on live communications though.  I know that some apps send max-MTU packets with the DF bit set.  Once we change the MTU, I expect the router to drop those packets and send an ICMP error back.  The X factor is how the individual apps will handle that in mid-stream: will they adapt on the fly?  Reset the connection and try again?  Or just sit down and cry until rebooted by a sysadmin?  I recognize that some apps may be different, I'm just looking to get an overall picture.

[Mohamed Sobair]

Dropping the MTU on the tunnel interface of the DMVPN should have no impact at all.

Your impact becomes when you increase the MTU of the DMVPN, hence fragmentation required which some application wouldnt adapt and would therefore reset the connection and try again.

This behaviour causes delay and occasionally packet drops and lead to performance degradation and timeout of many application.

However, lowering it to advicable Value should have NO impact at all.

Regards,

Mohamed

[paolo bevilacqua]

The reply above is not only 100% correct, but also 100% useful, and as such I have rated it.

The person that has low rated it should either come forward and explain why, or refrain from misusing again the rating system in the future.

[mikep1230]

I rated the post.  I meant no offense, and it's not personal: my rating reflects the fact that the post is 90% accurate, but unfortunately not useful, as it does not address the question I asked.  I'd welcome any assistance in revising my question to be more clear.

In more specific terms, here is the behavior I'm expecting:

1.  A TCP (for example) connection is initiated between two of my sites today.  The MSS value is set to 1360 bytes (which is only determined during the 3-way handshake), so the application expects to be able to send 1400B packets.

2.  The connection begins, and the 1400B packets flow.  Since my MTU is set to 1400, this is not a problem.

3.  I change the MTU to 1380.

4.  The next packet that arrives as part of this conversation will still be 1400B in size.  If the application set the DF bit, this packet will be dropped by the router, and an ICMP 'fragmentation required' message will be sent.

Clearly, dropping packets has an impact on the application (hence my disputing the statement that 'lowering the MTU will have no impact').  My question is: how do applications tend to handle this scenario?  Is the ICMP reply typically received and acted upon?  Or does the application continue to retransmit packets (receiving no ACK) until it times out?  If so, might the MTU information be cached somewhere, causing problems even with new connections?

Again, I recognize that each application or OS may well be different.  RFC 1191 describes some behavior that's relevant here, but it wouldn't surprise me if it there were variation in how it's implemented.  That's why I'm primarily looking for experience: has anyone done this?  How did it go?

[paolo bevilacqua]

I think that technical matters aside, you are having unreasonable expectations about what a freely supported forum can provide, especially considering that you came here having the historyof single post, apparently in the contingency of some problem only.

And again, low rating posts of great quality coming from senior members is exactly the way do de-incentivate others from helping you.Since it seems you're not even willing to acknoledge that, I can only whish you good luck.

[mikep1230]

I'm afraid we're going to have to agree to disagree, Paolo.

While I understand you contend that posts should be rated based on the intentions and history of the poster, I feel it's more appropriate to rate them based on their content (accuracy, relevance, etc).  I can only assume that those who designed this forum shared my opinion, as the rating scale is described in terms of how "helpful" the post is.  If I posted a *fantastic* cookie recipe in response to a NAT question, that would also be "not helpful."

When I solicited input from a technical community about their experience with a specific technical change, my only expectation was to learn from those who had relevant input and chose to share it.  I'm not sure if you feel that's unreasonable, if I somehow implied that I had any other expectation, or if you believe that it's inappropriate for someone's first post to be a technical question.

I apologize; I was remiss in not thanking Mohamed.  I do appreciate both the effort he put into it and his willingness to help.  I would likewise appreciate any other input on the *technical* question I asked.

I have to be honest, I do not at this point have any interest in debating these non-technical issues; further spam in this thread only serves to distract from the intended topic.  You're certainly welcome to either send me a private message or start a new discussion, whichever you prefer.

[Mohamed Sobair]

First Of All, Paolo, I do appreciate your comment, you guys actually who makes this forum useful , thats why you are being recognized as a Hall Of Fame.

For the Original Poster:

Your original post was about MTU and its affect on some application, TCP based application as you know can be adjusted with the (Maximumm Segment Size), the maximum segment size do corelate with MTU, how?

Technically the TCP based application increases its window size unless the packet is dropeed, if the TCP session is dropped, its decreased by half and starts increases its window size again. This behaviour can be adjusted by the MSS value which instruct the TCP application to a limited value define as a MSS.

At this point, the packet is originated from the Host with MSS of 1360, Since the packet goes over IPsec DMVPN, then the following values are added to the total MTU. (Assuming the tunnel is set with 1400 MTU).

1- IPsec Header which is 56byte.

2- The GRE and IP header of 24 byte.

The total MTU value of the packet becomes 1480.

Ok, if the router interface is set with an MTU value of 1450 for example, the packets needs to be fragmented , and the fragmentation can cause delays and drops on the Network which is not recommended.

Here, if you start a ping request with DF bit set, it simply means send the packet without fragmentation, the router then would drop the packet since the outgoing interface MTU is set to 1450, and would send an ICMP message indicating that fragmentation is needed but DF bit is set. So in this case, the TCP session is dropped and rested.

Leaving the ICMP ping aside, some application including TCP based, doesnt adapt with fragmentation which causes slowness and performance degradation on a Network, therefore, its always recommended to set accurate MTU values and MSS Value in order to avoid any fragmentation.

Note: The MSS value normally adjusted to a value that is minus 40 from the total Output MTU, this 40 is simply the TCP header value.

BTW:

This Forum represents a Knowledge base, Technical Information and  Second Cisco TAC, that helps solves original poster inquiries and requirment. Evaluating the post by using the rating system helps other users in the future. Thats what Paolo meant to indicate.

Thanks and Food Luck,

Mohamed

[Thompson Obuseh]

Hello Mohamed Sobair ,

Your explanation was on point. I  am having a similar challenge of slow network complains from some spokes  below is the configure in the hub and spoke can you view it can comment pls . I have attached the config for the hub and a spoke 

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Why is CEF disabled?

Why spoke's tunnel 70 not using the ip mtu, adjust-mss, pmtud?

Have you considered using transport mode?

Is transient congestion possible?  Physical interfaces gig but is end-to-end also gig?

[dgonzalezarias]

I am currently having issues with cameras trying to communicate with a server over DMVPN/IPsec. I am seeing fragmentation from the camera sending to the server. The MTU is set to 1400 and mss to 1360. Would there be a big impact on my network if I increase the mtu to 1500 or 1520? The packets are coming in at 1414.

[Joseph W. Doherty]

In my experience, camera video traffic generally uses UDP without DF set, as the camera isn't configured to retransmit dropped packets. So, on links that cannot handle the camera video traffic's packet size, it gets fragmented.

Insuring the link can handle the camera's video traffic max MTU, would avoid this fragmentation, but you cannot willy-nilly just increase the MTU - it depends on what the hardware can handle (and tunnel overhead).

What you might be able to do, is have the camera's use a smaller MTU to avoid fragmentation.

[Thotsaphon Lueangwattanaphong]

Paolo,

       I like your post.

Rgds,

Toshi

[Joseph W. Doherty · Accepted solution]

Disclaimer

The    Author of this posting offers the information contained within this    posting without consideration and with the reader's understanding that    there's no  implied or expressed suitability or fitness for any    purpose.  Information provided is for informational purposes only and    should not be construed as rendering professional advice of any kind.     Usage of this posting's information is solely at reader's own risk

Liability Disclaimer

In   no event shall Author be liable for any damages whatsoever (including,   without limitation, damages for loss of use, data or profit) arising    out of the use or inability to use the posting's information even if    Author has been advised of the possibility of such damage.

Posting

My understanding is, if the source gets the ICMP fragmentation message it will reduce the size of all newly sent packets for that flow.  As far as I know, once packet size is reduced, that flow will never increase the size.

I also recall older fragmentation left the sender to try various sized packets until fragmentation no longer needed.  I think newer ICMP implementations will also include what the size needs to be reduced to.  If this is correct, the former would cause more of a performance hit until the correct MTU was found.

From personal experience, I've noticed TCP flows that start up with a too large MTU take a noticable performance hit, at least initially, than starting with a non-fragmenting size (such as can be provided by the ip tcp adjust-mss command).  NB: this observation is from several years ago, although the adjust command is still very, very useful for TCP start up where you expect a MTU size drop (and you can allow for that).

Flows should adapt. Have had the situation where primary path across dedicated WAN link, normal MTU supported, but fail over was via VPN tunnel, with reduced MTU.

PS:

What does cause problems is when source doesn't receive ICMP message.  This is espeically annoying when the MTU only differs by a few bytes, like PPoE or incorrect allowance for MPLS tags.  Most packets get through, some don't.

Another option is to allow intermediate system to fragment packets, but that can overload the device doing the fragmentation and actually uses extra bandwidth as packet headers are replicated during fragmentation.

[mikep1230]

Thanks, Joseph!  That all makes sense.

I'd made the logical jump to "should look like a failover to VPN" as well -- that's exactly how our network works.  It's just not a very common occurrence.

Good tip on the ICMP bit; I'd thought of that too, and am reasonably confident that no ICMP will be blocked.  I did mean to research if "no ip unreachables" could cause an issue; I believe the 'fragmentation required' code is within the same type, and I wonder if Cisco routers make the distinction.

For the record, the fragmentation/CPU usage issue is exactly why we're making this change: our current MTU (1400) is actually about 2 bytes too high when you factor in 256-bit AES and a PPPoE Internet connection.

[Joseph W. Doherty]

Disclaimer

The     Author of this posting offers the information contained within this     posting without consideration and with the reader's understanding  that    there's no  implied or expressed suitability or fitness for  any    purpose.  Information provided is for informational purposes only  and    should not be construed as rendering professional advice of any  kind.     Usage of this posting's information is solely at reader's own  risk

Liability Disclaimer

In    no event shall Author be liable for any damages whatsoever  (including,   without limitation, damages for loss of use, data or  profit) arising    out of the use or inability to use the posting's  information even if    Author has been advised of the possibility of  such damage.

Posting

Although perhaps not applicable in you situation, if your flows can be subjected to a drop in MTU, as in a case such a VPN backup, you might consider using the ip tcp adjust-mss on both paths.  This insures, at least for TCP flows, they won't need to adjust MTU on the fly.  Doing so trades off the slight loss in bandwidth usage efficiency (on the path with the larger MTU) with no impact to the flow as it adjusts to the small path's MTU.  (If your not already using this command, you might consider doing so, assuming you platform supports it.)

Also with multiple active paths that have different, but known MTUs, you could also reduce the MTU on the "better" path, but forcing a unnecessary MTU adjust, when not actually needed, seems to offer little benefit since once a specific flow is on a particular path, it often stays on that path.

In the latter case of knowing I have two active paths with different MTUs, or a single path with smaller MTU (as your case?), I have considered modifying all clients' MTUs to match known path minimum MTU, but didn't think the benefit outweighs the work in most instances if more than a few hosts are involved.  (Have done it for single PC using a aDSL PPPoE connection.)