1500 MTU across a GRE creating Data Loss/CPU Spikes

[grant.walker]

Has anybody come across the following issue before :

We are running a GRE and have set MTU at 1500. When you ping across the tunnel you get a single 1500 MTU reply. ie you cannot see the fragmentation.

When running an integrity check across this link, every 60 seconds we see data loss of upto 60% for a couple of seconds. The higher the capcity the higher the data loss. This is also greater in one direction than in the other, so it is not a packet loss issue.

When we run at the std 1476, across the tunnel, we do not see this.

1. We are not sure if the issue is CPU related or MTU/GRE related.

2. Can anyone recommend a solution here.

CPE side is a 2811, PE side is a 7206VXRG2

Thanks

[Peter Paluch]

Hello Grant,

To me, this looks as an MTU issue. You said that pinging with MTU 1500 across the tunnel did not reveal any fragmentation. What do you mean by that? The fragmentation most probably took place, as the GRE overhad is 24 bytes in total, and not fragmenting the resulting packet would result in a 1524 bytes length - a value not acceptable by ordinary Ethernet . Did you use the df-bit option when pinging?

Do you personally see any reason why not to decrease the MTU on the tunnel to 1476?

Best regards,

Peter

[grant.walker]

Hello Peter,

We set the df bit and it works well at 1500.

The application requires an unfragemented packet and the application works, however we see a data loss spike every 60 seconds.

Regards

Grant

[Peter Paluch]

Hello Grant,

Interesting. Is that a pure GRE tunnel or is it also protected by IPsec? What are the MTUs of physical interfaces that carry the GRE-encapsulated traffic? Is there any special traffic being carried over the GRE tunnels, along with your application, that would have roughly the 60s period of occuring?

Best regards,

Peter

[grant.walker]

Hello Peter

It is a pure GRE. The MTU is 1480 from the underlying providor. It is a L3 medium, thus the initial requirement to establish the GRE.

There is no special traffic at all, just the testing program which works perfectly at both 1476 and 1480 if we use IP/IP GRE

I was thinking it may be a BGP update of sorts every 60 seconds but ruled that one out by the fact that 1476 passed no problem. It is only when we set the df bit on the GRE that this happens. I am wondering if there is not some kind of keep alive or buffering on the MTU size we are setting or something along that line. We have tried 3 different sites to see if it was site specific but we get the same across all 3 sites.

Regards

Grant

[milan.kulik]

Hi guys,

I remember I was playing with this some time ago in my lab and realised:

When I use

ip mtu 1500

the large packest fragmented due to the tunnel MTU are reassembled on the other tunnel side!

Even without "ip virtual-reassembly" used.

See https://supportforums.cisco.com/message/3347227#3347227

So even when you set the d-f bit on, the packets exceeding the MTU size of the tunnel medium (1500 Bytes on Ethernet) are being fragmented and reassembled again. So no chance to detect the fragmentation using Wireshark on the target machine, e.g.

Which could explain a CPU overload on smaller routers like 2811.

But no idea why just every 60 seconds.

Maybe it's really the BGP process consuming the CPU which has not enough capacity to reassemble GRE packets at the same time?

HTH,

Milan

[grant.walker]

Hi Milan

We have shipped a CIsci 2921 to see if this assists in correcting the issue, failing which we will look at a 3945.

I also beleived it to be BGP updates becasue I do recall that BGP Scanner used to run every 60 seconds, but not sure if that is still the case. Maybe someone out there can comment on this.

When we monitor CPU process we cannot see the sudden spike by BGP Scanner but maybe we are just missing it. It could well be fighting with the MTU fragmentation and reassembly going on, which it would not have an issue with at 1476 or 1480

Regard

Grant

[Joseph W. Doherty]

Disclaimer

The       Author of this posting offers the information contained within  this      posting without consideration and with the reader's  understanding   that    there's no implied or expressed suitability or  fitness for any    purpose.   Information provided is for informational  purposes only  and   should not   be construed as rendering professional  advice of any  kind.   Usage of  this  posting's information is solely  at reader's own  risk.

Liability Disclaimer

In       no event shall Author be liable for any damages whatsoever     (including,   without limitation, damages for loss of use, data or     profit) arising  out  of the use or inability to use the posting's     information even if  Author  has been advised of the possibility of    such  damage.

Posting

Peter asked a pertinent question in his first post, if the problem does not appear when you run an IP MTU of 1476, why not just do that?

Provider says IP MTU is 1480, but provider might be wrong or they're misconfigured.  Doesn't happen often, but even providers make mistakes.

Provider says IP MTU is 1480 but as Peter also noted, for typical GRE on standard Ethernet, it should be 1476.  The difference of 4 bytes makes me wonder if MPLS is involved "under the covers".  I've seen a provider fail to correctly allow for the 4 bytes a MPLS header.  What is the underlying WAN technology?

As to pings working at 1500 with DF bit set, perhaps the provider ignores the DF bit and fragments anyway.

From your posted charts, if nominal bandwidth is about 4 Mbps, a 2811 shouldn't normally have an issue keeping up.  This would also seemed to be confirmed if you don't see any CPU spikes.

From everything described, think it unlikely the issue is due to performance capabilities of your routers, more likely "something" with transport across the provider's network.