Fragmentation issue on GRE tunnels on Cisco NCS-5502-SE IOS-XR 7.3.2

[Alessandro Veras]

Hi Cisco eXPerts.

I'm having an issue with a GRE tunnel on the Cisco NCS-5502-SE router running the IOS-XR 7.3.2 64 bits.

This is my scenario

[server-a] ==> [core-router-1] === GRE tunnel ===== [ internet ] ===== GRE === [edge router] <== [server-b]

linux    <===  [ASR9004-7.3.2] ====================================== [NCS5502-SE-7.3.2] ===> linux

So basically, I'm trying to send large packets [1500bytes] from server-a to server b, transversing the GRE tunnel via the internet.

Any packet in the 1476 bytes size goes through the GRE tunnel with no issues. But if I try anything larger than 1476, it doesn't work.

It sees that the issue is with the NCS5500, as if I send packets from server-a to server b I'm able to see the packets arriving at server-b using tcpdump. But the other way around won't work.

I'm also unable to ping the /31 address of the remote interface using packets larger than 1476. So the issue for me is the NCS5502-SE not able to fragment packets larger than 1476 when forwarding it through the tunnel.

The configuration is pretty basic, but I'll share below:

Router A(Core Router - ASR9k  

interface tunnel-ip5
 mtu 1500
 vrf MY-VRF
 ipv4 address X.X.X.14 255.255.255.254
 ipv4 tcp-mss-adjust enable
 tunnel mode gre ipv4
 tunnel source Bundle-Ether498
 tunnel destination X.X.X.212

Router B (NCS5500):

interface tunnel-ip5
 mtu 1500
 ipv4 address X.X.X.15 255.255.255.254
 load-interval 30
 tunnel mode gre ipv4
 tunnel source TenGigE0/0/0/22/0
 tunnel destination X.X.X.238

[paul driver]

Hello
Fragmentation is occurring to the ip packet due the mtu overhead the GRE tunnel incorporates, IP MTU sets maximum size of packets that can be sent on an rtrs interface, So if you decrease this default MTU (1500) to accommodate for this GRE overhead (24) it will make sure unnecessary packet fragmentation to be reduced, A setting of 1400 best practice, in additional you could also append ip tcp adjust-mms as it will assist end hosts residing behind the gre rtrs see each other maximum payloads for any tcp sessions.

int tun1
ip mtu 1400
ip tcp adjust-mms 1360

[Alessandro Veras]

Hey @paul driver Thank you for your reply. I didn't mention but IOS-XR is properly setting the ipv4 MTU for the interface. So there's no fragmentation. That's actually the issue, fragmentation is not happening for larger packets than 1476.

RP/0/RP0/CPU0:router#show ipv4 interface tunnel-ip5
Fri Oct  7 14:07:57.907 UTC
tunnel-ip5 is Up, ipv4 protocol is Up 
  Vrf is default (vrfid 0x60000000)
  Internet address is X.X.X.X.15/31
  MTU is 1500 (1476 is available to IP)
  Helper address is not set
  Directed broadcast forwarding is disabled
  Outgoing access list is not set
  Inbound  common access list is not set, access list is not set
  Proxy ARP is disabled
  ICMP redirects are never sent
  ICMP unreachables are always sent
  ICMP mask replies are never sent
  Table Id is 0xe0000000

[Elliot Dierksen]

Just because the router is setting that available MTU doesn't mean that the upstream devices known about this and will adapt to it. @paul driver is correct in suggesting the adjust-mss command to resolve the issue. Changing the MTU isn't a requirement, but the adjust-mss is required.

[Alessandro Veras]

Hi.

tcp adjust-mss is used to avoid fragmentation. But there are cases, especially for UDP, that we can't avoid it. So let's go through the expected behavior, right?

So, it's expected that any packets larger than 1476 received by the edge router from server-b to be fragmented at the GRE interface. 

That's my issue. The router is not fragmenting the packet. I never see a packet arriving at server-a.

When I do the same test from server-a to server-b I can see that the core router is fragmenting the packet, and I can see the packets arriving at server-b

[MHM Cisco World]

we dont have idea about the ISP, it can asymmetric path, meaning 
from Server-A to Server-B path1
from Server-B to Server-A path 2
in path2 if there is any L2 SW then the packet can not fragment and drop. 
so better is run PMTUD and then check the MTU in each side finally use lower MTU as ref. to config tcp mss.

[Alessandro Veras]

Hi, it could be a possibility, considering by default, the GRE packets have the DF bit set in the IP header, but that's not the case.

I tried the tunnel dfbit disable command that removes the df bit from the GRE IP header, and it still doesn't work. But I got absolute proof that something is wrong with the Router after configuring a GRE tunnel between both edge routers locally via a directly connected interface, and the behavior is still the same. 

edge router 1 ====== be1 =========== be1====== edge router 2

[MHM Cisco World]

can I know the command you use to disable df-bit in tunnel ?

[Alessandro Veras]

Sure, 

tunnel dfbit disable