big packet drop on MPLS over GRE tunnel

[wenqiang_kun]

hi, 

here is my scenario setup,

host A - - router A -- PE 1 --- PE2 ---GRE tunnel over AT&T EVPN circuit----PE3- router B- host B

I am currently setup MPLS over GRE tunnel with Cisco ASR9901,  i have configured  MTU 1400/  TCP- MSS-adjust-enable on tunnel interface,  and "hw-module location 0/0/CPU0 tcp-mss-adjust np 1 value 1300"on ASR router,  GRE tunnel is up, OSPF/MPLS LDP neighbor are all up, MPLS VPN are also up, when doing end to end host ping from host, small packet are going through MPLS GRE tunnel without any packet drop, big size packet more than 1360 will get dropped; when doing ping to remote tunnel interface from one end Cisco ASR router, there is no packet loss, can someone help me to find out why big size packet got dropped? did i make wrong configuration somewhere? thanks in advance

WENLI

[MHM Cisco World]

I think you need more reduce mtu and mss,

Try with 1200 

[wenqiang_kun]

thanks  for your kindness to reply quickly, it is prod system, i cannot change any time, is there any reason to change MSS/MTU to 1200? we need to transfer regular size packet(1500) through MPLS link, thanks.

[MHM Cisco World]

I have half answer 
why end to end is drop and ping directly from ASR not drop?
the end to end meaning 100% that the packet is first encap with GRE (with IPsec ) then send via MPLS which also add overhead to packet 
directly not meaning that packet can encap, it can not encap this depend on the overlay routing. 

why 1200? I suggest this it not fix, you can use ping sweep to see the large end-to-end MTU size you network can handle with fragment. 

[wenqiang_kun]

if MTU 1400 is not configured on GRE tunnel interface,  GRE overhead 24 bye, MPLS 2 label 8 bytes,  big packet size can be go through 1500-24-8=1468. once MTU 1400 is configured on GRE tunnel interface, the big packet will around 1360. the problem is packet can not be fragment on router when sending host to host traffic.

[Joseph W. Doherty]

To my knowledge, MPLS adds to physical frames (much like tagged VLAN frams), i.e. from your perspective, MTU should be 1500 (for standard Ethernet).  (NB: this assuming MPLS vendor has configured their equipment correctly [historically, I did once have an instance when an MPLS vendor didn't - which they fixed when I noticed 8 bytes missing {wasn't AT&T - though I once spent almost 4 months arguing with AT&T over a link that wasn't working correctly IMO - they finally found the problem}]).

"Pure" GRE only uses 24 bytes, from MTU, so tunnel should be configure with an IP MTU (not MTU, w/o IP) of 1476 and an TCP adjust-mss of 1436 (assuming typical IP/TCP header length).

With the forging, you should be able (on your Cisco router) ping up to 1476 with DF set.

For any TCP traffic, or non-TCP traffic, with DF set and larger than tunnel's IP MTU, packet will get dropped.  W/o DF set, packet will be fragmented.  I.e. "the problem is packet can not be fragment on router when sending host to host traffic." if that's a hard requirement, you're going to have a problem.  If you need tunnel packets of 1500, you'll need to see if AT&T can deal with a jumbo Ethernet capable hand-off.  (Or, you might need to consider other options for sending "tunnel like" traffic supporting 1500 bytes.  Which, BTW, do you really, really need to use a GRE tunnel? )

[Joseph W. Doherty]

Can you ping between the two GRE hosting routers using DF and a MTU of 1500?

Do you know whether AT&T can or will support Ethernet MTU larger than 1500?

Your GRE tunnel is just GRE, i.e. no IPSec?

When you note you cannot send a packet larger that 1360 (with DF and using ping, correct?), that's actual packet size, correct?  (Reason I ask, if doing pings, some hosts (e.g. PING Cisco IOS vs Windows) don't calculate "size" the same way Cisco routers do.

[wenqiang_kun]

there is no IPsec, it is MPLS over GRE tunnel.  we run ping test on client router in both side, when sending packet less than 1360, it went through, for packet bigger than 1360, it got dropped.

[MHM Cisco World]

new IP header for GRE 20 bytes 
GRE  header is 4 bytes 
MPLS label is 4 but the min is 2 but we will assume it 3, this equal to 12 bytes 
even if you not run IPSec that not meaning that SP not run IPSec which have 20 for new header and 24 for IPsec header this give 44 header 
so total 
20+4+12+44 = 80 
MSS need additional 20 IP header and 20-60 bytes TCP header (include option)
this also 60
so total 140
and give in end 1360
I run simple lab GRE over IPSec 
with IPsec tunnel mode 
with IPsec transport mode 
I worse case GRE with IPSec tunnel max ping is 1410 !!!
this not include the TCP header and MPLS Header. 
so I am sorry to inform you that it real can be not more than 1360. 

can you more elaborate more about the config of MTU 1400 and not config MTU 1400 ?

[wenqiang_kun]

thanks you for spending time to discuss this issue, I think i can adjust MTU little higher,  GRE  tunnel overhead: 24byte, MPLS overhead: 8byes, MTU can be adjusted to 1468 Byte,  I will try it during our next change window. right now, the issue is  that GRE tunnel packet can be fragment properly when doing ping to remote tunnel IP from router,   but IP packet crossing MPLS link over GRE tunnel  has fragment issue when doing end host to end host test

[MHM Cisco World]

can you confirm that when ping remote tunnel IP you use local tunnel IP as source ?

I will give one idea may it help you 
GRE tunnel 
tunnel IP X
tunnel source Y 
tunnel destination Z

I want to find the max the MTU 
do
ping tunnel destination Z source tunnel source Y 
find the MTU 
then reduce that MTU with 
24 bytes (new IP header and 4 GRE header)

[wenqiang_kun]

tunnel-ip10 is up, line protocol is up
Interface state transitions: 1
Hardware is Tunnel
Description: GRE tunnel to xxxx
Internet address is 10.199.163.173/30
MTU 1400 bytes, BW 100 Kbit (Max: 100 Kbit)

MTU size is setup as 1400, 

xxx#ping 10.199.163.174 size 1376 df-bit
Mon Sep 19 15:04:05.681 EDT
Type escape sequence to abort.
Sending 5, 1376-byte ICMP Echos to 10.199.163.174, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 134/135/136 ms
xxx##ping 10.199.163.174 size 1377 df-bit

Type escape sequence to abort.
Sending 5, 1377-byte ICMP Echos to 10.199.163.174, timeout is 2 seconds:
.....

MSS windows size is setup as 1300,  maximum IP packet size: 1300+20(TCP)+24(IP)+8(MPLS)=1352, it is less than MTU 1400 setup on tunnel interface,  for some reason, MSS windows adjustment did not work on Cisco ASR 9K even tcp-mss-adjust enabe is setup on tunnel interface.

[MHM Cisco World]

xxx#ping 10.199.163.174 source 10.199.163.173  size 1x00 df-bit

Try this way friend,