GRE Tunnel and MTU issue

[pbelletty]

Hi Folks,

I have a peculiar problem that I can't seem to get my head around. Hope you can help.

Have an ADSL router (887) at a site which has a GRE tunnel to to a 3745.

The GRE tunnel is setup with default ip mtu of 1476.

If I ping from the 3745 to the ADSL router (or in the reverse direction)
with a packet size of 1500 bytes this works fine.

However if I ping from a router (R1) that is directly connected to 3745 to the ADSL router with a pkt size of
1500 bytes then the first ping succeeds while the subsequent pings fail.
Pkt sizes less than or equal to 1476 work okay.

Pinging between R1 and the 3745 with a packet size of 1500 bytes works fine.

If I set the tunnel ip mtu size to 1500 bytes then it works.

This is obviously something to do with fragmentation, but I don't undertsand why it
doesn't work with the default mtu set to 1476.

Any ideas? Is it a bug?

Cheers,
Phil

[Peter Paluch]

Hello Phil,

This is strange. It would make slightly more sense if all packets from R1 were lost but I am puzzled by the fact that the first ping actually makes it, only the subsequent pings are lost. By the way, what means "the first packet"? Is it always the first ICMP ECHO message in each two subsequently executed ping commands, or does is there a longer period necessary before the first ping can make it again?

And by the way, is it possible to ping from the ADSL router back to R1?

Best regards,

Peter

[pbelletty]

Hi Peter,

Thanks for the reply.

Looks like it is now doing something slightly different in that the first pings are not necessarily successful.

Here is a selection of ping tests from R1 to the ADSL router - GRE MTU Set to 1476

R1#p vrf nwmht 10.132.214.254

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.132.214.254, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 17/18/25 ms

R1#p vrf nwmht 10.132.214.254 size 1500

Type escape sequence to abort.
Sending 5, 1500-byte ICMP Echos to 10.132.214.254, timeout is 2 seconds:
!..!.
Success rate is 40 percent (2/5), round-trip min/avg/max = 42/42/42 ms

R1#p vrf nwmht 10.132.214.254 size 1500

Type escape sequence to abort.
Sending 5, 1500-byte ICMP Echos to 10.132.214.254, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

R1#p vrf nwmht 10.132.214.254 size 1500

Type escape sequence to abort.
Sending 5, 1500-byte ICMP Echos to 10.132.214.254, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

R1#p vrf nwmht 10.132.214.254 size 1500

Type escape sequence to abort.
Sending 5, 1500-byte ICMP Echos to 10.132.214.254, timeout is 2 seconds:
.!...
Success rate is 20 percent (1/5), round-trip min/avg/max = 42/42/42 ms

R1#p vrf nwmht 10.132.214.254 size 1476

Type escape sequence to abort.
Sending 5, 1476-byte ICMP Echos to 10.132.214.254, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 42/45/51 ms

PINGING FROM THE ADSL ROUTER TO R1

ADSL#p 10.132.219.160 sou loop99

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.132.219.160, timeout is 2 seconds:
Packet sent with a source address of 10.132.214.254
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 20/20/20 ms

ADSL#p 10.132.219.160 sou loop99 size 1476

Type escape sequence to abort.
Sending 5, 1476-byte ICMP Echos to 10.132.219.160, timeout is 2 seconds:
Packet sent with a source address of 10.132.214.254
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 44/46/48 ms

ADSL#p 10.132.219.160 sou loop99 size 1500

Type escape sequence to abort.
Sending 5, 1500-byte ICMP Echos to 10.132.219.160, timeout is 2 seconds:
Packet sent with a source address of 10.132.214.254
!....
Success rate is 20 percent (1/5), round-trip min/avg/max = 48/48/48 ms

ADSL#p 10.132.219.160 sou loop99 size 1500

Type escape sequence to abort.
Sending 5, 1500-byte ICMP Echos to 10.132.219.160, timeout is 2 seconds:
Packet sent with a source address of 10.132.214.254
..!..
Success rate is 20 percent (1/5), round-trip min/avg/max = 44/44/44 ms

ADSL1#p 10.132.219.160 sou loop99 size 1500

Type escape sequence to abort.
Sending 5, 1500-byte ICMP Echos to 10.132.219.160, timeout is 2 seconds:
Packet sent with a source address of 10.132.214.254
.....
Success rate is 0 percent (0/5)

Regards,

Phil

[Peter Paluch]

Phil,

If you used a ping size of 1477B, what would the result be? Also, use the repeat 30 to see a longer run of pings in one instance.

Is the GRE tunnel also compounded with IPsec protection, or is that a pure GRE?

Best regards,

Peter

[pbelletty]

Hi Peter,

1477B fails in a similar manner to 1500B. Please see below results.

ADSL#ping 10.132.219.160 sou loop99 size 1477

Type escape sequence to abort.

Sending 5, 1477-byte ICMP Echos to 10.132.219.160, timeout is 2 seconds:
Packet sent with a source address of 10.132.214.254
!....
Success rate is 20 percent (1/5), round-trip min/avg/max = 52/52/52 ms

ADSL#ping 10.132.219.160 sou loop99 size 1477 rep 30

Type escape sequence to abort.
Sending 30, 1477-byte ICMP Echos to 10.132.219.160, timeout is 2 seconds:
Packet sent with a source address of 10.132.214.254
!.................!...........
Success rate is 6 percent (2/30), round-trip min/avg/max = 44/48/52 ms

Best regards,

Phil

p.s. Forgot to mention that there is no IPSec just a plain old GRE Tunnel

[Peter Paluch]

Phil,

This is totally puzzling. Logic suggests that if the pings from the 3745 to the ADSL router are okay while the pings from R1 via 3745 to the ADSL router get lost, something must be wrong with routing and fragmenting the packets on 3745.

Do you believe you could run a debug on your 3745 to see more closely what is going on with the packets? Follow this suggestion:

configure terminal

access-list 100 permit icmp any any

end

debug ip packet 100 detail

debug ip error 100 detail

debug ip icmp

After setting up the debugs as shown here, ping the ADSL router from the R1 again (use pings larger than 1476B). This will perhaps help us narrow down any malignant processes on the 3745 should they take place.

Best regards,

Peter

[pbelletty]

Hi Peter,

Tried as you suggested.

Unfortunately the 3745 is a production router and is running IP CEF. As the packets are CEF switched debug does not work.

However I have noticed that if I send a pkt of 1477B for those packets that succeed I see a debug message indicating that the

packet is being fragmented.

Mar 17 09:04:57.734: IP: tableid=2, s=10.132.219.160 (FastEthernet1/0.20), d=10.132.214.254 (Tunnel10), routed via FIB

Mar 17 09:04:57.734: IP: s=10.132.219.160 (FastEthernet1/0.20), d=10.132.214.254 (Tunnel10), g=10.132.219.163, len 1477, forward

Mar 17 09:04:57.734:     ICMP type=8, code=0

Mar 17 09:04:57.734: IP: s=10.132.219.160 (FastEthernet1/0.20), d=10.132.214.254 (Tunnel10), len 1476, sending fragment

Mar 17 09:04:57.734:     IP Fragment, Ident = 7275, fragment offset = 0

Mar 17 09:04:57.734:     ICMP type=8, code=0

Mar 17 09:04:57.734: IP: s=10.132.219.160 (FastEthernet1/0.20), d=10.132.214.254 (Tunnel10), len 21, sending last fragment

Mar 17 09:04:57.734:     IP Fragment, Ident = 7275, fragment offset = 1456 Mar 17 09:04:57.734: IP: tableid=2, s=10.132.219.160 (FastEthernet1/0.20), d=10.132.214.254 (Tunnel10), routed via FIB
Mar 17 09:04:57.734: IP: s=10.132.219.160 (FastEthernet1/0.20), d=10.132.214.254 (Tunnel10), g=10.132.219.163, len 1477, forward
Mar 17 09:04:57.734:     ICMP type=8, code=0
Mar 17 09:04:57.734: IP: s=10.132.219.160 (FastEthernet1/0.20), d=10.132.214.254 (Tunnel10), len 1476, sending fragment
Mar 17 09:04:57.734:     IP Fragment, Ident = 7275, fragment offset = 0
Mar 17 09:04:57.734:     ICMP type=8, code=0
Mar 17 09:04:57.734: IP: s=10.132.219.160 (FastEthernet1/0.20), d=10.132.214.254 (Tunnel10), len 21, sending last fragment
Mar 17 09:04:57.734:     IP Fragment, Ident = 7275, fragment offset = 1456

Best regards,

Phil

[pbelletty]

Hi Peter,

Looks like there is a bug.

The Tunnel is configured within a vrf. If I remove this and run as part of the global routing table then it works.

I think I will raise a case with Cisco and see what they say.

Will let you know what Cisco say.

Thanks for you help.

Best regards,

Phil

[Peter Paluch · Accepted solution]

Hello Phil,

Yes, it does exhibit traits of a bug.

I knew that CEF-switched packets will not be displayed in my debugs. However, packet fragmentation does not occur in the CEF path but rather in process switching, and I was interested exactly in seeing the fragmentation process - therefore I did not have doubts that the fragmentation process will produce interesting debugs.

Were there absolutely no debug messages displayed for those packets that got lost? Those debugs would have been the most interesting - of course, if there were any.

Best regards,

Peter

[pbelletty]

Hi Peter,

Thanks for the reply. Yeah the debug did not show any output for the packets that were lost.

All the best

Phil

[pbelletty]

Hi Peter

Just to let you know that I raised a TAC case for this and the TAC engineer was of the opinion that this was a bug. I am running an old version that is no longer supported.

I haven't updated the IOS yet as I have a work around setting the tunnel MTU to 1500. I appreciate that this puts a small

additional load on the router.

Thanks for all you help.

All the very best

Phil

[Peter Paluch]

Hello Phil,

Thank you very much for letting me know!

Best regards,

Peter