Re: Cisco ISR First Generation drop GRE Tunnel trafic due to fragmenta

JEROME BOSC · ‎04-29-2022

Hi,

I am concerned regarding a performance issue on an ISR 3945 (ISR 1st generation).

The main issue is an MTU problem. We are using an MPLS over GRE tunnel, this impacct the MTU..

But the customer do not receive our ICMP packet indication for decrease it's MTU due to filtering. So the solution is on the customer side by decreasing it's MTU.

By the way, I was struggling troubleshooting this problem as :

A GRE tunnel interface received fragmented trafic and cannot cope when trafic increase (above 500kb/s ...)

On the router itself, there is no log, no counters, no buffer problem counter, no CPU peak, nothing indicate it drops IP fragmented packet ...

The only way is to sniff the lans on both side and check that packet arriving on one end of the router, do not cross the router ....

Have you ever face this situation ? How can you check counters or logs to validate those drops ?

Appart from changing the hardware (by an ISR 2 generation) that cope with fragments, have you any idea about a solution ?

Regards - Jerome.

David Ruess · ‎04-29-2022

Hello,

You may also need to set the TCP MSS adjust window.

on the Tunnel interface:

ip tcp adjust-mss <####> (Usually 40 less than the MTU) So if MTU was 1400 the MSS would be 1360 - do this on both sides.

Can you post the output of the show interface tunnel <#> command

-David

JEROME BOSC · ‎05-09-2022

Hi, here are the show tunnel interfaces. The first 2 tunnels (1001 & 1002) are connected to the datacenter.
The last one, tunnel 7 is connected to the end user CPE.
  
Tunnel interface FROM datacenter (incoming traffic) :

Tunnel1001 is up, line protocol is up
  Hardware is Tunnel
  Description: Tu1001_CAM3-MSRT6_Tu1005
  Internet address is A.B.C.D/30
  MTU 17916 bytes, BW 10000 Kbit/sec, DLY 50000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation TUNNEL, loopback not set
  Keepalive not set
  Tunnel linestate evaluation up
  Tunnel source E.F.G.H (Port-channel1.5), destination I.J.K.L
   Tunnel Subblocks:
      src-track:
         Tunnel1001 source tracking subblock associated with Port-channel1.5
          Set of tunnels with source Port-channel1.5, 2 members (includes iterators), on interface <OK>
  Tunnel protocol/transport GRE/IP
    Key disabled, sequencing disabled
    Checksumming of packets disabled
  Tunnel TTL 255, Fast tunneling enabled
  Tunnel transport MTU 1476 bytes
  Tunnel transmit bandwidth 8000 (kbps)
  Tunnel receive bandwidth 8000 (kbps)
  Last input 2w4d, output 00:00:01, output hang never
  Last clearing of "show interface" counters 39w3d
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/0 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     746777170 packets input, 500494953315 bytes, 0 no buffer
     Received 0 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     669341643 packets output, 171152714273 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out
Tunnel2001 is up, line protocol is up
  Hardware is Tunnel
  Description: Tu2001_CAM4-MSRT6_Tu1005
  Internet address is M.N.O.P/30
  MTU 17916 bytes, BW 10000 Kbit/sec, DLY 50000 usec,
     reliability 255/255, txload 6/255, rxload 28/255
  Encapsulation TUNNEL, loopback not set
  Keepalive not set
  Tunnel linestate evaluation up
  Tunnel source E.F.G.H (Port-channel1.5), destination Q.R.S.T
   Tunnel Subblocks:
      src-track:
         Tunnel2001 source tracking subblock associated with Port-channel1.5
          Set of tunnels with source Port-channel1.5, 2 members (includes iterators), on interface <OK>
  Tunnel protocol/transport GRE/IP
    Key disabled, sequencing disabled
    Checksumming of packets disabled
  Tunnel TTL 255, Fast tunneling enabled
  Tunnel transport MTU 1476 bytes
  Tunnel transmit bandwidth 8000 (kbps)
  Tunnel receive bandwidth 8000 (kbps)
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 39w3d
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/0 (size/max)
  5 minute input rate 1114000 bits/sec, 191 packets/sec
  5 minute output rate 256000 bits/sec, 168 packets/sec
     785326724 packets input, 467429994156 bytes, 0 no buffer
     Received 0 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     702640047 packets output, 183055026466 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out

//////////////////////////////////////////////////////////////////////////////////

Tunnel 7 : interface TO destination, CPE Router (outgoing traffic) :

Tunnel7 is up, line protocol is up
  Hardware is Tunnel
  Description: tunnel7-K307
  Internet address is R.S.T.W/30
  MTU 17916 bytes, BW 10000 Kbit/sec, RxBW 4000 Kbit/sec, DLY 50000 usec,
     reliability 255/255, txload 1/255, rxload 5/255
  Encapsulation TUNNEL, loopback not set
  Keepalive not set
  Tunnel linestate evaluation up
  Tunnel source S.S.S.S (Port-channel1.2), destination D.D.D.D
   Tunnel Subblocks:
      src-track:
         Tunnel7 source tracking subblock associated with Port-channel1.2
          Set of tunnels with source Port-channel1.2, 7 members (includes iterators), on interface <OK>
  Tunnel protocol/transport GRE/IP
    Key disabled, sequencing disabled
    Checksumming of packets disabled
  Tunnel TTL 255, Fast tunneling enabled
  Tunnel transport MTU 1476 bytes
  Tunnel transmit bandwidth 8000 (kbps)
  Tunnel receive bandwidth 8000 (kbps)
  Last input 00:00:07, output 00:00:00, output hang never
  Last clearing of "show interface" counters 8w1d
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/0 (size/max)
  5 minute input rate 81000 bits/sec, 35 packets/sec
  5 minute output rate 73000 bits/sec, 31 packets/sec
     68221885 packets input, 15503516922 bytes, 0 no buffer
     Received 0 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     72531044 packets output, 31402593836 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out

Hope it helps. By the way, using a 4331 fixed the problem. I just would like to know how can I troubleshoot those IP packet drops without any indicated counters ....

MHM Cisco World · ‎04-29-2022

Use extended Ping
with sweep
you will see until where the MTU is drop and then config the interface with that MTU.

Cisco ISR First Generation drop GRE Tunnel trafic due to fragmentation