12-25-2018 06:01 AM
I have two Cisco 2901 routers with simple GRE tunnel between them (no IPSec, etc).
Router A:
interface Tunnel3
ip address 10.10.253.9 255.255.255.252
keepalive 3 3
tunnel source <Router A IP>
tunnel destination <Router B IP>
Router B:
interface Tunnel3
ip address 10.10.253.10 255.255.255.252
keepalive 3 3
tunnel source <Router B IP>
tunnel destination<Router A IP>
Tunnel have no real traffic yet
I have linux server behind each router, so network topology is:
[Linux A] <=(eth)=> [RouterA] <=(GRE tunnel)=> [Router B] <=(eth)=> [Linux B]
The problem: [Router A] drops some packets from [Linux A] to [Linux B]
I found that the length of dropped packets are 1368, while smaller and bigger (with len not equal to 1368) packets are ok.
I found the following on [Router A]:
1. On each packet drop "sh int tun3 summ" OQD value ( pkts dropped from output queue) is increasing (
Interface IHQ IQD OHQ OQD RXBS RXPS TXBS TXPS TRTL
---------------------------------------------------------------------------------
* Tunnel3 0 0 0 167261 11000 19 58000 15 0
2. On each packet drop "show ip cef switching statistics" RP LES Fragmentation failed, DF / Drop value is increasing
Reason Drop Punt Punt2Host
RP LES No route 149 0 57
RP LES Packet destined for us 0 254525170 133518
RP LES No adjacency 142 0 0
RP LES Incomplete adjacency 8 0 0
RP LES TTL expired 0 0 273
RP LES Fragmentation failed, DF 173806 0 136270
RP LES Features 2 0 268
RP LES IP redirects 0 0 36
RP LES Unclassified reason 2 0 0
RP LES Neighbor resolution req 10 6 0
RP LES Fragmentation no pak 0 0 1510
RP LES Tun decap, gre keepalive 0 176585 0
RP LES Total 174119 254701761 271932
All Total 174119 254701761 271932
Packet size 1368 tells me to think about MTU size, but setting "ip mtu" from 1340/1368/1400 on both sides has no any effect.
"tunnel path-mtu-discovery" on both sides also has no any effect.
So, please help me to solve this problem.
here is the the "sh int tu3" output from [Router A]:
Tunnel3 is up, line protocol is up
Hardware is Tunnel
Internet address is 10.10.253.9/30
MTU 17916 bytes, BW 100 Kbit/sec, DLY 50000 usec,
reliability 255/255, txload 255/255, rxload 124/255
Encapsulation TUNNEL, loopback not set
Keepalive set (3 sec), retries 3
Tunnel linestate evaluation up
Tunnel source <Router A IP>, destination <Router B IP>
Tunnel protocol/transport GRE/IP
Key disabled, sequencing disabled
Checksumming of packets disabled
Tunnel TTL 255, Fast tunneling enabled
Tunnel transport MTU 1476 bytes
Tunnel transmit bandwidth 8000 (kbps)
Tunnel receive bandwidth 8000 (kbps)
Last input 4d02h, output 00:00:02, output hang never
Last clearing of "show interface" counters 16w1d
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 167310
Queueing strategy: fifo
Output queue: 0/0 (size/max)
5 minute input rate 49000 bits/sec, 63 packets/sec
5 minute output rate 184000 bits/sec, 58 packets/sec
1767847443 packets input, 87177447 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
1863557134 packets output, 838702796 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out
Is it ok that interface MTU is 17916 while tunnel transport mtu is 1476 ?
Solved! Go to Solution.
12-25-2018 12:32 PM
12-25-2018 06:17 AM
Here is the simple example:
Linux A IP - 10.2.62.2
Linux B IP - 10.11.16.83
From router B I'm telneting to port 80 of Linux A:
LinuxB# telnet 10.2.62.2 80
Trying 10.2.62.2...
Connected to 10.2.62.2.
Escape character is '^]'.
GET /
Connection closed by foreign host.
Here is the tcpdump output from LinuxA:
17:07:53.394529 IP 10.11.16.83.58526 > 10.2.62.2.80: Flags [S], seq 2982056106, win 65535, options [mss 1380,nop,wscale 5,nop,nop,TS val 1081021219 ecr 0,sackOK,eol], length 0
17:07:53.394601 IP 10.2.62.2.80 > 10.11.16.83.58526: Flags [S.], seq 93690137, ack 2982056107, win 28960, options [mss 1460,sackOK,TS val 2630416160 ecr 1081021219,nop,wscale 7], length 0
17:07:53.420421 IP 10.11.16.83.58526 > 10.2.62.2.80: Flags [.], ack 1, win 4104, options [nop,nop,TS val 1081021245 ecr 2630416160], length 0
17:07:57.191199 IP 10.11.16.83.58526 > 10.2.62.2.80: Flags [P.], seq 1:8, ack 1, win 4104, options [nop,nop,TS val 1081024967 ecr 2630416160], length 7: HTTP: GET /
17:07:57.191250 IP 10.2.62.2.80 > 10.11.16.83.58526: Flags [.], ack 8, win 227, options [nop,nop,TS val 2630419956 ecr 1081024967], length 0
17:07:57.195669 IP 10.2.62.2.80 > 10.11.16.83.58526: Flags [.], seq 1:2737, ack 8, win 227, options [nop,nop,TS val 2630419961 ecr 1081024967], length 2736: HTTP
17:07:57.195686 IP 10.2.62.2.80 > 10.11.16.83.58526: Flags [P.], seq 2737:5442, ack 8, win 227, options [nop,nop,TS val 2630419961 ecr 1081024967], length 2705: HTTP
17:07:57.195929 IP 10.2.62.2.80 > 10.11.16.83.58526: Flags [F.], seq 5442, ack 8, win 227, options [nop,nop,TS val 2630419961 ecr 1081024967], length 0
17:07:57.222287 IP 10.11.16.83.58526 > 10.2.62.2.80: Flags [.], ack 1, win 4104, options [nop,nop,TS val 1081024997 ecr 2630419956,nop,nop,sack 1 {4105:5442}], length 0
17:07:57.222324 IP 10.11.16.83.58526 > 10.2.62.2.80: Flags [.], ack 1, win 4104, options [nop,nop,TS val 1081024997 ecr 2630419956,nop,nop,sack 1 {4105:5442}], length 0
17:07:57.237384 IP 10.2.62.2.80 > 10.11.16.83.58526: Flags [.], seq 1:1369, ack 8, win 227, options [nop,nop,TS val 2630420003 ecr 1081024997], length 1368: HTTP
17:07:57.469410 IP 10.2.62.2.80 > 10.11.16.83.58526: Flags [.], seq 1:1369, ack 8, win 227, options [nop,nop,TS val 2630420235 ecr 1081024997], length 1368: HTTP
17:07:57.618785 IP 10.11.16.83.58526 > 10.2.62.2.80: Flags [P.], seq 8:10, ack 1, win 4104, options [nop,nop,TS val 1081025387 ecr 2630419956,nop,nop,sack 1 {4105:5442}], length 2: HTTP
17:07:57.618840 IP 10.2.62.2.80 > 10.11.16.83.58526: Flags [R], seq 93690138, win 0, length 0
Here is the tcpdump output from LinuxB:
17:07:52.853202 IP 10.11.16.83.58526 > 10.2.62.2.http: Flags [S], seq 3937114344, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 1081021219 ecr 0,sackOK,eol], length 0
17:07:52.879587 IP 10.2.62.2.http > 10.11.16.83.58526: Flags [S.], seq 1915917093, ack 3937114345, win 28960, options [mss 1380,sackOK,TS val 2630416160 ecr 1081021219,nop,wscale 7], length 0
17:07:52.879628 IP 10.11.16.83.58526 > 10.2.62.2.http: Flags [.], ack 1, win 4104, options [nop,nop,TS val 1081021245 ecr 2630416160], length 0
17:07:56.650275 IP 10.11.16.83.58526 > 10.2.62.2.http: Flags [P.], seq 1:8, ack 1, win 4104, options [nop,nop,TS val 1081024967 ecr 2630416160], length 7: HTTP: GET /
17:07:56.676571 IP 10.2.62.2.http > 10.11.16.83.58526: Flags [.], ack 8, win 227, options [nop,nop,TS val 2630419956 ecr 1081024967], length 0
17:07:56.681361 IP 10.2.62.2.http > 10.11.16.83.58526: Flags [P.], seq 4105:5442, ack 8, win 227, options [nop,nop,TS val 2630419961 ecr 1081024967], length 1337: HTTP
17:07:56.681363 IP 10.2.62.2.http > 10.11.16.83.58526: Flags [F.], seq 5442, ack 8, win 227, options [nop,nop,TS val 2630419961 ecr 1081024967], length 0
17:07:56.681399 IP 10.11.16.83.58526 > 10.2.62.2.http: Flags [.], ack 1, win 4104, options [nop,nop,TS val 1081024997 ecr 2630419956,nop,nop,sack 1 {4105:5442}], length 0
17:07:56.681408 IP 10.11.16.83.58526 > 10.2.62.2.http: Flags [.], ack 1, win 4104, options [nop,nop,TS val 1081024997 ecr 2630419956,nop,nop,sack 1 {4105:5442}], length 0
17:07:57.077574 IP 10.11.16.83.58526 > 10.2.62.2.http: Flags [P.], seq 8:10, ack 1, win 4104, options [nop,nop,TS val 1081025387 ecr 2630419956,nop,nop,sack 1 {4105:5442}], length 2: HTTP
17:07:57.103786 IP 10.2.62.2.http > 10.11.16.83.58526: Flags [R], seq 1915917094, win 0, length 0
At lease 3 packets from LinuxA to LinuxB are missing (two packets size is 1368 bytes, and one packet size 2736=2x1368 bytes). Packets are the following:
17:07:57.195669 IP 10.2.62.2.80 > 10.11.16.83.58526: Flags [.], seq 1:2737, ack 8, win 227, options [nop,nop,TS val 2630419961 ecr 1081024967], length 2736: HTTP
17:07:57.237384 IP 10.2.62.2.80 > 10.11.16.83.58526: Flags [.], seq 1:1369, ack 8, win 227, options [nop,nop,TS val 2630420003 ecr 1081024997], length 1368: HTTP
17:07:57.469410 IP 10.2.62.2.80 > 10.11.16.83.58526: Flags [.], seq 1:1369, ack 8, win 227, options [nop,nop,TS val 2630420235 ecr 1081024997], length 1368: HTTP
12-25-2018 10:00 AM
Hello,
typically, on GRE only (no IPSec) tunnels, MTU size would be set to 1476. Have you tried that value ?
interface Tunnel 3
ip mtu 1476
12-25-2018 12:32 PM
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide