03-15-2013 08:13 PM - edited 02-21-2020 06:46 PM
Hi experts,
I'm trying to troubleshoot a random packet drop issue for an IPSec tunnel between two VTIs. For over a month, we didn't see any issue, and starting today, we have up to 30% packet loss across an IPSec tunnel.
After some analysis, I concluded that the packet loss happens somewhere on the path from the uc520 to the 2921. Packet counts show up correctly on the uc520 physical egress interface, but the packet count is low on the ingress interface on the 2921.
Pings outside the tunnel along the same path are fine.
I also cleared the tunnels on both ends and after they reestablished, the issue was still present.
Any pointers on finding where the packets get lost?
rr-hq-2921#ping 10.1.13.1 source g0/1 rep 100
Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 10.1.13.1, timeout is 2 seconds:
Packet sent with a source address of 10.1.1.1
!!..!.!!!!!!!!!..!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
..!!.!!!!!!!!!!!.!!!!!!!!.!!!!
Topology:
[uc520]==ONT=={{{cloud}}}==MODEM==[2921]
Test:
2921# clear counters g0/0
Clear "show interface" counters on this interface [confirm]
%CLEAR-5-COUNTERS: Clear counter on interface GigabitEthernet0/0
Execute on uc520: ping <inside interface IP of 2921> source <inside interface of uc520> timeout 0 rep 4000
This is supposed to rapidly increase the remote packet count by 4000 packets, like it did on the uc520 egress interface
2921# sho int g0/0 | i packets input
3348 packets input, 607812 bytes, 0 no buffer <<<<< MISSING ~650 packets
2921# sho int g0/0
GigabitEthernet0/0 is up, line protocol is up
Hardware is CN Gigabit Ethernet, address is XXXXXXXX
Description: OUTSIDE - WAN port
Internet address is XXX.XXX.XXX.XXX/YY
MTU 1500 bytes, BW 35000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full Duplex, 1Gbps, media type is RJ45
output flow-control is XON, input flow-control is XON
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:00:42
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 75000 bits/sec, 51 packets/sec
30 second output rate 77000 bits/sec, 52 packets/sec
3456 packets input, 619794 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
3454 packets output, 632194 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
Solved! Go to Solution.
03-16-2013 02:37 PM
Good infor
Now, have you asked your ISP if they have made any recent changes?
I think that your suspcious is correct and if the number of packets do not match, then probably something in the middle has changed, since it was working before with the same configuration and IOS versions.
HTH.
03-16-2013 02:37 PM
Good infor
Now, have you asked your ISP if they have made any recent changes?
I think that your suspcious is correct and if the number of packets do not match, then probably something in the middle has changed, since it was working before with the same configuration and IOS versions.
HTH.
03-16-2013 05:42 PM
Hi Javier,
Please explain to me how I should explain this technically elaborate issue to either ISP tech support? :-P
Well, I tried my best and ended up on the phone for 5 hours with 6 different techs between Verizon and TWC BC. I should get paid for explaining them the basics of networking.
Anyhow, my last desperate attempt was to ask the tech to reboot my ONT so I'd get a new IP. Maybe some traffic balancer or filter didn't like my source and destination IP combination. Maybe it was cursed.
Ring. Ring. I finally got an awesome tech (John) from Verizon who actually knew what he was talking about. I connected my Verizon supplied router again and asked if he could log into it or run pings from it remotely (to show him that I'm not crazy). Though other techs told me that was not possible, he did in just a few seconds without much pain. He saw the pings failing as well. Then he said pings from the Verizon ONT gateway were successful, so I assumed it must have been an issue somewhere in Verizon's neck of the (network) woods where the problem persisted.
Long story short: The new IP address worked like a charm and no more packet drops.
03-16-2013 10:06 PM
Way to go man!!
I know it is not easy... Well done, I give you 5 stars
Keep it up!!!
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: