IPSec Tunnel Random Packet Drops

[roadrunner.travel]

Hi experts,

I'm trying to troubleshoot a random packet drop issue for an IPSec tunnel between two VTIs. For over a month, we didn't see any issue, and starting today, we have up to 30% packet loss across an IPSec tunnel.

After some analysis, I concluded that the packet loss happens somewhere on the path from the uc520 to the 2921. Packet counts show up correctly on the uc520 physical egress interface, but the packet count is low on the ingress interface on the 2921.

Pings outside the tunnel along the same path are fine.

I also cleared the tunnels on both ends and after they reestablished, the issue was still present.

Any pointers on finding where the packets get lost?

rr-hq-2921#ping 10.1.13.1 source g0/1 rep 100         

Type escape sequence to abort.

Sending 100, 100-byte ICMP Echos to 10.1.13.1, timeout is 2 seconds:

Packet sent with a source address of 10.1.1.1

!!..!.!!!!!!!!!..!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

..!!.!!!!!!!!!!!.!!!!!!!!.!!!!

Topology:

[uc520]==ONT=={{{cloud}}}==MODEM==[2921]

Test:

2921# clear counters g0/0          

Clear "show interface" counters on this interface [confirm]

%CLEAR-5-COUNTERS: Clear counter on interface GigabitEthernet0/0

Execute on uc520: ping <inside interface IP of 2921> source <inside interface of uc520> timeout 0 rep 4000

This is supposed to rapidly increase the remote packet count by 4000 packets, like it did on the uc520 egress interface

2921# sho int g0/0 | i packets input

     3348 packets input, 607812 bytes, 0 no buffer  <<<<< MISSING ~650 packets

2921# sho int g0/0                 

GigabitEthernet0/0 is up, line protocol is up

  Hardware is CN Gigabit Ethernet, address is XXXXXXXX

  Description: OUTSIDE - WAN port

  Internet address is XXX.XXX.XXX.XXX/YY

  MTU 1500 bytes, BW 35000 Kbit/sec, DLY 10 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full Duplex, 1Gbps, media type is RJ45

  output flow-control is XON, input flow-control is XON

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 00:00:00, output 00:00:00, output hang never

  Last clearing of "show interface" counters 00:00:42

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  30 second input rate 75000 bits/sec, 51 packets/sec

  30 second output rate 77000 bits/sec, 52 packets/sec

     3456 packets input, 619794 bytes, 0 no buffer

     Received 0 broadcasts (0 IP multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 0 multicast, 0 pause input

     3454 packets output, 632194 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 unknown protocol drops

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 pause output

     0 output buffer failures, 0 output buffers swapped out

[Javier Portuguez · Accepted solution]

Good infor

Now, have you asked your ISP if they have made any recent changes?

I think that your suspcious is correct and if the number of packets do not match, then probably something in the middle has changed, since it was working before with the same configuration and IOS versions.

HTH.

[roadrunner.travel]

Hi Javier,

Please explain to me how I should explain this technically elaborate issue to either ISP tech support? :-P

Well, I tried my best and ended up on the phone for 5 hours with 6 different techs between Verizon and TWC BC. I should get paid for explaining them the basics of networking.

Anyhow, my last desperate attempt was to ask the tech to reboot my ONT so I'd get a new IP. Maybe some traffic balancer or filter didn't like my source and destination IP combination. Maybe it was cursed.

Ring. Ring. I finally got an awesome tech (John) from Verizon who actually knew what he was talking about. I connected my Verizon supplied router again and asked if he could log into it or run pings from it remotely (to show him that I'm not crazy). Though other techs told me that was not possible, he did in just a few seconds without much pain. He saw the pings failing as well. Then he said pings from the Verizon ONT gateway were successful, so I assumed it must have been an issue somewhere in Verizon's neck of the (network) woods where the problem persisted.

Long story short: The new IP address worked like a charm and no more packet drops.

[Javier Portuguez]

Way to go man!!

I know it is not easy... Well done, I give you 5 stars

Keep it up!!!