dmvpn tunnel packet drop

[peter.zhu]

we have situation in one of spoke site, it face packet drop with dmvpn hub under bgp, but no drop to hub physical interface. to me, the physical link is fine as no drop to hub physical interface. Can anyone help understand what cause packet drop in the tunnel ?

[marce1000]

 - Check this thread for hints :

               https://community.cisco.com/t5/vpn/packet-loss-via-dmvpn-tunnel-but-not-across-wan/td-p/1683925

 M.

[Georg Pauwen]

Hello,

hard to tell, can you post the tunnel config ? Make sure the mtu is set to 1400 (ip mtu 1400) and the 'ip tcp adjust-mss 1360' value is set on your tunnel. 

Configuring 'ip tcp mtu-path-discovery' on the tunnel may help as well.

[Joseph W. Doherty]

Do I understand, correctly, between a hub and a spoke, you "see" drops when using DMVPN but none when not?

If so, please explain "where" you see the "drops" and how you compare DMVPN between hub and spoke vs. when not using DMVPN?  (Reason for the last question, often outside of DMVPN, you might do ping tests between hub and spoke, while within a tunnel, besides doing ping tests, tunnel might be carrying traffic.)

It may be help if you would further describe both the physical and logical components.

[peter.zhu]

Hi Joseph, when we do lan to lan ping test between hub and spoke, the packet drop occurred, when do ping the WAN interface, no drop. However, we found out the packet drop happened in one of hop on the path. This is quite interested that no drop between end to end interface ping, but drop in one of hop. Now, we swing the traffic to another hub as workaround, no more packet drop in the tunnel.

twkhsxr01#trace 159.12.192.227 so gi0/0/1
Type escape sequence to abort.
Tracing the route to 159.12.192.227
VRF info: (vrf in name/id, vrf out name/id)
  1 122.147.199.161 [AS 65003] 0 msec 0 msec 4 msec
  2 220.228.21.157 [AS 65003] 0 msec 0 msec
    220.228.21.153 [AS 65003] 4 msec
  3 220.228.23.141 [AS 65003] 8 msec
    220.228.23.145 [AS 65003] 8 msec 8 msec
  4 192.72.107.129 [AS 65003] 8 msec
    192.72.107.97 [AS 65003] 8 msec
    192.72.107.189 [AS 65003] 8 msec
  5 139.175.58.145 [AS 65003] 8 msec 8 msec 4 msec
  6 192.72.155.150 [AS 65003] 8 msec 8 msec 8 msec
  7 192.72.250.10 [AS 65003] 8 msec 8 msec 8 msec  not allowed ping after from hop 7
  8 103.123.252.133 [AS 65003] 8 msec
    103.123.253.133 [AS 65003] 8 msec 8 msec
  9 223.26.64.137 [AS 65003] 8 msec 8 msec
    223.26.64.141 [AS 65003] 8 msec
 10 113.21.95.23 [AS 65003] 12 msec 8 msec 8 msec
 11 113.21.84.98 [AS 65003] 8 msec 8 msec 4 msec
 12 140.222.4.177 [AS 65003] 64 msec 64 msec 68 msec
 13 202.95.64.170 [AS 65003] 68 msec 68 msec 64 msec    <<<<<<<<< packet drop in hop 13
 14 159.12.192.3 [AS 65003] 68 msec 64 msec 68 msec

end to end no drop

twkhsxr01#ping 159.12.192.227 rep 1000
Type escape sequence to abort.
Sending 1000, 100-byte ICMP Echos to 159.12.192.227, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (1000/1000), round-trip min/avg/max = 56/58/68 ms

packet drop in hop 13
twkhsxr01#ping 202.95.64.170  rep 100
Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 202.95.64.170, timeout is 2 seconds:
Packet sent with a source address of 10.111.20.1
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!.!!!!!!!!
!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!
Success rate is 97 percent (97/100), round-trip min/avg/max = 68/69/72 ms

[Georg Pauwen]

Hello,

when I ping that IP address with different packet sizes, I get weird results. The last successful ping is with a packet size of 1442, anything bigger is dropped. Does the path to the other hub go through that hop (202.95.64.170) as well ?

C:\Users\pauwe>ping -l 1442 -f 202.95.64.170

Pinging 202.95.64.170 with 1442 bytes of data:
Reply from 202.95.64.170: bytes=1442 time=348ms TTL=240
Reply from 202.95.64.170: bytes=1442 time=345ms TTL=240
Reply from 202.95.64.170: bytes=1442 time=347ms TTL=240
Reply from 202.95.64.170: bytes=1442 time=346ms TTL=240

Ping statistics for 202.95.64.170:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 345ms, Maximum = 348ms, Average = 346ms

C:\Users\pauwe>ping -l 1444 -f 202.95.64.170

Pinging 202.95.64.170 with 1444 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for 202.95.64.170:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

[peter.zhu]

Yes Geroge, this hop is the on the path from other spoke site. However, no packet dorp observed from other site. This is indeed weird.

[Joseph W. Doherty]

Possibly an ISP is having hardware issues or congestion on that one interface, in one direction.  I've seen it happen (rarely) before.

Might be worthwhile to reach out to the provider that has that hop and bring this to their attention.