Packet Loss via DMVPN Tunnel but Not Across WAN

[Michael Bruton]

Scenario:

Central Router (WAN: 1.1.1.1) <--> Internet <--> (WAN: Dynamic IP) Branch Router
Tunnel 172.31.254.1/26                                     Tunnel 172.31.254.9/26

Central router is a Cisco 1811 running IOS c181x-advipservicesk9-mz.151-4.M.bin.
Branch router is a Cisco 1941 running IOS c1900-universalk9-mz.SPA.151-4.M.bin.

When I do a Ping test directly from the branch to central router over the Internet I have no packet loss:

branch#ping 1.1.1.1 source GigabitEthernet 0/0 repeat 1000
Type escape sequence to abort.
Sending 1000, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
Packet sent with a source address of 192.168.0.100
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
(...)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (1000/1000), round-trip min/avg/max = 40/41/60 ms
branch#

When doing a Ping test over the DMVPN tunnel (which is using the WAN IP as source) I see packetloss.

branch#ping 172.31.254.1 source Tunnel 3 repeat 1000
Type escape sequence to abort.
Sending 1000, 100-byte ICMP Echos to 172.31.254.1, timeout is 2 seconds:
Packet sent with a source address of 172.31.254.9
!!!!!!!!!!.!!!!!!!!!!.!.!!!!!!.!!!!!..!!!!!!..!!!!!!!!.!!.!!!!!.!!!!!!
!!!!!!.!!!!!.!!!.!!!!!!!!!!!..!!!!.!.!.!!!!!.!!!!!!!!!.!..!!!.!.!!!!!.
(...)
!!!!!!.!!!.!!!!.!!!!.!.!!.!!!!!!!!!!!!!!!.!!.!!!!!!!!!.!!!.!!.!.!!!!!.
..!!!!!!!!!!..!!!!!!
Success rate is 79 percent (795/1000), round-trip min/avg/max = 40/43/568 ms
branch#

Central:

interface Tunnel0
description Testing (DMVPN)
bandwidth 10000
ip address 172.31.254.1 255.255.255.192
no ip redirects
ip mtu 1400
ip nhrp authentication testing
ip nhrp map multicast dynamic
ip nhrp network-id 1
ip nhrp holdtime 600
ip nhrp redirect
ip tcp adjust-mss 1360
no ip split-horizon eigrp 1
tunnel source FastEthernet1
tunnel mode gre multipoint
tunnel key 100003
tunnel bandwidth transmit 10000
tunnel bandwidth receive 10000
tunnel protection ipsec profile secure_profile shared

Branch:

interface Tunnel3
description Testing (DMVPN)
bandwidth 2000
ip address 172.31.254.9 255.255.255.192
no ip redirects
ip mtu 1400
ip nhrp authentication testing
ip nhrp map multicast 1.1.1.1
ip nhrp map 172.31.254.1 1.1.1.1
ip nhrp network-id 1
ip nhrp holdtime 300
ip nhrp nhs 172.31.254.1
ip nhrp shortcut
ip nhrp redirect
ip tcp adjust-mss 1360
no ip split-horizon eigrp 1
delay 1000
tunnel source GigabitEthernet0/0
tunnel mode gre multipoint
tunnel key 100003
tunnel bandwidth transmit 2000
tunnel bandwidth receive 2000
tunnel protection ipsec profile secure_profile shared

Crypto parameters on both central and branch routers:

crypto isakmp policy 1
authentication pre-share

crypto ipsec transform-set secure_transform-set esp-3des esp-sha-hmac
mode transport

crypto ipsec profile secure_profile
set transform-set secure_transform-set

I disabled CEF on both the central and branch routers and no success.  The EIGRP neighborship appears to be stable...  Been banging my head against the wall too long, any ideas?

[wzhang]

Hi,

First I would look at these common places where packet drops can often occur: 1) interface drops (show interface), 2) switching code/feature drops (show ip cef swtiching statistics feature), and 3) crypto drops (show crypto ipsec sa detail and show crypto engine accelerator stat). If you don't see any drops in any of these places, then it may get a little tricky to figuure out exactly where the drop is happening, but you want to start by narrowing down where (which device, hub or spoke) packets are dropped and in which direction (ingress or egress). To do that, you can set up a controlled test, where you can account for the exactly number of packets being sent. With your example, you can run the 1000 count ping test and set the dscp bits to a certain value so that you can uniquely identify them on the receiving end by using either ACL, ip precedence accounting, or EPC. If you send 1000 packets from the spoke, but only see 800 on the hub's physical interface, then you know the problem is either on the spoke itself or in the network. Do the same for the return direction and you should have a pretty good idea of which device you need to focus on troubleshooting from there.

Hope this helps,

Thanks,

Wen

[TCAM]

Encountered the similar problem, just curious, were you able to resolve it and what was the fix?

[roadrunner.travel]

Did this issue ever get resolved. I'm encountering something similar and I have narrowed down the packet drops to the ingress and egress SP.

[Javier Portuguez]

Please follow Wen's post and suggestions.

A packet-loss issue is not an easy thing to resolve, unless the correct troubleshooting takes place or at least more details are collected.

[aalex1055]

Hello, 

Same Issue here.

It is NOT the ISP, transport is OK.

Only traffic going through the DMVPN gets dropped. 

Has anyone managed to solve this?

[coreycomputer]

I have an issue as well where I have a remote site that is having packet drops and a very slow internet connection via a tunnel interface.  Does anyone know what this may be?

[amkatoking]

issues of asymmetric routing and mismatched MTU can cause some packet losses.