Dual DMVPN - EIGRP Routing Problem with 2 ISPs

Joshua Ashley · ‎02-24-2011

All,

Right now I have a DMVPN environment with a single hub and roughly 20 spokes. I have been tasked out to create a redundant DMVPN tunnel using our secondary ISP with our existing hub to all of our existing sites (please view attachment for the topology and configs). It seems that both tunnels come up because when I do a "sh crypto isakmp sa" I see both tunnels (see below); however, when I do a "show ip eigrp neighbor" I only see the original tunnel NOT the second redundant tunnel that I have built. I am under the impression that 2 different adjacency will form, 1 per tunnel. Also when I shut tunnel0 or the original tunnel down there is NO failover. The secondary tunnel state is "QM_IDLE" and its status is "ACTIVE" I am going to paste my configs here as well as in the attached diagram. ANY help would be greatly appreciated.

*************************************HUB******************************************

crypto isakmp policy 1
encr aes
authentication pre-share
group 2
!
crypto isakmp policy 10
encr 3des
authentication pre-share
group 2
crypto isakmp key XXXXXXXXXXXXXXXXX address 0.0.0.0 0.0.0.0
crypto isakmp invalid-spi-recovery
crypto isakmp keepalive 10
crypto isakmp nat keepalive 100

!

crypto ipsec transform-set test esp-aes esp-sha-hmac
crypto ipsec transform-set test1 esp-3des esp-sha-hmac
!
crypto ipsec profile dhaprof
set transform-set test
!
crypto ipsec profile dhaprof1
set transform-set test1
!
interface Loopback0
ip address 10.0.102.1 255.255.255.255
!
interface Tunnel0
description - DMVPN Tunnel Interface - Hub configuration
bandwidth 1544
ip address 10.0.100.1 255.255.255.0
no ip redirects
ip mtu 1400
no ip next-hop-self eigrp 100
no ip next-hop-self eigrp 1
ip nhrp authentication 12345
ip nhrp map multicast dynamic
ip nhrp network-id 100
ip nhrp holdtime 300
ip virtual-reassembly
ip tcp adjust-mss 1360
no ip split-horizon eigrp 100
no ip mroute-cache
load-interval 60
delay 400
keepalive 5 4
tunnel source GigabitEthernet0/0
tunnel mode gre multipoint
tunnel key XYZ
tunnel protection ipsec profile dhaprof1 shared
!
interface Tunnel1
description - BACKUP DMVPN Tunnel Interface - TELEPAK
ip address 10.0.101.1 255.255.255.0
no ip redirects
ip mtu 1400
no ip next-hop-self eigrp 100
no ip next-hop-self eigrp 1
ip nhrp authentication 12345
ip nhrp map multicast dynamic
ip nhrp network-id 101
ip nhrp holdtime 300
ip virtual-reassembly
ip tcp adjust-mss 1360
no ip split-horizon eigrp 100
no ip mroute-cache
load-interval 60
delay 500
keepalive 5 4
tunnel source FastEthernet0/1/0
tunnel mode gre multipoint
tunnel key ABC
tunnel protection ipsec profile dhaprof1 shared
!
router eigrp 100
network 10.0.100.0 0.0.0.255 !!Tunnel0
network 10.0.101.0 0.0.0.255 !!Tunnel1
network 10.0.102.1 0.0.0.0 !!Loopack0
no auto-summary
!
*************************************SPOKE1****************************************
crypto isakmp policy 1
encr aes
authentication pre-share
group 2
!
crypto isakmp policy 10
encr 3des
crypto isakmp key XXXXXXXXXXXXXXX address 0.0.0.0 0.0.0.0
crypto isakmp invalid-spi-recovery
crypto isakmp keepalive 10
!
!
crypto ipsec transform-set test esp-aes esp-sha-hmac
crypto ipsec transform-set test1 esp-3des esp-sha-hmac
!
crypto ipsec profile dhaprof
set transform-set test
!
crypto ipsec profile dhaprof1
set transform-set test1
!
interface Tunnel0
description - Connection to AllScripts
bandwidth 1544
ip address 10.0.100.100 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp authentication 12345
ip nhrp map multicast 68.153.126.167
ip nhrp map 10.0.100.1 68.153.126.167
ip nhrp network-id 100
ip nhrp holdtime 300
ip nhrp nhs 10.0.100.1
ip virtual-reassembly
ip tcp adjust-mss 1360
no ip mroute-cache
load-interval 60
delay 400
keepalive 5 4
tunnel source FastEthernet4
tunnel mode gre multipoint
tunnel key XYZ
tunnel protection ipsec profile dhaprof1 shared
!
interface Tunnel1
description - CONNECTION TO TELEPAK BACKUP DMVPN ROUTER
bandwidth 1000
ip address 10.0.101.100 255.255.255.0
no ip redirects
ip nhrp authentication 12345
ip nhrp map multicast 76.8.245.254
ip nhrp map 10.0.101.1 76.8.245.254
ip nhrp network-id 101
ip nhrp holdtime 300
ip nhrp nhs 10.0.101.1
ip virtual-reassembly
ip tcp adjust-mss 1360
no ip mroute-cache
load-interval 60
delay 500
keepalive 5 4
tunnel source FastEthernet4
tunnel mode gre multipoint
tunnel key ABC
tunnel protection ipsec profile dhaprof1 shared
!
interface FastEthernet4
ip address dhcp
ip nat outside
ip virtual-reassembly
duplex auto
speed auto
!
router eigrp 100
network 10.0.100.0 0.0.0.255 !!Tunnel0
network 10.0.101.0 0.0.0.255 !!Tunnel1
network 10.254.253.0 0.0.0.255 !!LAN
no auto-summary

*******************************************************SPOKE1 Troubleshooting*******************************

SPOKE1#sh ip nhrp detail

10.0.100.1/32 via 10.0.100.1
   Tunnel0 created 00:34:34, never expire
   Type: static, Flags: used
   NBMA address: 68.153.126.167
10.0.101.1/32 via 10.0.101.1
   Tunnel1 created 00:58:24, never expire
   Type: static, Flags: used
   NBMA address: 76.8.245.254

SPOKE1#sh crypto isakmp sa

IPv4 Crypto ISAKMP SA
dst             src             state          conn-id slot status
76.8.245.254    192.168.1.20    QM_IDLE           2001    0 ACTIVE
68.153.126.167 192.168.1.20    QM_IDLE           2002    0 ACTIVE

IPv6 Crypto ISAKMP SA

SPOKE1#sh ip eigrp neighbors detail

IP-EIGRP neighbors for process 100
H   Address                 Interface       Hold Uptime   SRTT   RTO Q Seq
                                            (sec)         (ms)       Cnt Num
0   10.0.100.1              Tu0               10 00:37:55   40   360 0 3693
   Version 12.4/1.2, Retrans: 5, Retries: 0, Prefixes: 40

Kasiraman S · ‎02-24-2011

Joshua,

Is that just a issue with EIGRP adjacency or even not able to reach the neighbor itself. Can you try reach the tunnel neighbor through ICMP and let me know. Also please provide the show crypto ipsec sa peer A.B.C.D output from both Hub and Spoke.

Thanks,

Kasi

Joshua Ashley · ‎02-25-2011

Kasiraman,

Thanks for the quick reply. I can ping the IPs between the orginal Tunnel (Tunnel0), but not on the new redundant tunnel (Tunnel1). I think its an EIGRP problem since everything looks like the tunnel is up, but just not passing traffic. Here are the ping and the "show crypto isakmp peer A.B.C.D" results.

**********************************************************************SPOKE1*******************************************************************************************

SPOKE1#ping 10.0.100.1 !!Tunnel0

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.100.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/15/16 ms

Wireless_Test#ping 10.0.101.1 !!Tunnel1

Type escape sequence to abort

Sending 5, 100-byte ICMP Echos to 10.0.101.1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

SPOKE1#sh crypto isakmp peers 68.153.126.167
Peer: 68.153.126.167 Port: 4500 Local: 192.168.1.20
Phase1 id: 68.153.126.167

SPOKE1#sh crypto isakmp peers 76.8.245.254
Peer: 76.8.245.254 Port: 4500 Local: 192.168.1.20
Phase1 id: 76.8.245.254

*********************************************************************HUB****************************************************************************************************

DMVPNHUB#ping 10.0.100.100 !!!!Tunnel0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.100.100, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 16/16/20 ms

DMVPNHUB#ping 10.0.101.100 !!Tunnel1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.101.100, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

DMVPNHUB#sh crypto isakmp peers 74.240.29.102
Peer: 74.240.29.102 Port: 54689 Local: 68.153.126.167
Phase1 id: 192.168.1.20
Peer: 74.240.29.102 Port: 54399 Local: 76.8.245.254
Phase1 id: 192.168.1.20

Again thanks for the quick response and any help would be greatly appreciated.

Kasiraman S · ‎02-25-2011

Hi,

Looks like issue with the DMVPN itself because you are not able to reach the next hop IP itself. We see the ISAKMP is up, but check if its passing the data traffic end to end or not to make sure IPSEC is good.

Hit "show crypto ipsec sa" on both the end and see if you able to see both encaps and decaps are incrementing, not needed to be same but should increment in both like below one.
__________________________
Router#show crypto IPSec sa - GOOD SIGN
local ident (addr/mask/prot/port): (172.16.1.1/255.255.255.255/47/0)
remote ident (addr/mask/prot/port): (172.17.0.1/255.255.255.255/47/0)
#pkts encaps: 121, #pkts encrypt: 121, #pkts digest: 121
#pkts decaps: 118, #pkts decrypt: 118, #pkts verify: 118
__________________________
Router#show crypto IPSEC sa - BAD SIGN
local ident (addr/mask/prot/port): (172.16.1.1/255.255.255.255/47/0)
remote ident (addr/mask/prot/port): (172.17.0.1/255.255.255.255/47/0)
#pkts encaps: 154, #pkts encrypt: 154, #pkts digest: 154
#pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
__________________________

2) Also check in both end to see if the ACL hits are incrementing or not. ACL hits count increment in the initiating end but go to the other end and see if the Hit count is incrementing or not.

3) show ip nhrp nhs detail and see if failed count is increasing or not. if we see any problem here we have to recheck the config again.

Thanks,

Kasi

Joshua Ashley · ‎02-25-2011

It does look like I might have an IPSec issue. I have checked my settings.. everything looks to be in order.. how can I farther troubleshoot this issue?? Here is the output

SPOKE1t#sh crypto ipsec sa

interface: Tunnel1
Crypto map tag: dhaprof1-head-1, local addr 192.168.1.20

   protected vrf: (none)
   local ident (addr/mask/prot/port): (192.168.1.20/255.255.255.255/47/0)
   remote ident (addr/mask/prot/port): (76.8.245.254/255.255.255.255/47/0)
   current_peer 76.8.245.254 port 500
     PERMIT, flags={origin_is_acl,ipsec_sa_request_sent}
    #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
    #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 21018, #recv errors 0

     local crypto endpt.: 192.168.1.20, remote crypto endpt.: 76.8.245.254
     path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet4
     current outbound spi: 0x0(0)

inbound esp sas:

     inbound ah sas:

     inbound pcp sas:

outbound esp sas:

outbound ah sas:

outbound pcp sas:

   protected vrf: (none)
   local ident (addr/mask/prot/port): (192.168.1.20/255.255.255.255/47/0)
   remote ident (addr/mask/prot/port): (68.153.126.167/255.255.255.255/47/0)
   current_peer 68.153.126.167 port 4500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 18584, #pkts encrypt: 18584, #pkts digest: 18584
    #pkts decaps: 18614, #pkts decrypt: 18614, #pkts verify: 18614
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 1, #recv errors 0

     local crypto endpt.: 192.168.1.20, remote crypto endpt.: 68.153.126.167
     path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet4
     current outbound spi: 0x37D34B8A(936594314)

     inbound esp sas:
      spi: 0x3FE7BDC6(1072152006)
        transform: esp-3des esp-sha-hmac ,
        in use settings ={Tunnel UDP-Encaps, }
        conn id: 51, flow_id: Motorola SEC 2.0:51, crypto map: dhaprof1-head-1
        sa timing: remaining key lifetime (k/sec): (4407017/1270)
        IV size: 8 bytes
        replay detection support: Y
        Status: ACTIVE

inbound ah sas:

inbound pcp sas:

     outbound esp sas:
      spi: 0x37D34B8A(936594314)
        transform: esp-3des esp-sha-hmac ,
        in use settings ={Tunnel UDP-Encaps, }
        conn id: 52, flow_id: Motorola SEC 2.0:52, crypto map: dhaprof1-head-1
        sa timing: remaining key lifetime (k/sec): (4407098/1270)
        IV size: 8 bytes
        replay detection support: Y
        Status: ACTIVE

outbound ah sas:

outbound pcp sas:

interface: Tunnel0
Crypto map tag: dhaprof1-head-1, local addr 192.168.1.20

   protected vrf: (none)
   local ident (addr/mask/prot/port): (192.168.1.20/255.255.255.255/47/0)
   remote ident (addr/mask/prot/port): (76.8.245.254/255.255.255.255/47/0)
   current_peer 76.8.245.254 port 500
     PERMIT, flags={origin_is_acl,ipsec_sa_request_sent}
    #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
    #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 21018, #recv errors 0

     local crypto endpt.: 192.168.1.20, remote crypto endpt.: 76.8.245.254
     path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet4
     current outbound spi: 0x0(0)

inbound esp sas:

inbound ah sas:

inbound pcp sas:

outbound esp sas:

outbound ah sas:

outbound pcp sas:

   protected vrf: (none)
   local ident (addr/mask/prot/port): (192.168.1.20/255.255.255.255/47/0)
   remote ident (addr/mask/prot/port): (68.153.126.167/255.255.255.255/47/0)
   current_peer 68.153.126.167 port 4500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 18584, #pkts encrypt: 18584, #pkts digest: 18584
    #pkts decaps: 18614, #pkts decrypt: 18614, #pkts verify: 18614
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 1, #recv errors 0

     local crypto endpt.: 192.168.1.20, remote crypto endpt.: 68.153.126.167
     path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet4
     current outbound spi: 0x37D34B8A(936594314)

     inbound esp sas:
      spi: 0x3FE7BDC6(1072152006)
        transform: esp-3des esp-sha-hmac ,
        in use settings ={Tunnel UDP-Encaps, }
        conn id: 51, flow_id: Motorola SEC 2.0:51, crypto map: dhaprof1-head-1
        sa timing: remaining key lifetime (k/sec): (4407017/1270)
        IV size: 8 bytes
        replay detection support: Y
        Status: ACTIVE

inbound ah sas:

inbound pcp sas:

     outbound esp sas:
      spi: 0x37D34B8A(936594314)
        transform: esp-3des esp-sha-hmac ,
        in use settings ={Tunnel UDP-Encaps, }
        conn id: 52, flow_id: Motorola SEC 2.0:52, crypto map: dhaprof1-head-1
        sa timing: remaining key lifetime (k/sec): (4407098/1270)
        IV size: 8 bytes
        replay detection support: Y
        Status: ACTIVE

outbound ah sas:

outbound pcp sas:

Kishore Chennupati · ‎02-25-2011

hi Joshua,

SPOKE1t#sh crypto ipsec sa

interface: Tunnel1
Crypto map tag: dhaprof1-head-1, local addr 192.168.1.20

   protected vrf: (none)
   local ident (addr/mask/prot/port): (192.168.1.20/255.255.255.255/47/0)
   remote ident (addr/mask/prot/port): (76.8.245.254/255.255.255.255/47/0)
   current_peer 76.8.245.254 port 500
     PERMIT, flags={origin_is_acl,ipsec_sa_request_sent}
    #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
    #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 21018, #recv errors 0

Can you please put the "ip mtu 1400 " on the Tunnel1 at the Spoke site.

HTH,

Regards,

Please rate if helpful

Joshua Ashley · ‎03-02-2011

Kishore,

Here is the output after adding "ip myu 1400" to the spoke:

************************************************************************************

SPOKE1#sh crypto ipsec sa

interface: Tunnel1
Crypto map tag: dhaprof1-head-1, local addr 192.168.1.20

   protected vrf: (none)
   local ident (addr/mask/prot/port): (192.168.1.20/255.255.255.255/47/0)
   remote ident (addr/mask/prot/port): (76.8.245.254/255.255.255.255/47/0)
   current_peer 76.8.245.254 port 500
     PERMIT, flags={origin_is_acl,ipsec_sa_request_sent}
    #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
    #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 40516, #recv errors 0

Looks like I am still having errors.. any other troubleshoting steps you can think of??

josh

Kishore Chennupati · ‎03-02-2011

Hi Joshua,

Sorry, I only gave you the pointers in my prev post and didn't quite explain.

From your SPOKE1#sh crypto ipsec sa , looks like your IPsec is not established at all. If you look SPOKE1t#sh crypto ipsec sa there are no spi's created . You can compare the output to the connected one and not connected one.

Also you are doign NAT on your spoke site. You need to permit the NAT-T which is UDP 4500 port. Do you have anything that is firewalling this port for 76.8.245.254. Did this ever work? or you have just added this second tunnel as backup and hasnt worked?

===Established

protected vrf: (none)
   local ident (addr/mask/prot/port): (192.168.1.20/255.255.255.255/47/0)
   remote ident (addr/mask/prot/port): (68.153.126.167/255.255.255.255/47/0)
   current_peer 68.153.126.167 port 4500

==Not Established

protected vrf: (none)
local ident (addr/mask/prot/port): (192.168.1.20/255.255.255.255/47/0)
remote ident (addr/mask/prot/port): (76.8.245.254/255.255.255.255/47/0)
current_peer 76.8.245.254 port 500

HTH,

Regards,

Please rate if helpful