We are running a DMVPN dual hub and spoke configuration using ASR router for the hubs and 2811 routers for the spoke routers. We have recently gone past 3000 remote locations and have discovered an issue that we are struggling with. On the some spoke routers (we don't know for sure how many) we are seeing that the show dmvpn in some cases responds with IKE or NHRP with one of the hub peers (see below)
ro1-13349#sho dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
N - NATed, L - Local, X - No Socket
# Ent --> Number of NHRP entries with same NBMA peer
NHS Status: E --> Expecting Replies, R --> Responding
UpDn Time --> Up or Down Time for a Tunnel
==========================================================================
Interface: Tunnel1, IPv4 NHRP Details
IPv4 Registration Timer: 30 seconds
IPv4 NHS: 10.1.0.1 RE
Type:Spoke, Total NBMA Peers (v4/v6): 1
# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb Target Network
----- --------------- --------------- ----- -------- ----- -----------------
1 A.B.C.D 10.1.0.1 UP 6d14h S 10.1.0.1/32
Interface: Tunnel2, IPv4 NHRP Details
IPv4 Registration Timer: 30 seconds
IPv4 NHS: 10.2.0.1 E
Type:Spoke, Total NBMA Peers (v4/v6): 1
# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb Target Network
----- --------------- --------------- ----- -------- ----- -----------------
1 A.B.C.D 10.2.0.1 IKE 3w6d S 10.2.0.1/32
The state will cycle between IKE and NHRP and UP. We have captured the data 3 times from our 3000+ connections and we have seen about 15 to 20 on each data capture with 1 location that has been on each list.
Is there any additional logging that could help determine the cause? We recently added dmvpn logging on 32 branches and the typical message we see is the following
Apr 4 10:34:29.619 CDT: %DMVPN-5-NHRP_NHS: Tunnel2 10.2.0.1 is DOWN
Apr 4 10:35:53.048 CDT: %DMVPN-3-NHRP_ERROR: Registration Request failed for 10.2.0.1 on Tunnel2
In some cases we get the following
Apr 4 14:28:40.558 CDT: %DMVPN-7-CRYPTO_SS: Tunnel2-A.B.C.D socket is DOWN
Clearing crypto sessions or a shut no shut on the tunnel rarely fixes the problem. If it does the issue comes back. We are using a mix of pre-shared key and CA crypto authenticaion. We are using Version 12.4(24)T1 as the IOS based on other issues.
Please provide any insight you may have on this type of issue. I will add more as we uncover more information or have any pertinent data to add.
