DMVPN problem with 2 hubs

[george.tsanava]

hello

I have dmvpn phase 1, with 2 hubs, 20 spokes and eigrp, HUB1 is main and HUB2 is backup. if HUB1 is down all traffic from spokes goes to HUB2 immediately in seconds, but when HUB1 comes up traffic from spokes automatically goes back to HUB1 after 20-30 minutes and this is too long, this is problem.

"show dmvpn" command on spokes displays that tunnels to HUB1 are in nhrp state, and if I use "clear crypto session" command manually on any spoke traffic from this spoke immediately goes to HUB1.

one month ago i tested and everything was working fine. but when i last time tested it 2 days ago this problem occured.

what should be reason and how to fix it? 

sorry for my english, I am new to dmvpn :)

thanks in advance.

[Marcin Latosiewicz]

Look into NHRP holdtimes and registration timers. 

The default values are quite long.

[george.tsanava]

registration timer is 30 seconds, holdtime is 60 seconds, on all routers.

[re775 · Accepted solution]

Hi George,

I can think of two possible event that would explain the behavior you're experiencing.

a)  DMVPN state change.

b) Change in the routing table.

You can troubleshoot each of the above issue to identify which one is causing the problem, and then isolate it.  For starters, you should make sure that the DMVPN remain in a stable 'up' state.

You mention "pokes displays that tunnels to HUB1 are in nhrp state" --- This confirm DMVPN is 'stuck' and not fully operational. 

I can suggest that you review some useful troubleshooting details here:

http://www.cisco.com/c/en/us/support/docs/security/dynamic-multipoint-vpn-dmvpn/111976-dmvpn-troubleshoot-00.html#verifynhrpreg

Have a look at these details:

~~~

Interface: Tunnel100, IPv4 NHRP Details 
Type:Spoke, NHRP Peers:2, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1 192.168.1.1  172.28.1.1    UP    1d21h     S
     1 192.168.1.2  172.28.1.2    UP    1d21h     S

~~~

You will see similar output in your setup, so you would want to keep an eye on the "UpDn" time, as it will tell you how long the DMVPN has been up.

If the DMVPN remain stable while you're experiencing the problem, then focus on troubleshooting the routing protocol you're using across the dmvpn tunnel.

If the DMVPN is unstable, verify connectivity between the spoke and hub NBMA address, and ensure connectivity remain stable.  you can use  'debug dmvpn error crypto and debug dmvpn error nhrp" to help identifying the problem if it is DMVPN related.

There are a lot of assumption in my suggestions, because you have not posted the configuration :).

But it would help if you post the config.  Best of luck with your efforts.

Thanks

re775

[george.tsanava]

hi re775

it isn't problem of routing because as i remember dmvpn was down and unstable, yesterday i added on all routers next commands: 

crypto isakmp keepalive 10 3 periodic
crypto isakmp invalid-spi-recovery

and tested, everything was fine :) 

i'm planning to test again in next week and if same problem occurs, now i know on what to focus & i'll try to find real reason :)

thanks for your respond it is very helpful. 

[george.tsanava]

hello 

I have tested today and only one spoke was late :)

i have debug nhrp packet output from SPOKE:

NHRP: Send Registration Request via Tunnel20 vrf 0, packet size: 108

(F) afn: IPv4(1), type: IP(800), hop: 255, ver: 1

shtl: 4(NSAP), sstl: 0(NSAP)

pktsz: 108 extoff: 52

(M) flags: "unique nat ", reqid: 91209 

 src NBMA: X.X.X.X

src protocol: 10.150.150.126, dst protocol: 10.150.150.1

(C-1) code: no error(0)

prefix: 32, mtu: 17912, hd_time: 60

addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 0

NHRP: Send Registration Request via Tunnel50 vrf 0, packet size: 108

NHRP: Receive Registration Reply via Tunnel50 vrf 0, packet size: 128

NHRP: Send Registration Request via Tunnel20 vrf 0, packet size: 108

NHRP: Send Registration Request via Tunnel50 vrf 0, packet size: 108

NHRP: Receive Registration Reply via Tunnel50 vrf 0, packet size: 128

as it looks like there is no Registration Reply Received on tunnel 20

tunnel 20 is tunnel to MAIN-HUB and tunnel 50 is tunnel to BACKUP-HUB

at the same time on MAIN-HUB there was several this type logs: 

%CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr= Z.Z.Z.Z , prot=50, spi=0x1FD5979E(534091678), srcaddr=X.X.X.X, input interface=GigabitEthernet0/1.582
%CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed
        connection id=5165, sequence number=56718

I'm hopefully waiting for your response about how to resolve this,

many thanks :)

[re775 · Accepted solution]

Hi 

From my experience, where I have encountered the error "%CRYPTO-4-RECVD_PKT_INV_SPI", it was due to "nhrp" stuck in IKE state.

This error i encounter after 'flipping' traffic from hub-a to hub-b and/or with a primary hub is 'no-longer' available for any reason, and the traffic is sent to the backup hub.

My suggestion is do a "clear crypto sa peer x.x.x.x" on the spoke, and that would put the spoke into a "UP" nhrp state, if all other configs are OK.

​For reference:  http://www.cisco.com/c/en/us/support/docs/security-vpn/ipsec-negotiation-ike-protocols/115801-technote-iosvpn-00.html

The above information is my humble information based on my experience with this error message, and given the fact I have very limited with the config and topology, and traffic details, the information provide is 'as is' :).

Best of luck.