Hi community,
We have a client with 50 DMVPN spokes, connected with 2 Hubs. Everything was working fine and suddenly the Tunnel adjency started flapping with all spokes expect 1:
%DUAL-5-NBRCHANGE: EIGRP-IPv4 600: Neighbor x.x.x.x (TunnelX) is up: new adjacency
%DUAL-5-NBRCHANGE: EIGRP-IPv4 600: Neighbor x.x.x.x (TunnelX) is down: holding time expired
Below you can find the debugs on both HUB and SPOKE:
SPOKE: DEBUG DMVPN DETAIL ALL
Jan 16 10:20:04.431: ISAKMP: (1796):Sending an IKE IPv4 Packet.
Jan 16 10:20:04.431: ISAKMP: (1796):purging node 1271632275
Jan 16 10:20:04.431: ISAKMP: (1796):Input = IKE_MESG_FROM_PEER, IKE_MESG_KEEP_ALIVE
Jan 16 10:20:04.431: ISAKMP: (1796):Old State = IKE_P1_COMPLETE New State = IKE_P1_COMPLETE
Jan 16 10:20:04.847: ISAKMP: (0):retransmitting phase 1 MM_NO_STATE...
Jan 16 10:20:04.847: ISAKMP: (0):: incrementing error counter on sa, attempt 2 of 5: retransmit phase 1
Jan 16 10:20:04.847: ISAKMP: (0):retransmitting phase 1 MM_NO_STATE
Jan 16 10:20:04.847: ISAKMP-PAK: (0):sending packet to x.x.x.x my_port 500 peer_port 500 (I) MM_NO_STATE
Jan 16 10:20:04.847: ISAKMP: (0):Sending an IKE IPv4 Packet.
Jan 16 10:20:04.847: ISAKMP: (0):retransmitting phase 1 MM_NO_STATE...
Jan 16 10:20:04.847: ISAKMP: (0):: incrementing error counter on sa, attempt 2 of 5: retransmit phase 1
Jan 16 10:20:04.847: ISAKMP: (0):retransmitting phase 1 MM_NO_STATE
Jan 16 10:20:04.847: ISAKMP-PAK: (0):sending packet to x.x.x.x my_port 500 peer_port 500 (I) MM_NO_STATE
Jan 16 10:20:04.847: ISAKMP: (0):Sending an IKE IPv4 Packet.
Jan 16 10:20:04.847: ISAKMP: (0):retransmitting phase 1 MM_NO_STATE...
Jan 16 10:20:04.847: ISAKMP: (0):: incrementing error counter on sa, attempt 2 of 5: retransmit phase 1
Jan 16 10:20:04.847: ISAKMP: (0):retransmitting phase 1 MM_NO_STATE
Jan 16 10:20:04.847: ISAKMP-PAK: (0):sending packet to x.x.x.x my_port 500 peer_port 500 (I) MM_NO_STATE
Jan 16 10:20:04.847: ISAKMP: (0):Sending an IKE IPv4 Packet.
Jan 16 10:20:04.851: ISAKMP: (0):retransmitting phase 1 MM_NO_STATE...
Jan 16 10:20:04.851: ISAKMP: (0):: incrementing error counter on sa, attempt 2 of 5: retransmit phase 1
Jan 16 10:20:04.851: ISAKMP: (0):retransmitting phase 1 MM_NO_STATE
Jan 16 10:20:04.851: ISAKMP-PAK: (0):sending packet to x.x.x.x my_port 500 peer_port 500 (I) MM_NO_STATE
Jan 16 10:20:04.851: ISAKMP: (0):Sending an IKE IPv4 Packet.
Jan 16 10:20:05.403: ISAKMP: (1778):set new node 436367964 to QM_IDLE
Jan 16 10:20:05.403: ISAKMP-PAK: (1778):sending packet to x.x.x.x my_port 4500 peer_port 4500 (I) QM_IDLE
Jan 16 10:20:05.403: ISAKMP: (1778):Sending an IKE IPv4 Packet.
Jan 16 10:20:05.403: ISAKMP: (1778):purging node 436367964
Jan 16 10:20:05.403: ISAKMP: (1778):Input = IKE_MESG_FROM_TIMER, IKE_TIMER_IM_ALIVE
Jan 16 10:20:05.403: ISAKMP: (1778):Old State = IKE_P1_COMPLETE New State = IKE_P1_COMPLETE
Jan 16 10:20:05.407: ISAKMP: (1796):purging node 1737591358
Jan 16 10:20:05.455: ISAKMP-PAK: (1778):received packet from x.x.x.x dport 4500 sport 4500 Global (I) QM_IDLE
Jan 16 10:20:05.455: ISAKMP: (1778):set new node 1270361410 to QM_IDLE
Jan 16 10:20:05.455: ISAKMP: (1778):processing HASH payload. message ID = 1270361410
Jan 16 10:20:05.455: ISAKMP: (1778):deleting node 1270361410 error FALSE reason "Informational (in) state 1"
Jan 16 10:20:05.455: ISAKMP: (1778):Input = IKE_MESG_FROM_PEER, IKE_INFO_NOTIFY
Jan 16 10:20:05.455: ISAKMP: (1778):Old State = IKE_P1_COMPLETE New State = IKE_P1_COMPLETE
Jan 16 10:20:06.147: ISAKMP: (1778):purging node 1427283446
Jan 16 10:20:06.663: ISAKMP-PAK: (1778):received packet from x.x.x.x dport 4500 sport 4500 Global (I) QM_IDLE
Jan 16 10:20:06.663: ISAKMP: (1778):set new node 271027263 to QM_IDLE
Jan 16 10:20:06.663: ISAKMP: (1778):processing HASH payload. message ID = 271027263
Jan 16 10:20:06.663: ISAKMP: (1778):deleting node 271027263 error FALSE reason "Informational (in) state 1"
Jan 16 10:20:06.663: ISAKMP: (1778):Input = IKE_MESG_FROM_PEER, IKE_INFO_NOTIFY
Jan 16 10:20:06.663: ISAKMP: (1778):Old State = IKE_P1_COMPLETE New State = IKE_P1_COMPLETE
HUB DMVPN DEBUG DETAIL ALL:
an 17 13:55:37.378: NHRP RIB_RWATCH: Debugging is ON
Jan 17 13:55:37.758: ISAKMP-PAK: (19081):received packet from x.x.x.x dport 4500 sport 16826 Global (R) QM_IDLE
Jan 17 13:55:37.758: ISAKMP: (19081):set new node 2616653086 to QM_IDLE
Jan 17 13:55:37.758: ISAKMP: (19081):processing HASH payload. message ID = 2616653086
Jan 17 13:55:37.758: ISAKMP: (19081):deleting node 2616653086 error FALSE reason "Informational (in) state 1"
Jan 17 13:55:37.758: ISAKMP: (19081):Input = IKE_MESG_FROM_PEER, IKE_INFO_NOTIFY
Jan 17 13:55:37.758: ISAKMP: (19081):Old State = IKE_P1_COMPLETE New State = IKE_P1_COMPLETE
Jan 17 13:55:37.758: ISAKMP: (19081):set new node 3112926514 to QM_IDLE
Jan 17 13:55:37.759: ISAKMP-PAK: (19081):sending packet to x.x.x.x my_port 4500 peer_port 16826 (R) QM_IDLE
Jan 17 13:55:37.759: ISAKMP: (19081):Sending an IKE IPv4 Packet.
Jan 17 13:55:37.759: ISAKMP: (19081):purging node 3112926514
Jan 17 13:55:37.759: ISAKMP: (19081):Input = IKE_MESG_FROM_PEER, IKE_MESG_KEEP_ALIVE
Jan 17 13:55:37.759: ISAKMP: (19081):Old State = IKE_P1_COMPLETE New State = IKE_P1_COMPLETE
Jan 17 13:55:37.941: ISAKMP-PAK: (0):received packet from x.x.x.x dport 500 sport 13081 Global (R) MM_NO_STATE
Jan 17 13:55:37.950: ISAKMP: (19081):purging node 2025426289
Jan 17 13:55:38.090: ISAKMP: (19069):purging node 156923107
Jan 17 13:55:38.168: ISAKMP: (19069):purging node 2404492623
Jan 17 13:55:38.411: ISAKMP: (19127):set new node 1884892475 to QM_IDLE
Jan 17 13:55:38.412: ISAKMP-PAK: (19127):sending packet to x.x.x.x my_port 4500 peer_port 32017 (R) QM_IDLE
Jan 17 13:55:38.412: ISAKMP: (19127):Sending an IKE IPv4 Packet.
Jan 17 13:55:38.412: ISAKMP: (19127):purging node 1884892475
Jan 17 13:55:38.412: ISAKMP: (19127):Input = IKE_MESG_FROM_TIMER, IKE_TIMER_IM_ALIVE
Jan 17 13:55:38.412: ISAKMP: (19127):Old State = IKE_P1_COMPLETE New State = IKE_P1_COMPLETE
Jan 17 13:55:38.508: ISAKMP-PAK: (19127):received packet from x.x.x.x dport 4500 sport 32017 Global (R) QM_IDLE
Jan 17 13:55:38.508: ISAKMP: (19127):set new node 2015868514 to QM_IDLE
Jan 17 13:55:38.509: ISAKMP: (19127):processing HASH payload. message ID = 2015868514
Jan 17 13:55:38.509: ISAKMP: (19127):deleting node 2015868514 error FALSE reason "Informational (in) state 1"
Jan 17 13:55:38.509: ISAKMP: (19127):Input = IKE_MESG_FROM_PEER, IKE_INFO_NOTIFY
Jan 17 13:55:38.509: ISAKMP: (19127):Old State = IKE_P1_COMPLETE New State = IKE_P1_COMPLETE
Jan 17 13:55:38.795: ISAKMP-PAK: (19068):received packet from x.x.x.x dport 4500 sport 42076 Global (R) QM_IDLE
Jan 17 13:55:38.795: ISAKMP: (19068):set new node 2591780475 to QM_IDLE
Jan 17 13:55:38.795: ISAKMP: (19068):processing HASH payload. message ID = 2591780475
Jan 17 13:55:38.795: ISAKMP: (19068):deleting node 2591780475 error FALSE reason "Informational (in) state 1"
Jan 17 13:55:38.795: ISAKMP: (19068):Input = IKE_MESG_FROM_PEER, IKE_INFO_NOTIFY
Jan 17 13:55:38.795: ISAKMP: (19068):Old State = IKE_P1_COMPLETE New State = IKE_P1_COMPLETE
Jan 17 13:55:38.795: ISAKMP: (19068):set new node 2994271834 to QM_IDLE
Jan 17 13:55:38.795: ISAKMP-PAK: (19068):sending packet to x.x.x.x my_port 4500 peer_port 42076 (R) QM_IDLE
Jan 17 13:55:38.795: ISAKMP: (19068):Sending an IKE IPv4 Packet.
Jan 17 13:55:38.795: ISAKMP: (19068):purging node 2994271834
Jan 17 13:55:38.795: ISAKMP: (19068):Input = IKE_MESG_FROM_PEER, IKE_MESG_KEEP_ALIVE
Jan 17 13:55:38.795: ISAKMP: (19068):Old State = IKE_P1_COMPLETE New State = IKE_P1_COMPLETE
Any help would be appreciated
Best regards,
