lost connection from DMVPN Peers

[georgesofroniadis]

hi,

i would like to express my issue and if you are able to advice me . i have a company router IR1101 with internet connection provided with private ipv4  address in the WAN interface. my company uses DMVPN in order to connect to company infrastructure. The issue is tha the connection of DMVPN is lost every 4 hours for 20-30 minutes. I checked the logs and i saw that there is authentication failure in the exchange message of the ikev2 . it is sometrhing that i cannot to explain since the site and router worked prefectly before this issue appeared. 

Do you have any suggestions?

[MHM Cisco World]

Run ip sla between spoke and hub make tunnel always UP.

It can ikev2 sa issue 

MHM

[balaji.bandi]

what kind of logs can you post the logs here to look ?

is this Logs from head end or branch router, can you post both the logs ?

is this only 1 Router losing connection every 4 hours ? or all other branch routers ?

what is the head end router, what IOS code running both the sides ?

Guide lines how to troubleshoot :

https://www.cisco.com/c/en/us/support/docs/security/dynamic-multipoint-vpn-dmvpn/111976-dmvpn-troubleshoot-00.html

https://www.cisco.com/c/en/us/support/docs/ip/internet-key-exchange-ike/116858-problem-replay-00.html

[georgesofroniadis]

the logs are from debugging the crypto ikev2. Yes only that one router has the issue. it is the spoke router. only logs on this spoke. about the others i dont have access.

atthaches is the log file. the IOS is the following: ir1101-universalk9.17.06.01a.SPA.bin
thanks!

[Giuseppe Larosa]

Hello @georgesofroniadis ,

at some point in the debug we see may requests no one of them get an answer

so at some point we see:

>> Oct 1 2024 13:33:40.872 UTC: IKEv2:(SESSION ID = 2,SA ID = 9):Sending Packet [To 193.x.y.207:4500/From 10.0.0.10:4500/VRF i0:f0]
Initiator SPI : C61FA05DBCD4E8BD - Responder SPI : 574998F98E3084B6 Message id: 1
IKEv2 IKE_AUTH Exchange REQUEST
Payload contents:

there are many attempts later we see a line that it is important :

Oct 1 2024 13:33:12.065 UTC: IKEv2:(SESSION ID = 1,SA ID = 8):Attrib type: split-dns, length: 0
Oct 1 2024 13:33:12.065 UTC: IKEv2:(SESSION ID = 1,SA ID = 8):Attrib type: banner, length: 0
--More--  Oct 1 2024 13:33:12.065 UTC: IKEv2:(SESSION ID = 1,SA ID = 8):Attrib type: config-url, length: 0
Oct 1 2024 13:33:12.065 UTC: IKEv2:(SESSION ID = 1,SA ID = 8):Attrib type: backup-gateway, length: 0
Oct 1 2024 13:33:12.065 UTC: IKEv2:(SESSION ID = 1,SA ID = 8):Attrib type: def-domain, length: 0
Oct 1 2024 13:33:12.065 UTC: IKEv2:(SESSION ID = 1,SA ID = 8):Have config mode data to send
Oct 1 2024 13:33:12.066 UTC: IKEv2:(SESSION ID = 1,SA ID = 8):Check for EAP exchange

SKF

finally:

Oct 1 2024 13:33:44.047 UTC: IKEv2-ERROR:(SESSION ID = 1,SA ID = 1):: Maximum number of retransmissions reached
Oct 1 2024 13:33:44.047 UTC: IKEv2:(SESSION ID = 1,SA ID = 1):Auth exchange failed
Oct 1 2024 13:33:44.047 UTC: IKEv2-ERROR:(SESSION ID = 1,SA ID = 1):: Auth exchange failed
Oct 1 2024 13:33:44.048 UTC: IKEv2:(SESSION ID = 1,SA ID = 1):Abort exchange
Oct 1 2024 13:33:44.051 UTC: IKEv2:(SESSION ID = 1,SA ID = 1):Deleting SA
Oct 1 2024 13:33:44.051 UTC: IKEv2:(SA ID = 1):[IKEv2 -> PKI] Close PKI Session
Oct 1 2024 13:33:44.052 UTC: IKEv2:(SA ID = 1):[PKI -> IKEv2] Closing of PKI Session PASSED
Oct 1 2024 13:33:46.616 UTC: IKEv2:(SESSION ID = 2,SA ID = 9):Retransmitting packet

note : if one side thinks the other part is a backup gateway and the primary is up it will stop to negotiate with your device.

at least I see this on Cisco Firepower 1140 with release 7.2.5.1 no IKEv2 exchange with backup peer occurs when primary is up if to make a test configure on FW side a wrong primary address so that the primary IKEv2 fails the remote does not accepts negotiation on its secondary outside because its own primary is still alive.

Hope to help

Giuseppe

[MHM Cisco World]

Show ip nhrp nhs detail 
show dmvpn detail 
debug dmvpn detail all 

MHM

[balaji.bandi]

Do you have access to Hub Router, only you have access to spoke ?

if you have only Spoke, then you need to co-ordinate other side Hub side to get some Logs.

[georgesofroniadis]

Thanks for your answer and support! much appreciated! 

[georgesofroniadis]

hi i attach new log on this topic.

i enable debug dmvpn all all. i am suspecting that could a customer issue. because from our side nothing change.

Could you advice me for some other debug options?

[MHM Cisco World]

I ask you before these debug, please share it 

Show ip nhrp nhs detail 
show dmvpn detail 
debug dmvpn detail all 

[georgesofroniadis]

i am sharing the logs

[MHM Cisco World]

    1 xx.xx1.206      10.126.160.1   IKE 00:29:48     S    10.126.160.1/32
    1 xx.xx1.207      10.126.160.2  NHRP 00:29:51     S    10.126.160.2/32

These inform me that there are two Hub in in your DMVPN 
one stop in IKE and other in NHRP 

there is req send 3250 and only 99 reply 

10.126.160.1   E  NBMA Address: xx.xx1.206 priority = 0 cluster = 0  req-sent 3250  req-failed 3  repl-recv 99 (00:30:28 ago)
10.126.160.2   E  NBMA Address: xx.xx1.207 priority = 0 cluster = 0  req-sent 3250  req-failed 3  repl-recv 99 (00:30:28 ago)

Can I see Spoke tunnel config ?

MHM

[georgesofroniadis]

interface Tunnel10
bandwidth 5000
ip address 10.126.189.159 255.255.224.0
no ip redirects
no ip proxy-arp
ip mtu 1300
ip nat outside
ip nhrp authentication Pit10
ip nhrp network-id 101261600
ip nhrp nhs 10.126.160.1 nbma xx.xx.1.206
ip nhrp nhs 10.126.160.2 nbma xx.xx.1.207
zone-member security DMVPN
ip tcp adjust-mss 1260
load-interval 30
if-state nhrp
qos pre-classify
tunnel source GigabitEthernet0/0/0
tunnel mode gre multipoint
tunnel key xxxxxxxx
tunnel protection ipsec profile IPSEC-PROFILE ikev2-profile IKE2-CERT-PROFILE

[MHM Cisco World]

    1 xx.xx1.206      10.126.160.1   IKE 00:29:48     S    10.126.160.1/32 <<- 

 ip nhrp nhs 10.126.160.1 nbma xx.xx.1.206 <<- remove this Hub 
and check 

NV-36200-WAN1#show dmvpn detail <<- this must show statc UP for Hub .207

MHM 

[georgesofroniadis]

=========================================================================

Interface Tunnel10 is up/down, Addr. is 10.126.189.159, VRF "global"
Tunnel Src./Dest. addr: 10.0.0.10/Multipoint, Tunnel VRF "global"
Protocol/Transport: "multi-GRE/IP", Protect "IPSEC-PROFILE"
Interface State Control: Enabled
nhrp event-publisher : Disabled

IPv4 NHS:
10.126.160.2 E NBMA Address: xx.xx1.207 priority = 0 cluster = 0
Type:Spoke, Total NBMA Peers (v4/v6): 1

# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb Target Network
----- --------------- --------------- ----- -------- ----- -----------------
1 xx.xx1.207 10.126.160.2 IKE 00:57:43 S 10.126.160.2/32

Crypto Session Details:
--------------------------------------------------------------------------------

Interface: Tunnel10
Session: [0xFFFF5A54E3B0]
Crypto Session Status: DOWN
fvrf: (none), IPSEC FLOW: permit 47 host 10.0.0.10 host xx.xx1.207
Active SAs: 0, origin: crypto map
Inbound: #pkts dec'ed 21668 drop 0 life (KB/Sec) 0/0
Outbound: #pkts enc'ed 484 drop 72 life (KB/Sec) 0/0
Outbound SPI : 0x 0, transform :

still down. useful information could be that the connection is restored for 20 -30 min every 4 hours