DMVPN hub upgrade 2921 to 4331, spoke IR1101 reg fail IR829 good

[magla0001]

After upgrading our DMVPN hub from Cisco 2921 to Cisco 4431, spoke that are Cisco IR1101 stopped establishing tunnel back to the Cisco 4431 hub. Spoke that are Cisco 1921, Cisco 2911, Cisco 2921, and IR829 are all able to establish tunnel back to the Cisco 4331 hub. I am able to ping the cellular interface's static IP between hub and spoke. These spoke Cisco IR1101 has an established tunnel to another Hub that is still Cisco 2921, just the tunnel to Cisco 4431 would not establish. Any help would be appreciated.

[MHM Cisco World]

Show dmvpn detail <<- if the dmvpn stop in ipsec then check 
show crypto socket <<- share this

1- SA transform 

2- dh group

MHM

[ccieexpert]

in addition to what MHM said, refer to this:

https://www.cisco.com/c/en/us/support/docs/security/dynamic-multipoint-vpn-dmvpn/111976-dmvpn-troubleshoot-00.html

if the HUB has too many spokes only run debugs on spoke (and run conditional debugs for this spoke on the hub):

debug crypto isakmp

debug crypto ipsec

debug crypto socket

[magla0001]

Thank you for your suggestions.

I removed the tunnel protection ipsec profile HUBIPSECPROFCP from both hub and spoke tunnel interface, and the tunnel established between them but obviously not encrypted. Here is what I gathered from show crypto sockets and show dmvpn detail but did not get anything from debug.

[MHM Cisco World]

the state ""IKE"" meaning the ipsec phaseI is failed 
most case come as I mention before from group is not support in some spokes or hub 
so 
debug crypto isakmp <<- in hub will give you exact the issue is it group or other SA mismatch 
what is group you use for phaseI in old and new router and spokes ?

[magla0001]

Here is what I captured from the hub using debug crypto ikev2. I do not use isakmp. It looks like it failed to receive the auth, hence timer expired, but did not get anything from the spoke using the same debug. From spoke "show ip nhrp nhs detail" I get req-sent 102 but req-failed 0, and repl-recv 0. Why is sent incrementing but no failed or reply. Is it because "Show crypto session detail" Outbound #pkts enc'ed 0 drop 392 show it gets dropped and inbound is zero.

Aug 26 2024 14:32:00.689 PDT: IKEv2:Received Packet [From 172.16.10.27:512/To 172.16.10.43:500/VRF i0:f0]
Initiator SPI : 307C31D5AACDDA8F - Responder SPI : 0000000000000000 Message id: 0
IKEv2 IKE_SA_INIT Exchange REQUEST
Payload contents:

Aug 26 2024 14:32:00.689 PDT: IKEv2:parsing SA payload SA
Aug 26 2024 14:32:00.690 PDT: IKEv2:parsing KE payload KE
Aug 26 2024 14:32:00.690 PDT: IKEv2:parsing N payload N
Aug 26 2024 14:32:00.690 PDT: IKEv2:parsing VID payload VID
Aug 26 2024 14:32:00.690 PDT: IKEv2:parsing VID payload VID
Aug 26 2024 14:32:00.690 PDT: IKEv2:parsing VID payload VID
Aug 26 2024 14:32:00.690 PDT: IKEv2:parsing VID payload VID
Aug 26 2024 14:32:00.690 PDT: IKEv2:parsing NOTIFY payload NOTIFY(NAT_DETECTION_SOURCE_IP)
Aug 26 2024 14:32:00.690 PDT: IKEv2:parsing NOTIFY payload NOTIFY(NAT_DETECTION_DESTINATION_IP)

Aug 26 2024 14:32:00.690 PDT: IKEv2:(SESSION ID = 5950,SA ID = 16):Verify SA init message
Aug 26 2024 14:32:00.690 PDT: IKEv2:(SESSION ID = 5950,SA ID = 16):Insert SA
Aug 26 2024 14:32:00.690 PDT: IKEv2:Searching Policy with fvrf 0, local address 172.16.10.43
Aug 26 2024 14:32:00.690 PDT: IKEv2:Found Policy 'HUBIKEVPOLCP'
Aug 26 2024 14:32:00.690 PDT: IKEv2:(SESSION ID = 5950,SA ID = 16):Processing IKE_SA_INIT message
Aug 26 2024 14:32:00.691 PDT: IKEv2:(SA ID = 16):[IKEv2 -> PKI] Retrieve configured trustpoint(s)
Aug 26 2024 14:32:00.691 PDT: IKEv2:(SA ID = 16):[PKI -> IKEv2] Retrieved trustpoint(s): 'SLA-TrustPoint' 'TP-self-signed-2720882985'
Aug 26 2024 14:32:00.691 PDT: IKEv2:(SA ID = 16):[IKEv2 -> PKI] Get Public Key Hashes of trustpoints
Aug 26 2024 14:32:00.691 PDT: IKEv2:(SA ID = 16):[PKI -> IKEv2] Getting of Public Key Hashes of trustpoints PASSED
Aug 26 2024 14:32:00.691 PDT: IKEv2:(SA ID = 16):[IKEv2 -> PKI] Start PKI Session
Aug 26 2024 14:32:00.691 PDT: IKEv2:(SA ID = 16):[PKI -> IKEv2] Starting of PKI Session PASSED
Aug 26 2024 14:32:00.691 PDT: IKEv2:(SESSION ID = 5950,SA ID = 16):[IKEv2 -> Crypto Engine] Computing DH public key, DH Group 15
Aug 26 2024 14:32:00.691 PDT: IKEv2:(SESSION ID = 5950,SA ID = 16):(SA ID = 16):[Crypto Engine -> IKEv2] DH key Computation PASSED
Aug 26 2024 14:32:00.691 PDT: IKEv2:(SESSION ID = 5950,SA ID = 16):Request queued for computation of DH key
Aug 26 2024 14:32:00.692 PDT: IKEv2:(SESSION ID = 5950,SA ID = 16):[IKEv2 -> Crypto Engine] Computing DH secret key, DH Group 15
Aug 26 2024 14:32:00.757 PDT: IKEv2:(SESSION ID = 5950,SA ID = 16):(SA ID = 16):[Crypto Engine -> IKEv2] DH key Computation PASSED
Aug 26 2024 14:32:00.757 PDT: IKEv2:(SESSION ID = 5950,SA ID = 16):Request queued for computation of DH secret
Aug 26 2024 14:32:00.757 PDT: IKEv2-ERROR:(SESSION ID = 5940,SA ID = 2):: Failed to receive the AUTH msg before the timer expired
Aug 26 2024 14:32:00.757 PDT: IKEv2:(SESSION ID = 5940,SA ID = 2):Auth exchange failed
Aug 26 2024 14:32:00.757 PDT: IKEv2-ERROR:(SESSION ID = 5940,SA ID = 2):: Auth exchange failed
Aug 26 2024 14:32:00.757 PDT: IKEv2:(SESSION ID = 5940,SA ID = 2):Abort exchange
Aug 26 2024 14:32:00.757 PDT: IKEv2:(SESSION ID = 5940,SA ID = 2):Deleting SA
Aug 26 2024 14:32:00.757 PDT: IKEv2:(SA ID = 2):[IKEv2 -> PKI] Close PKI Session
Aug 26 2024 14:32:00.757 PDT: IKEv2:(SA ID = 2):[PKI -> IKEv2] Closing of PKI Session PASSED
Aug 26 2024 14:32:00.758 PDT: IKEv2:(SESSION ID = 5950,SA ID = 16):(SA ID = 16):[IKEv2 -> Crypto Engine] Calculate SKEYSEED and create rekeyed IKEv2 SA
Aug 26 2024 14:32:00.758 PDT: IKEv2:(SESSION ID = 5950,SA ID = 16):(SA ID = 16):[Crypto Engine -> IKEv2] SKEYSEED calculation and creation of rekeyed IKEv2 SA PASSED
Aug 26 2024 14:32:00.758 PDT: IKEv2:IKEv2 responder - no config data to send in IKE_SA_INIT exch
Aug 26 2024 14:32:00.758 PDT: IKEv2:(SESSION ID = 5950,SA ID = 16):Generating IKE_SA_INIT message
Aug 26 2024 14:32:00.758 PDT: IKEv2:(SESSION ID = 5950,SA ID = 16):IKE Proposal: 1, SPI size: 0 (initial negotiation),
Num. transforms: 4
AES-CBC SHA256 SHA256 DH_GROUP_3072_MODP/Group 15
Aug 26 2024 14:32:00.758 PDT: IKEv2:(SA ID = 16):[IKEv2 -> PKI] Retrieve configured trustpoint(s)
Aug 26 2024 14:32:00.758 PDT: IKEv2:(SA ID = 16):[PKI -> IKEv2] Retrieved trustpoint(s): 'SLA-TrustPoint' 'TP-self-signed-2720882985'
Aug 26 2024 14:32:00.758 PDT: IKEv2:(SA ID = 16):[IKEv2 -> PKI] Get Public Key Hashes of trustpoints
Aug 26 2024 14:32:00.759 PDT: IKEv2:(SA ID = 16):[PKI -> IKEv2] Getting of Public Key Hashes of trustpoints PASSED

Aug 26 2024 14:32:00.759 PDT: IKEv2:(SESSION ID = 5950,SA ID = 16):Sending Packet [To 172.16.10.27:512/From 172.16.10.43:500/VRF i0:f0]
Initiator SPI : 307C31D5AACDDA8F - Responder SPI : E3E002EA56B6ED1B Message id: 0

[MHM Cisco World]

So you use ikev2  not ikev1 

Ok 

Debug crypto ikev2 error 

Then disable it and run 

Debug crypto ikev2 packet 

Share both debug please 

MHM

[magla0001]

Hi MHM,

Thank you for helping with this. Please find below the results of the debug crypto ikev2 packet and error.

Aug 27 2024 07:35:45.054 PDT: IKEv2-ERROR:: Packet is a retransmission
Aug 27 2024 07:35:46.773 PDT: IKEv2-ERROR:: Packet is a retransmission
Aug 27 2024 07:35:47.847 PDT: IKEv2-ERROR:(SESSION ID = 32024,SA ID = 5):: Failed to receive the AUTH msg before the timer expired
Aug 27 2024 07:35:47.847 PDT: IKEv2-ERROR:(SESSION ID = 32024,SA ID = 5):: Auth exchange failed
Aug 27 2024 07:35:48.924 PDT: IKEv2-ERROR:: Packet is a retransmission
Aug 27 2024 07:35:51.167 PDT: IKEv2-ERROR:(SESSION ID = 32025,SA ID = 6):: Failed to receive the AUTH msg before the timer expired
Aug 27 2024 07:35:51.167 PDT: IKEv2-ERROR:(SESSION ID = 32025,SA ID = 6):: Auth exchange failed
----------------------------------------------------------------------------------------------------------------
Aug 27 2024 07:35:57.205 PDT: IKEv2-PAK:Next payload: SA, version: 2.0 Exchange type: IKE_SA_INIT, flags: INITIATOR Message id: 0, length: 646
Payload contents:
SA Next payload: KE, reserved: 0x0, length: 48
last proposal: 0x0, reserved: 0x0, length: 44
Proposal: 1, Protocol id: IKE, SPI size: 0, #trans: 4 last transform: 0x3, reserved: 0x0: length: 12
type: 1, reserved: 0x0, id: AES-CBC
last transform: 0x3, reserved: 0x0: length: 8
type: 2, reserved: 0x0, id: SHA256
last transform: 0x3, reserved: 0x0: length: 8
type: 3, reserved: 0x0, id: SHA256
last transform: 0x0, reserved: 0x0: length: 8
type: 4, reserved: 0x0, id: DH_GROUP_3072_MODP/Group 15
KE Next payload: N, reserved: 0x0, length: 392
DH group: 15, Reserved: 0x0
N Next payload: VID, reserved: 0x0, length: 36
VID Next payload: VID, reserved: 0x0, length: 23
VID Next payload: VID, reserved: 0x0, length: 19
VID Next payload: VID, reserved: 0x0, length: 23
VID Next payload: NOTIFY, reserved: 0x0, length: 21
NOTIFY(NAT_DETECTION_SOURCE_IP) Next payload: NOTIFY, reserved: 0x0, length: 28
Security protocol id: Unknown - 0, spi size: 0, type: NAT_DETECTION_SOURCE_IP
NOTIFY(NAT_DETECTION_DESTINATION_IP) Next payload: NONE, reserved: 0x0, length: 28
Security protocol id: Unknown - 0, spi size: 0, type: NAT_DETECTION_DESTINATION_IP

-------------------------------------------------------------------------------------

Aug 27 2024 07:36:03.286 PDT: IKEv2-PAK:(SESSION ID = 32044,SA ID = 10):Next payload: SA, version: 2.0 Exchange type: IKE_SA_INIT, flags: RESPONDER MSG-RESPONSE Message id: 0, length: 679
Payload contents:
SA Next payload: KE, reserved: 0x0, length: 48
last proposal: 0x0, reserved: 0x0, length: 44
Proposal: 1, Protocol id: IKE, SPI size: 0, #trans: 4 last transform: 0x3, reserved: 0x0: length: 12
type: 1, reserved: 0x0, id: AES-CBC
last transform: 0x3, reserved: 0x0: length: 8
type: 2, reserved: 0x0, id: SHA256
last transform: 0x3, reserved: 0x0: length: 8
type: 3, reserved: 0x0, id: SHA256
last transform: 0x0, reserved: 0x0: length: 8
type: 4, reserved: 0x0, id: DH_GROUP_3072_MODP/Group 15
KE Next payload: N, reserved: 0x0, length: 392
DH group: 15, Reserved: 0x0
N Next payload: VID, reserved: 0x0, length: 36
VID Next payload: VID, reserved: 0x0, length: 23
VID Next payload: VID, reserved: 0x0, length: 19
VID Next payload: VID, reserved: 0x0, length: 23
VID Next payload: NOTIFY, reserved: 0x0, length: 21
NOTIFY(NAT_DETECTION_SOURCE_IP) Next payload: NOTIFY, reserved: 0x0, length: 28
Security protocol id: Unknown - 0, spi size: 0, type: NAT_DETECTION_SOURCE_IP
NOTIFY(NAT_DETECTION_DESTINATION_IP) Next payload: CERTREQ, reserved: 0x0, length: 28
Security protocol id: Unknown - 0, spi size: 0, type: NAT_DETECTION_DESTINATION_IP
CERTREQ Next payload: NOTIFY, reserved: 0x0, length: 25
Cert encoding Hash and URL of PKIX
NOTIFY(HTTP_CERT_LOOKUP_SUPPORTED) Next payload: NONE, reserved: 0x0, length: 8

------------------------------------------------------------------------------------------------------------------------------------------------
Aug 27 2024 07:36:13.207 PDT: IKEv2-ERROR:(SESSION ID = 32032,SA ID = 2):: Failed to receive the AUTH msg before the timer expired
Aug 27 2024 07:36:13.207 PDT: IKEv2-ERROR:(SESSION ID = 32032,SA ID = 2):: Auth exchange failed
Aug 27 2024 07:36:13.273 PDT: IKEv2-ERROR:(SESSION ID = 32033,SA ID = 3):: Failed to receive the AUTH msg before the timer expired
Aug 27 2024 07:36:13.273 PDT: IKEv2-ERROR:(SESSION ID = 32033,SA ID = 3):: Auth exchange failed
Aug 27 2024 07:36:13.986 PDT: IKEv2-ERROR:: Packet is a retransmission
Aug 27 2024 07:36:14.373 PDT: IKEv2-ERROR:: Packet is a retransmission
Aug 27 2024 07:36:14.528 PDT: IKEv2-ERROR:: Packet is a retransmission
Aug 27 2024 07:36:15.196 PDT: IKEv2-ERROR:(SESSION ID = 32034,SA ID = 16):: Failed to receive the AUTH msg before the timer expired
Aug 27 2024 07:36:15.196 PDT: IKEv2-ERROR:(SESSION ID = 32034,SA ID = 16):: Auth exchange failed
Aug 27 2024 07:36:15.603 PDT: IKEv2-ERROR:: Packet is a retransmission
Aug 27 2024 07:36:15.884 PDT: IKEv2-ERROR:(SESSION ID = 32035,SA ID = 17):: Failed to receive the AUTH msg before the timer expired
Aug 27 2024 07:36:15.884 PDT: IKEv2-ERROR:(SESSION ID = 32035,SA ID = 17):: Auth exchange failed
Aug 27 2024 07:36:18.320 PDT: IKEv2-ERROR:(SESSION ID = 32036,SA ID = 5):: Failed to receive the AUTH msg before the timer expired
Aug 27 2024 07:36:18.320 PDT: IKEv2-ERROR:(SESSION ID = 32036,SA ID = 5):: Auth exchange failed
Aug 27 2024 07:36:18.419 PDT: IKEv2-ERROR:(SESSION ID = 32037,SA ID = 18):: Failed to receive the AUTH msg before the timer expired
Aug 27 2024 07:36:18.419 PDT: IKEv2-ERROR:(SESSION ID = 32037,SA ID = 18):: Auth exchange failed
Aug 27 2024 07:36:19.285 PDT: IKEv2-ERROR:: Packet is a retransmission
Aug 27 2024 07:36:19.744 PDT: IKEv2-ERROR:(SESSION ID = 32038,SA ID = 23):: Failed to receive the AUTH msg before the timer expired
Aug 27 2024 07:36:19.744 PDT: IKEv2-ERROR:(SESSION ID = 32038,SA ID = 23):: Auth exchange failed
Aug 27 2024 07:36:20.605 PDT: IKEv2-ERROR:: Packet is a retransmission
Aug 27 2024 07:36:20.751 PDT: IKEv2-ERROR:(SESSION ID = 32039,SA ID = 25):: Failed to receive the AUTH msg before the timer expired
Aug 27 2024 07:36:20.751 PDT: IKEv2-ERROR:(SESSION ID = 32039,SA ID = 25):: Auth exchange failed
Aug 27 2024 07:36:22.174 PDT: IKEv2-ERROR:: Packet is a retransmission
Aug 27 2024 07:36:22.174 PDT: IKEv2-ERROR:: Packet is a retransmission
Aug 27 2024 07:36:24.590 PDT: IKEv2-ERROR:: Packet is a retransmission
Aug 27 2024 07:36:25.761 PDT: IKEv2-ERROR:: Packet is a retransmission
Aug 27 2024 07:36:26.028 PDT: IKEv2-ERROR:(SESSION ID = 32040,SA ID = 6):: Failed to receive the AUTH msg before the timer expired
Aug 27 2024 07:36:26.028 PDT: IKEv2-ERROR:(SESSION ID = 32040,SA ID = 6):: Auth exchange failed
Aug 27 2024 07:36:26.583 PDT: IKEv2-ERROR:(SESSION ID = 32041,SA ID = 26):: Failed to receive the AUTH msg before the timer expired
Aug 27 2024 07:36:26.583 PDT: IKEv2-ERROR:(SESSION ID = 32041,SA ID = 26):: Auth exchange failed
Aug 27 2024 07:36:27.013 PDT: IKEv2-ERROR:: Packet is a retransmission
Aug 27 2024 07:36:27.272 PDT: IKEv2-ERROR:(SESSION ID = 32042,SA ID = 27):: Failed to receive the AUTH msg before the timer expired
Aug 27 2024 07:36:27.272 PDT: IKEv2-ERROR:(SESSION ID = 32042,SA ID = 27):: Auth exchange failed
Aug 27 2024 07:36:28.886 PDT: IKEv2-ERROR:: Packet is a retransmission
Aug 27 2024 07:36:29.921 PDT: IKEv2-ERROR:: Packet is a retransmission
Aug 27 2024 07:36:30.799 PDT: IKEv2-ERROR:(SESSION ID = 32043,SA ID = 7):: Failed to receive the AUTH msg before the timer expired
Aug 27 2024 07:36:30.799 PDT: IKEv2-ERROR:(SESSION ID = 32043,SA ID = 7):: Auth exchange failed
Aug 27 2024 07:36:31.231 PDT: IKEv2-ERROR:(SESSION ID = 32044,SA ID = 10):: Failed to receive the AUTH msg before the timer expired
Aug 27 2024 07:36:31.231 PDT: IKEv2-ERROR:(SESSION ID = 32044,SA ID = 10):: Auth exchange failed

[MHM Cisco World]

What is type of auth yoh use rsa or preshared ?

MHM

[magla0001]

We use preshared and checked that matched end to end.

[MHM Cisco World]

type: 1, reserved: 0x0, id: AES-CBC
last transform: 0x3, reserved: 0x0: length: 8
type: 2, reserved: 0x0, id: SHA256
last transform: 0x3, reserved: 0x0: length: 8
type: 3, reserved: 0x0, id: SHA256
last transform: 0x0, reserved: 0x0: length: 8
type: 4, reserved: 0x0, id: DH_GROUP_3072_MODP/Group 15

type: 1, reserved: 0x0, id: AES-CBC
last transform: 0x3, reserved: 0x0: length: 8
type: 2, reserved: 0x0, id: SHA256
last transform: 0x3, reserved: 0x0: length: 8
type: 3, reserved: 0x0, id: SHA256
last transform: 0x0, reserved: 0x0: length: 8
type: 4, reserved: 0x0, id: DH_GROUP_3072_MODP/Group 15

phase1 use AES-CBC SHA256 DH group 15
both spoke and hub use PreShared Key 
so 
do
show crypto ikev2 sa detail <<- in both hub and spoke see the SA is same and use or not 

MHM

[magla0001]

I tried to change the lifetime to 86400 but still appear 120 from the hub. I did 120 both hub and spoke and got it to 120 both, but still would not establish connection. Spoke appear to not recognize Encr, DH that is configured but they are the same configured at hub and spoke.

-----------------------------------------------------------

SPOKE01#show crypto ikev2 sa detailed
Tunnel-id Local Remote fvrf/ivrf Status
6 172.16.10.27/500 172.16.10.43/500 none/none IN-NEG
Encr: Unknown - 0, PRF: Unknown - 0, Hash: None, DH Grp:0, Auth sign: Unknown - 0, Auth verify: Unknown - 0
Life/Active Time: 86400/0 sec
CE id: 0, Session-id: 0
Local spi: B14A61F5F841DFF3 Remote spi: 0000000000000000
Status Description: Initiator waiting for INIT response
Local id: 172.16.10.27
Remote id:
Local req msg id: 0 Remote req msg id: 0
Local next msg id: 1 Remote next msg id: 0
Local req queued: 0 Remote req queued: 0
Local window: 1 Remote window: 1
DPD configured for 0 seconds, retry 0
Fragmentation not configured.
Dynamic Route Update: disabled
Extended Authentication not configured.
NAT-T is not detected
Cisco Trust Security SGT is disabled
Initiator of SA : Yes
PEER TYPE: Other

---------------------------------------------------
HUB01#show crypto ikev2 sa detailed

Tunnel-id Local Remote fvrf/ivrf Status
2 172.16.10.43/500 172.16.10.27/512 none/none IN-NEG
Encr: AES-CBC, keysize: 256, PRF: SHA256, Hash: SHA256, DH Grp:15, Auth sign: Unknown - 0, Auth verify: Unknown - 0
Life/Active Time: 120/0 sec
CE id: 0, Session-id: 0
Local spi: 35760CA3B920F961 Remote spi: D77D2A3F366B5147
Status Description: Responder waiting for AUTH message
Local id:
Remote id:
Local req msg id: 0 Remote req msg id: 1
Local next msg id: 0 Remote next msg id: 1
Local req queued: 0 Remote req queued: 1
Local window: 1 Remote window: 1
DPD configured for 0 seconds, retry 0
Fragmentation not configured.
Dynamic Route Update: disabled
Extended Authentication not configured.
NAT-T is detected outside
Cisco Trust Security SGT is disabled
Initiator of SA : No
PEER TYPE: IOS-XE

[MHM Cisco World · Accepted solution]

NAT-T is detected outside <<- this meaning spoke is behind NAT 

So dmvpn must config NAT aware first is change ipsec from tunnel to transport mode

Second what is the SA of transform you use 

MHM

[MHM Cisco World]

any update ?

MHM

[magla0001]

Hi MHM,

Thank you for your help! Ever since we implemented our cellular backup, since maybe around 2015, our vendor requires us to source NAT using the static IP of our cellular interface. If it was not NAT'ed connection would bounce up and down. After you mentioned of NAT issue, I decided to remove NAT'ing at the hub, but not the spoke, as these spoke connect to several hubs that does not have this issue. So far it does work. Just yesterday, I tried to test 2 remote spokes, removing NAT, and they connect to both hubs that are 2921 (with NAT implemented) and 4331 (without NAT). I am leaving this for a couple of weeks and rebooting both spoke and hub to make sure I will not have issue once implemented everywhere else.

I really appreciate your guidance in troubleshooting this issue and following up.