Hi all,
Out of the blue yesterday two of my spokes in my Phase3-DMVPN stopped being able to communicate between themselves. All other spokes are still able to communicate with each other AND communicate with the two spokes having the problem. The Hub is still able to communicate with both spokes as well.
Let's call the spokes Tom and Jerry
Tom and Jerry learn about each other through NHRP and share routes using iBGP, as do the rest of the Spokes.
After a bunch of troubleshooting various possibilities, I've narrowed down the symptoms a bit:
If there is no NHRP peer session between Tom and Jerry and I send some packets (ping) from Tom through the tunnel, the Session establishes, and ~15 pings will go through until they stop. From there on out, no communication between the two will work over the tunnel until I clear the dmvpn peer session. Then the same pattern repeats. Pings work for 15 frames or so, and then they stop until I manually clear the session or wait for the ~15 minutes for the session to drop on its own.
#debug cryp ikev2 error#debug cryp ipsec error#terminal mon
Shows the SA go down when I clear the dmvpn peer session
Tom#clear dmv sess peer Jerry
%IKEV2-5-SA_DOWN: SA DOWN
And then comes up after I start a ping
But without fail, a few seconds in, the SA goes down again, with a few errors I can't make reason of.
Tom#ping Jerry repeat 1
Type escape sequence to abort.
Sending 1, 100-byte ICMP Echos to Jerry, timeout is 2 seconds:
!
Success rate is 100 percent (1/1), round-trip min/avg/max = 24/24/24 ms
Oct 23 17:18:51: %IKEV2-5-RECV_CONNECTION_REQUEST: Received a IKE_INIT_SA request
Oct 23 17:18:51.267: IKEv2-ERROR:Failed to retrieve Certificate Issuer list
Oct 23 17:18:51.275: IKEv2-ERROR:Failed to retrieve Certificate Issuer list
Oct 23 17:18:51.279: insert of map into mapdb AVL failed, map + ace pair already exists on the mapdb
Oct 23 17:18:51: %IKEV2-5-OSAL_INITIATE_TUNNEL: Received request to establish an IPsec tunnel; local traffic selector = Address Range: Tom-Tom Protocol: 47 Port Range: 0-65535 ; remote traffic selector = Address Range: Jerry-Jerry 7 Protocol: 47 Port Range: 0-65535
Oct 23 17:18:51.451: IPSEC(ipsec_get_crypto_session_id):
Invalid Payload Id
Oct 23 17:18:51.451: IKEv2-ERROR:Error constructing config reply
Oct 23 17:18:51: %IKEV2-5-SA_UP: SA UP
Oct 23 17:18:51.451: IPSEC(ipsec_get_crypto_session_id):
Invalid Payload Id
Oct 23 17:18:51.639: IPSEC(ipsec_get_crypto_session_id):
Invalid Payload Id
Oct 23 17:18:51: %IKEV2-5-SA_UP: SA UP
Oct 23 17:18:51.639: IPSEC(ipsec_get_crypto_session_id):
Invalid Payload Id
Oct 23 17:18:51.651: IPSEC: sa null
Oct 23 17:18:51.651: IPSEC(send_delete_notify_kmi): not sending KEY_ENGINE_DELETE_SAS
Oct 23 17:18:51: %IKEV2-5-SA_DOWN: SA DOWN
After all this though, the two still seem to be fully peered. And stay that way about ~15 minutes before the session drops without much warning or say.
Tom#show dmvpn
Interface: Tunnel1, IPv4 NHRP Details
Type:Spoke, NHRP Peers:6,
# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb
----- --------------- --------------- ----- -------- -----
1 Jerry Jerry-T UP 00:05:54 DT1
Tom#show ip nhrp
Jerry-T/32 via Jerry-T
Tunnel1 created 00:07:04, expire 01:52:55
Type: dynamic, Flags: router used nhop rib
NBMA address: Jerry
AND the SA is still present..
Tom#show crypto ipsec sa
interface: Tunnel1
Crypto map tag: Tunnel1-head-0, local addr Tom
protected vrf: (none)
local ident (addr/mask/prot/port): (Tom/255.255.255.255/47/0)
remote ident (addr/mask/prot/port): (Jerry/255.255.255.255/47/0)
current_peer Jerry port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 46, #pkts encrypt: 46, #pkts digest: 46
#pkts decaps: 7, #pkts decrypt: 7, #pkts verify: 7
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 0, #recv errors 0
local crypto endpt.: Tom, remote crypto endpt.: Jerry
plaintext mtu 1458, path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/1
current outbound spi: 0x3792E6EC(932374252)
PFS (Y/N): N, DH group: none
inbound esp sas:
spi: 0x4E27AEBD(1311223485)
transform: esp-aes esp-sha-hmac ,
in use settings ={Transport, }
conn id: 991, flow_id: Onboard VPN:991, sibling_flags 80000000, crypto map: Tunnel1-head-0
sa timing: remaining key lifetime (k/sec): (4364220/3054)
IV size: 16 bytes
replay detection support: Y
Status: ACTIVE(ACTIVE)
inbound ah sas:
inbound pcp sas:
outbound esp sas:
spi: 0x3792E6EC(932374252)
transform: esp-aes esp-sha-hmac ,
in use settings ={Transport, }
conn id: 992, flow_id: Onboard VPN:992, sibling_flags 80000000, crypto map: Tunnel1-head-0
sa timing: remaining key lifetime (k/sec): (4364215/3054)
IV size: 16 bytes
replay detection support: Y
Status: ACTIVE(ACTIVE)
outbound ah sas:
outbound pcp sas:Pings between the public IP addresses of Tom and Jerry hold rock solid the whole time.
This is ONLY happening between Tom and Jerry and I'm worried about the rate of hair loss I'm currently experiencing....help.
--
Sure, understanding today's complex world of the future is a little like having bees live in your head. But, there they are.
