03-25-2018 07:10 AM - edited 03-12-2019 05:08 AM
Hi guys,
we are deploying DMVPN solution and we faced issue with flapping. Two sites were flapping every minute. When spoke1 was up, spoke2 was down, after one minutes it was inverted (I think it was caused by DPD). After some time I found out that it is caused by same private IP on WAN interface from provider. When I changed it, it became stable.
I was thinking that this design is supported so I tried to lab it a bit and play with that. I created simple topology. Firstly I tried it without tunnel protection. It was working and stable. Then I added tunnel protection with ISAKMP. This solution was also working and stable. Finally I changed configuration for tunnel protection with IKEv2 and again it was unstable.
Could you please advice me, if this design with same post-nat IP is supported with IKEv2 or if I have there some configuration problem? Configs are attached to this post.
HUB - ASR-1001-X
SPOKEs - ISR4221
Thank you a lot for your time!
Martin
Crypto config:
crypto keyring ISAKMP_KEYRING pre-shared-key address 0.0.0.0 0.0.0.0 key cisco123
crypto isakmp policy 10
encr aes 256
hash sha256
authentication pre-share
crypto isakmp profile ISAKMP_PROFILE
keyring ISAKMP_KEYRING
match identity address 0.0.0.0
crypto ikev2 proposal IKE2_PROPOSAL encryption aes-cbc-256 integrity sha256 group 5 crypto ikev2 policy IKE2_POLICY proposal IKE2_PROPOSAL crypto ikev2 keyring IKEV2_KEY peer DMVPN address 0.0.0.0 0.0.0.0 pre-shared-key cisco123 ! crypto ikev2 profile IKEV2_PROFILE match identity remote address 0.0.0.0 authentication remote pre-share authentication local pre-share keyring local IKEV2_KEY dpd 30 5 on-demand crypto ipsec transform-set ESP-AES256-SHA256 esp-aes 256 esp-sha256-hmac mode transport
crypto ipsec profile ISAKMP_IPSEC_PROFILE set transform-set ESP-AES256-SHA256 set isakmp-profile ISAKMP_PROFILE crypto ipsec profile IKEV2_IPSEC_PROFILE set transform-set ESP-AES256-SHA256 set ikev2-profile IKEV2_PROFILE
interface Tunnel10
tunnel protection ipsec profile ISAKMP_IPSEC_PROFILE/IKEV2_IPSEC_PROFILE
ISAKMP stable:
HUB(config-if)#do sh dmv det Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete N - NATed, L - Local, X - No Socket T1 - Route Installed, T2 - Nexthop-override C - CTS Capable # Ent --> Number of NHRP entries with same NBMA peer NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting UpDn Time --> Up or Down Time for a Tunnel ========================================================================== Interface Tunnel10 is up/up, Addr. is 10.208.0.1, VRF "" Tunnel Src./Dest. addr: 1.1.1.1/MGRE, Tunnel VRF "" Protocol/Transport: "multi-GRE/IP", Protect "ISAKMP_IPSEC_PROFILE" Interface State Control: Enabled nhrp event-publisher : Disabled Type:Hub, Total NBMA Peers (v4/v6): 2 # Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb Target Network ----- --------------- --------------- ----- -------- ----- ----------------- 1 1.1.1.2 10.208.0.2 UP 00:01:16 DN 10.208.0.2/32 Claimed Addr. 10.0.0.2 1 1.1.1.3 10.208.0.3 UP 00:01:16 DN 10.208.0.3/32 Claimed Addr. 10.0.0.2 Crypto Session Details: -------------------------------------------------------------------------------- Interface: Tunnel10 Session: [0xE5DF97F0] Session ID: 0 IKEv1 SA: local 1.1.1.1/4500 remote 1.1.1.2/4500 Active Capabilities:N connid:1001 lifetime:23:58:32 Crypto Session Status: UP-ACTIVE fvrf: (none), Phase1_id: 10.0.0.2 IPSEC FLOW: permit 47 host 1.1.1.1 host 1.1.1.2 Active SAs: 2, origin: crypto map Inbound: #pkts dec'ed 13 drop 0 life (KB/Sec) 4373572/3523 Outbound: #pkts enc'ed 15 drop 0 life (KB/Sec) 4373572/3523 Outbound SPI : 0xA1091D4F, transform : esp-256-aes esp-sha256-hmac Socket State: Open Interface: Tunnel10 Session: [0xE5DF96F8] Session ID: 0 IKEv1 SA: local 1.1.1.1/4500 remote 1.1.1.3/4500 Active Capabilities:N connid:1002 lifetime:23:58:33 Crypto Session Status: UP-ACTIVE fvrf: (none), Phase1_id: 10.0.0.2 IPSEC FLOW: permit 47 host 1.1.1.1 host 1.1.1.3 Active SAs: 2, origin: crypto map Inbound: #pkts dec'ed 12 drop 0 life (KB/Sec) 4188666/3523 Outbound: #pkts enc'ed 15 drop 0 life (KB/Sec) 4188665/3523 Outbound SPI : 0x10D8DD44, transform : esp-256-aes esp-sha256-hmac Socket State: Open Pending DMVPN Sessions:
IKEv2 unstable:
HUB(config-if)#do sh dmv det Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete N - NATed, L - Local, X - No Socket T1 - Route Installed, T2 - Nexthop-override C - CTS Capable # Ent --> Number of NHRP entries with same NBMA peer NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting UpDn Time --> Up or Down Time for a Tunnel ========================================================================== Interface Tunnel10 is up/up, Addr. is 10.208.0.1, VRF "" Tunnel Src./Dest. addr: 1.1.1.1/MGRE, Tunnel VRF "" Protocol/Transport: "multi-GRE/IP", Protect "IKEV2_IPSEC_PROFILE" Interface State Control: Enabled nhrp event-publisher : Disabled Type:Hub, Total NBMA Peers (v4/v6): 2 # Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb Target Network ----- --------------- --------------- ----- -------- ----- ----------------- 1 UNKNOWN 10.208.0.2 NHRP never IX 10.208.0.2/32 1 1.1.1.3 10.208.0.3 UP 00:00:32 DN 10.208.0.3/32 Claimed Addr. 10.0.0.2 Crypto Session Details: -------------------------------------------------------------------------------- Interface: Tunnel10 Session: [0xE5DF97F0] Session ID: 5 IKEv2 SA: local 1.1.1.1/4500 remote 1.1.1.3/4500 Active Capabilities:N connid:2 lifetime:23:59:27 Crypto Session Status: UP-ACTIVE fvrf: (none), Phase1_id: 10.0.0.2 IPSEC FLOW: permit 47 host 1.1.1.1 host 1.1.1.3 Active SAs: 2, origin: crypto map Inbound: #pkts dec'ed 9 drop 0 life (KB/Sec) 4248630/3567 Outbound: #pkts enc'ed 11 drop 0 life (KB/Sec) 4248630/3567 Outbound SPI : 0x12931605, transform : esp-256-aes esp-sha256-hmac Socket State: Open Pending DMVPN Sessions: HUB(config-if)#
Logs:
*Mar 25 13:45:59.341: %DUAL-5-NBRCHANGE: EIGRP-IPv4 400: Neighbor 10.208.0.3 (Tunnel10) is up: new adjacency *Mar 25 13:46:22.951: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=1.1.1.1, prot=50, spi=0xE6615F3A(3865141050), srcaddr=1.1.1.2, input interface=Ethernet0/0 *Mar 25 13:46:43.541: %DUAL-5-NBRCHANGE: EIGRP-IPv4 400: Neighbor 10.208.0.2 (Tunnel10) is down: holding time expired *Mar 25 13:47:02.147: %DUAL-5-NBRCHANGE: EIGRP-IPv4 400: Neighbor 10.208.0.2 (Tunnel10) is up: new adjacency *Mar 25 13:47:39.955: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=1.1.1.1, prot=50, spi=0xC2756891(3262474385), srcaddr=1.1.1.3, input interface=Ethernet0/0 *Mar 25 13:47:43.653: %DUAL-5-NBRCHANGE: EIGRP-IPv4 400: Neighbor 10.208.0.3 (Tunnel10) is down: holding time expired *Mar 25 13:48:19.962: %DUAL-5-NBRCHANGE: EIGRP-IPv4 400: Neighbor 10.208.0.3 (Tunnel10) is up: new adjacency *Mar 25 13:48:41.831: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=1.1.1.1, prot=50, spi=0x2C09E01(46177793), srcaddr=1.1.1.2, input interface=Ethernet0/0 *Mar 25 13:49:03.691: %DUAL-5-NBRCHANGE: EIGRP-IPv4 400: Neighbor 10.208.0.2 (Tunnel10) is down: holding time expired *Mar 25 13:49:22.267: %DUAL-5-NBRCHANGE: EIGRP-IPv4 400: Neighbor 10.208.0.2 (Tunnel10) is up: new adjacency *Mar 25 13:50:01.207: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=1.1.1.1, prot=50, spi=0x933A4A1C(2470070812), srcaddr=1.1.1.3, input interface=Ethernet0/0 *Mar 25 13:50:03.796: %DUAL-5-NBRCHANGE: EIGRP-IPv4 400: Neighbor 10.208.0.3 (Tunnel10) is down: holding time expired
Solved! Go to Solution.
03-26-2018 09:23 AM
Thank you all for your inputs, finally I found out what is the issue during long debug session. IKEv2 is using local IDs to create IKE PAIRs. By default it uses it's outgoing WAN IP (which is in my situation same). When I changed local ID of one peer it became stable.
*Mar 26 15:55:07.158: IKEv2:(SESSION ID = 28,SA ID = 1):Session with IKE ID PAIR (10.0.0.2, 1.1.1.1) is UP
HUB#sh cry ikev2 sa detailed
IPv4 Crypto IKEv2 SA
Tunnel-id Local Remote fvrf/ivrf Status
2 1.1.1.1/4500 1.1.1.3/4500 none/none READY
Encr: AES-CBC, keysize: 256, PRF: SHA256, Hash: SHA256, DH Grp:5, Auth sign: PSK, Auth verify: PSK
Life/Active Time: 86400/189 sec
CE id: 1046, Session-id: 40
Status Description: Negotiation done
Local spi: 5419AF3A0493577D Remote spi: 7F53D536C545E790
Local id: 1.1.1.1
Remote id: 10.0.0.2
Local req msg id: 0 Remote req msg id: 2
Local next msg id: 0 Remote next msg id: 2
Local req queued: 0 Remote req queued: 2
Local window: 5 Remote window: 5
DPD configured for 0 seconds, retry 0
Fragmentation not configured.
Extended Authentication not configured.
NAT-T is detected outside
Cisco Trust Security SGT is disabled
Initiator of SA : No
Tunnel-id Local Remote fvrf/ivrf Status
1 1.1.1.1/4500 1.1.1.2/4500 none/none READY
Encr: AES-CBC, keysize: 256, PRF: SHA256, Hash: SHA256, DH Grp:5, Auth sign: PSK, Auth verify: PSK
Life/Active Time: 86400/109 sec
CE id: 1047, Session-id: 41
Status Description: Negotiation done
Local spi: CF10CBB7B38AB072 Remote spi: 62E4C391ED0154C2
Local id: 1.1.1.1
Remote id: 1.2.3.4
Local req msg id: 0 Remote req msg id: 2
Local next msg id: 0 Remote next msg id: 2
Local req queued: 0 Remote req queued: 2
Local window: 5 Remote window: 5
DPD configured for 0 seconds, retry 0
Fragmentation not configured.
Extended Authentication not configured.
NAT-T is detected outside
Cisco Trust Security SGT is disabled
Initiator of SA : No
IPv6 Crypto IKEv2 SA
Spoke1 config:
crypto ikev2 profile IKEV2_PROFILE
match identity remote address 0.0.0.0
identity local address 1.2.3.4
authentication remote pre-share
authentication local pre-share
keyring local IKEV2_KEY
dpd 30 5 on-demand
03-25-2018 10:05 AM
03-25-2018 10:50 AM - edited 03-25-2018 10:53 AM
Hi Mohammed,
public IP of spokes are different (according to drawn topology it is 1.1.1.2 and 1.1.1.3). But those ISP owned routers are doing NAT for their private ranges on LAN side and there is connected ISR4221 which from DHCP gets address 10.0.0.2. So both spokes has same private (post-NAT) IP and when they communicate to HUB they have different public IP.
03-26-2018 12:07 AM
LoL,
who said two spokes cant behind same ip public. look on my picture, thats 2 spoke on same ip public.
i look your comment without research.
03-26-2018 04:27 AM
but you are using different NBMA which makes it working. When I said same public IP, that is the NBMA not a natted IP like ADSL.
03-26-2018 04:55 AM
03-26-2018 05:15 AM
03-26-2018 12:50 AM
I think's that probelm with Peer NBMA Addr <== may be on spoke 1 and 2 are same. if u use modem from isp, this will gift u ip address on your dhcp interfaces. example. ISP A gift u ip 192.168.1.100 on your spoke router and ISP B gift U ip 192.168.1.100 too. so thats ip will be record on dmvpn HUB as Host IP
Peer: 114.125.121.129 port 4500
Session ID: 0
IKEv1 SA: local X.X.X.28/4500 remote 114.125.121.129/4500 Active
IPSEC FLOW: permit 47 host X.X.X.28 host 192.168.1.101 <== this ip local from isp and will be IP NMBA Addr.
My solution u can set static IP on interface Roouter your connect to ISP or u can request ISP to gift u ip local from other network.
Example : ISP A 192.168.1.100/24
ISP B 192.168.2.100/24
If u have probelm, lets me know
Thanks You
03-26-2018 01:11 AM
Hi Agussang,
Thank you for your reply. For sure it is one of solutions, to ask ISP to readdress LAN side. But it is not prefereble as it needs some interaction with other IT guys (it is not anymore plug-and-play solution). From my point of view there is some issue with CRYPTO IKEv2 configuration. Because when I run the same topology with ISAKMP IPSEC Profile tunnel protection, connection is stable even both spokes have same post-nat IP (you can check CLI output of DMVPN in first post - ISAKMP part).
I don't believe that IKEv2 doesn't support some design compared to IKEv1 (ISAKMP). I am convinced that I am missing something in config part what causes unstable IKEv2 deployment.
03-26-2018 01:40 AM
Last I see your configuration u use tunnel source Ethernet0/0 <== and u use same ip address 10.0.0.2/24 on spoke 1 and spoke 2. i think nothing wrong with your configuration.
If u want solution plug and play, u can use other modem in up router so u set Static IP on router and DHCP on your modem.
This Simple Topology
ISP A ==> Modem ISP DHCP ==> Modem additional Static IP ==> Router.
Last time i use this toplogy on 20 Branch Office and DMVPN can run normally, to modem additional i use fortigate so include IPS and Firewall.
03-26-2018 09:23 AM
Thank you all for your inputs, finally I found out what is the issue during long debug session. IKEv2 is using local IDs to create IKE PAIRs. By default it uses it's outgoing WAN IP (which is in my situation same). When I changed local ID of one peer it became stable.
*Mar 26 15:55:07.158: IKEv2:(SESSION ID = 28,SA ID = 1):Session with IKE ID PAIR (10.0.0.2, 1.1.1.1) is UP
HUB#sh cry ikev2 sa detailed
IPv4 Crypto IKEv2 SA
Tunnel-id Local Remote fvrf/ivrf Status
2 1.1.1.1/4500 1.1.1.3/4500 none/none READY
Encr: AES-CBC, keysize: 256, PRF: SHA256, Hash: SHA256, DH Grp:5, Auth sign: PSK, Auth verify: PSK
Life/Active Time: 86400/189 sec
CE id: 1046, Session-id: 40
Status Description: Negotiation done
Local spi: 5419AF3A0493577D Remote spi: 7F53D536C545E790
Local id: 1.1.1.1
Remote id: 10.0.0.2
Local req msg id: 0 Remote req msg id: 2
Local next msg id: 0 Remote next msg id: 2
Local req queued: 0 Remote req queued: 2
Local window: 5 Remote window: 5
DPD configured for 0 seconds, retry 0
Fragmentation not configured.
Extended Authentication not configured.
NAT-T is detected outside
Cisco Trust Security SGT is disabled
Initiator of SA : No
Tunnel-id Local Remote fvrf/ivrf Status
1 1.1.1.1/4500 1.1.1.2/4500 none/none READY
Encr: AES-CBC, keysize: 256, PRF: SHA256, Hash: SHA256, DH Grp:5, Auth sign: PSK, Auth verify: PSK
Life/Active Time: 86400/109 sec
CE id: 1047, Session-id: 41
Status Description: Negotiation done
Local spi: CF10CBB7B38AB072 Remote spi: 62E4C391ED0154C2
Local id: 1.1.1.1
Remote id: 1.2.3.4
Local req msg id: 0 Remote req msg id: 2
Local next msg id: 0 Remote next msg id: 2
Local req queued: 0 Remote req queued: 2
Local window: 5 Remote window: 5
DPD configured for 0 seconds, retry 0
Fragmentation not configured.
Extended Authentication not configured.
NAT-T is detected outside
Cisco Trust Security SGT is disabled
Initiator of SA : No
IPv6 Crypto IKEv2 SA
Spoke1 config:
crypto ikev2 profile IKEV2_PROFILE
match identity remote address 0.0.0.0
identity local address 1.2.3.4
authentication remote pre-share
authentication local pre-share
keyring local IKEV2_KEY
dpd 30 5 on-demand
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide