Solved: Re: DMVPN Phase 3 IKEv2 flapping issue, same post-NAT IP

martinkiska · ‎03-25-2018

Hi guys,

we are deploying DMVPN solution and we faced issue with flapping. Two sites were flapping every minute. When spoke1 was up, spoke2 was down, after one minutes it was inverted (I think it was caused by DPD). After some time I found out that it is caused by same private IP on WAN interface from provider. When I changed it, it became stable.

I was thinking that this design is supported so I tried to lab it a bit and play with that. I created simple topology. Firstly I tried it without tunnel protection. It was working and stable. Then I added tunnel protection with ISAKMP. This solution was also working and stable. Finally I changed configuration for tunnel protection with IKEv2 and again it was unstable.

Could you please advice me, if this design with same post-nat IP is supported with IKEv2 or if I have there some configuration problem? Configs are attached to this post.

HUB - ASR-1001-X

SPOKEs - ISR4221

Thank you a lot for your time!

Martin

Crypto config:

crypto keyring ISAKMP_KEYRING
  pre-shared-key address 0.0.0.0 0.0.0.0 key cisco123
crypto isakmp policy 10
 encr aes 256
 hash sha256
 authentication pre-share
crypto isakmp profile ISAKMP_PROFILE
 keyring ISAKMP_KEYRING
 match identity address 0.0.0.0

crypto ikev2 proposal IKE2_PROPOSAL
 encryption aes-cbc-256
 integrity sha256
 group 5
crypto ikev2 policy IKE2_POLICY
 proposal IKE2_PROPOSAL
crypto ikev2 keyring IKEV2_KEY
 peer DMVPN
  address 0.0.0.0 0.0.0.0
  pre-shared-key cisco123
 !
crypto ikev2 profile IKEV2_PROFILE
 match identity remote address 0.0.0.0
 authentication remote pre-share
 authentication local pre-share
 keyring local IKEV2_KEY
 dpd 30 5 on-demand

crypto ipsec transform-set ESP-AES256-SHA256 esp-aes 256 esp-sha256-hmac
 mode transport

crypto ipsec profile ISAKMP_IPSEC_PROFILE
 set transform-set ESP-AES256-SHA256
 set isakmp-profile ISAKMP_PROFILE
crypto ipsec profile IKEV2_IPSEC_PROFILE
 set transform-set ESP-AES256-SHA256
 set ikev2-profile IKEV2_PROFILE

interface Tunnel10
 tunnel protection ipsec profile ISAKMP_IPSEC_PROFILE/IKEV2_IPSEC_PROFILE

ISAKMP stable:

HUB(config-if)#do sh dmv det
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
        N - NATed, L - Local, X - No Socket
        T1 - Route Installed, T2 - Nexthop-override
        C - CTS Capable
        # Ent --> Number of NHRP entries with same NBMA peer
        NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
        UpDn Time --> Up or Down Time for a Tunnel
==========================================================================

Interface Tunnel10 is up/up, Addr. is 10.208.0.1, VRF ""
   Tunnel Src./Dest. addr: 1.1.1.1/MGRE, Tunnel VRF ""
   Protocol/Transport: "multi-GRE/IP", Protect "ISAKMP_IPSEC_PROFILE"
   Interface State Control: Enabled
   nhrp event-publisher : Disabled
Type:Hub, Total NBMA Peers (v4/v6): 2

# Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb    Target Network
----- --------------- --------------- ----- -------- ----- -----------------
    1 1.1.1.2              10.208.0.2    UP 00:01:16    DN      10.208.0.2/32
               Claimed Addr. 10.0.0.2
    1 1.1.1.3              10.208.0.3    UP 00:01:16    DN      10.208.0.3/32
               Claimed Addr. 10.0.0.2


Crypto Session Details:
--------------------------------------------------------------------------------

Interface: Tunnel10
Session: [0xE5DF97F0]
  Session ID: 0
  IKEv1 SA: local 1.1.1.1/4500 remote 1.1.1.2/4500 Active
          Capabilities:N connid:1001 lifetime:23:58:32
  Crypto Session Status: UP-ACTIVE
  fvrf: (none), Phase1_id: 10.0.0.2
  IPSEC FLOW: permit 47 host 1.1.1.1 host 1.1.1.2
        Active SAs: 2, origin: crypto map
        Inbound:  #pkts dec'ed 13 drop 0 life (KB/Sec) 4373572/3523
        Outbound: #pkts enc'ed 15 drop 0 life (KB/Sec) 4373572/3523
   Outbound SPI : 0xA1091D4F, transform : esp-256-aes esp-sha256-hmac
    Socket State: Open

Interface: Tunnel10
Session: [0xE5DF96F8]
  Session ID: 0
  IKEv1 SA: local 1.1.1.1/4500 remote 1.1.1.3/4500 Active
          Capabilities:N connid:1002 lifetime:23:58:33
  Crypto Session Status: UP-ACTIVE
  fvrf: (none), Phase1_id: 10.0.0.2
  IPSEC FLOW: permit 47 host 1.1.1.1 host 1.1.1.3
        Active SAs: 2, origin: crypto map
        Inbound:  #pkts dec'ed 12 drop 0 life (KB/Sec) 4188666/3523
        Outbound: #pkts enc'ed 15 drop 0 life (KB/Sec) 4188665/3523
   Outbound SPI : 0x10D8DD44, transform : esp-256-aes esp-sha256-hmac
    Socket State: Open

Pending DMVPN Sessions:

IKEv2 unstable:

HUB(config-if)#do sh dmv det
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
        N - NATed, L - Local, X - No Socket
        T1 - Route Installed, T2 - Nexthop-override
        C - CTS Capable
        # Ent --> Number of NHRP entries with same NBMA peer
        NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
        UpDn Time --> Up or Down Time for a Tunnel
==========================================================================

Interface Tunnel10 is up/up, Addr. is 10.208.0.1, VRF ""
   Tunnel Src./Dest. addr: 1.1.1.1/MGRE, Tunnel VRF ""
   Protocol/Transport: "multi-GRE/IP", Protect "IKEV2_IPSEC_PROFILE"
   Interface State Control: Enabled
   nhrp event-publisher : Disabled
Type:Hub, Total NBMA Peers (v4/v6): 2

# Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb    Target Network
----- --------------- --------------- ----- -------- ----- -----------------
    1 UNKNOWN              10.208.0.2  NHRP    never    IX      10.208.0.2/32
    1 1.1.1.3              10.208.0.3    UP 00:00:32    DN      10.208.0.3/32
               Claimed Addr. 10.0.0.2


Crypto Session Details:
--------------------------------------------------------------------------------

Interface: Tunnel10
Session: [0xE5DF97F0]
  Session ID: 5
  IKEv2 SA: local 1.1.1.1/4500 remote 1.1.1.3/4500 Active
          Capabilities:N connid:2 lifetime:23:59:27
  Crypto Session Status: UP-ACTIVE
  fvrf: (none), Phase1_id: 10.0.0.2
  IPSEC FLOW: permit 47 host 1.1.1.1 host 1.1.1.3
        Active SAs: 2, origin: crypto map
        Inbound:  #pkts dec'ed 9 drop 0 life (KB/Sec) 4248630/3567
        Outbound: #pkts enc'ed 11 drop 0 life (KB/Sec) 4248630/3567
   Outbound SPI : 0x12931605, transform : esp-256-aes esp-sha256-hmac
    Socket State: Open

Pending DMVPN Sessions:

HUB(config-if)#

Logs:

*Mar 25 13:45:59.341: %DUAL-5-NBRCHANGE: EIGRP-IPv4 400: Neighbor 10.208.0.3 (Tunnel10) is up: new adjacency
*Mar 25 13:46:22.951: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=1.1.1.1, prot=50, spi=0xE6615F3A(3865141050), srcaddr=1.1.1.2, input interface=Ethernet0/0
*Mar 25 13:46:43.541: %DUAL-5-NBRCHANGE: EIGRP-IPv4 400: Neighbor 10.208.0.2 (Tunnel10) is down: holding time expired
*Mar 25 13:47:02.147: %DUAL-5-NBRCHANGE: EIGRP-IPv4 400: Neighbor 10.208.0.2 (Tunnel10) is up: new adjacency
*Mar 25 13:47:39.955: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=1.1.1.1, prot=50, spi=0xC2756891(3262474385), srcaddr=1.1.1.3, input interface=Ethernet0/0
*Mar 25 13:47:43.653: %DUAL-5-NBRCHANGE: EIGRP-IPv4 400: Neighbor 10.208.0.3 (Tunnel10) is down: holding time expired
*Mar 25 13:48:19.962: %DUAL-5-NBRCHANGE: EIGRP-IPv4 400: Neighbor 10.208.0.3 (Tunnel10) is up: new adjacency
*Mar 25 13:48:41.831: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=1.1.1.1, prot=50, spi=0x2C09E01(46177793), srcaddr=1.1.1.2, input interface=Ethernet0/0
*Mar 25 13:49:03.691: %DUAL-5-NBRCHANGE: EIGRP-IPv4 400: Neighbor 10.208.0.2 (Tunnel10) is down: holding time expired
*Mar 25 13:49:22.267: %DUAL-5-NBRCHANGE: EIGRP-IPv4 400: Neighbor 10.208.0.2 (Tunnel10) is up: new adjacency
*Mar 25 13:50:01.207: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=1.1.1.1, prot=50, spi=0x933A4A1C(2470070812), srcaddr=1.1.1.3, input interface=Ethernet0/0
*Mar 25 13:50:03.796: %DUAL-5-NBRCHANGE: EIGRP-IPv4 400: Neighbor 10.208.0.3 (Tunnel10) is down: holding time expired

martinkiska · ‎03-26-2018

Thank you all for your inputs, finally I found out what is the issue during long debug session. IKEv2 is using local IDs to create IKE PAIRs. By default it uses it's outgoing WAN IP (which is in my situation same). When I changed local ID of one peer it became stable.

*Mar 26 15:55:07.158: IKEv2:(SESSION ID = 28,SA ID = 1):Session with IKE ID PAIR (10.0.0.2, 1.1.1.1) is UP


HUB#sh cry ikev2 sa detailed
 IPv4 Crypto IKEv2 SA

Tunnel-id Local Remote fvrf/ivrf Status
2 1.1.1.1/4500 1.1.1.3/4500 none/none READY
 Encr: AES-CBC, keysize: 256, PRF: SHA256, Hash: SHA256, DH Grp:5, Auth sign: PSK, Auth verify: PSK
 Life/Active Time: 86400/189 sec
 CE id: 1046, Session-id: 40
 Status Description: Negotiation done
 Local spi: 5419AF3A0493577D Remote spi: 7F53D536C545E790
 Local id: 1.1.1.1
 Remote id: 10.0.0.2
 Local req msg id: 0 Remote req msg id: 2
 Local next msg id: 0 Remote next msg id: 2
 Local req queued: 0 Remote req queued: 2
 Local window: 5 Remote window: 5
 DPD configured for 0 seconds, retry 0
 Fragmentation not configured.
 Extended Authentication not configured.
 NAT-T is detected outside
 Cisco Trust Security SGT is disabled
 Initiator of SA : No

Tunnel-id Local Remote fvrf/ivrf Status
1 1.1.1.1/4500 1.1.1.2/4500 none/none READY
 Encr: AES-CBC, keysize: 256, PRF: SHA256, Hash: SHA256, DH Grp:5, Auth sign: PSK, Auth verify: PSK
 Life/Active Time: 86400/109 sec
 CE id: 1047, Session-id: 41
 Status Description: Negotiation done
 Local spi: CF10CBB7B38AB072 Remote spi: 62E4C391ED0154C2
 Local id: 1.1.1.1
 Remote id: 1.2.3.4
 Local req msg id: 0 Remote req msg id: 2
 Local next msg id: 0 Remote next msg id: 2
 Local req queued: 0 Remote req queued: 2
 Local window: 5 Remote window: 5
 DPD configured for 0 seconds, retry 0
 Fragmentation not configured.
 Extended Authentication not configured.
 NAT-T is detected outside
 Cisco Trust Security SGT is disabled
 Initiator of SA : No

 IPv6 Crypto IKEv2 SA

Spoke1 config:
crypto ikev2 profile IKEV2_PROFILE
 match identity remote address 0.0.0.0
 identity local address 1.2.3.4
 authentication remote pre-share
 authentication local pre-share
 keyring local IKEV2_KEY
 dpd 30 5 on-demand

View solution in original post

Mohammed al Baqari · ‎03-25-2018

You can't have two spokes behind same public IP. That won't work. HUB won't
be able to distinguish between them.

martinkiska · ‎03-25-2018

Hi Mohammed,

public IP of spokes are different (according to drawn topology it is 1.1.1.2 and 1.1.1.3). But those ISP owned routers are doing NAT for their private ranges on LAN side and there is connected ISR4221 which from DHCP gets address 10.0.0.2. So both spokes has same private (post-NAT) IP and when they communicate to HUB they have different public IP.

agussang · ‎03-26-2018

LoL,

who said two spokes cant behind same ip public. look on my picture, thats 2 spoke on same ip public.
i look your comment without research.

Mohammed al Baqari · ‎03-26-2018

but you are using different NBMA which makes it working. When I said same public IP, that is the NBMA not a natted IP like ADSL.

agussang · ‎03-26-2018

NBMA is NBMA
IP PUBLIC is IP PUBLIC

Any alibi again ?

Mohammed al Baqari · ‎03-26-2018

-##I don't need to give you any. I know my capabilities and don't need
someone like you to judge

agussang · ‎03-26-2018

I think's that probelm with Peer NBMA Addr <== may be on spoke 1 and 2 are same. if u use modem from isp, this will gift u ip address on your dhcp interfaces. example. ISP A gift u ip 192.168.1.100 on your spoke router and ISP B gift U ip 192.168.1.100 too. so thats ip will be record on dmvpn HUB as Host IP

Peer: 114.125.121.129 port 4500

Session ID: 0

IKEv1 SA: local X.X.X.28/4500 remote 114.125.121.129/4500 Active

IPSEC FLOW: permit 47 host X.X.X.28 host 192.168.1.101 <== this ip local from isp and will be IP NMBA Addr.

My solution u can set static IP on interface Roouter your connect to ISP or u can request ISP to gift u ip local from other network.

Example : ISP A 192.168.1.100/24

ISP B 192.168.2.100/24

If u have probelm, lets me know

Thanks You

martinkiska · ‎03-26-2018

Hi Agussang,

Thank you for your reply. For sure it is one of solutions, to ask ISP to readdress LAN side. But it is not prefereble as it needs some interaction with other IT guys (it is not anymore plug-and-play solution). From my point of view there is some issue with CRYPTO IKEv2 configuration. Because when I run the same topology with ISAKMP IPSEC Profile tunnel protection, connection is stable even both spokes have same post-nat IP (you can check CLI output of DMVPN in first post - ISAKMP part).

I don't believe that IKEv2 doesn't support some design compared to IKEv1 (ISAKMP). I am convinced that I am missing something in config part what causes unstable IKEv2 deployment.

agussang · ‎03-26-2018

Last I see your configuration u use tunnel source Ethernet0/0 <== and u use same ip address 10.0.0.2/24 on spoke 1 and spoke 2. i think nothing wrong with your configuration.

If u want solution plug and play, u can use other modem in up router so u set Static IP on router and DHCP on your modem.

This Simple Topology

ISP A ==> Modem ISP DHCP ==> Modem additional Static IP ==> Router.

Last time i use this toplogy on 20 Branch Office and DMVPN can run normally, to modem additional i use fortigate so include IPS and Firewall.

martinkiska · ‎03-26-2018

Thank you all for your inputs, finally I found out what is the issue during long debug session. IKEv2 is using local IDs to create IKE PAIRs. By default it uses it's outgoing WAN IP (which is in my situation same). When I changed local ID of one peer it became stable.

*Mar 26 15:55:07.158: IKEv2:(SESSION ID = 28,SA ID = 1):Session with IKE ID PAIR (10.0.0.2, 1.1.1.1) is UP


HUB#sh cry ikev2 sa detailed
 IPv4 Crypto IKEv2 SA

Tunnel-id Local Remote fvrf/ivrf Status
2 1.1.1.1/4500 1.1.1.3/4500 none/none READY
 Encr: AES-CBC, keysize: 256, PRF: SHA256, Hash: SHA256, DH Grp:5, Auth sign: PSK, Auth verify: PSK
 Life/Active Time: 86400/189 sec
 CE id: 1046, Session-id: 40
 Status Description: Negotiation done
 Local spi: 5419AF3A0493577D Remote spi: 7F53D536C545E790
 Local id: 1.1.1.1
 Remote id: 10.0.0.2
 Local req msg id: 0 Remote req msg id: 2
 Local next msg id: 0 Remote next msg id: 2
 Local req queued: 0 Remote req queued: 2
 Local window: 5 Remote window: 5
 DPD configured for 0 seconds, retry 0
 Fragmentation not configured.
 Extended Authentication not configured.
 NAT-T is detected outside
 Cisco Trust Security SGT is disabled
 Initiator of SA : No

Tunnel-id Local Remote fvrf/ivrf Status
1 1.1.1.1/4500 1.1.1.2/4500 none/none READY
 Encr: AES-CBC, keysize: 256, PRF: SHA256, Hash: SHA256, DH Grp:5, Auth sign: PSK, Auth verify: PSK
 Life/Active Time: 86400/109 sec
 CE id: 1047, Session-id: 41
 Status Description: Negotiation done
 Local spi: CF10CBB7B38AB072 Remote spi: 62E4C391ED0154C2
 Local id: 1.1.1.1
 Remote id: 1.2.3.4
 Local req msg id: 0 Remote req msg id: 2
 Local next msg id: 0 Remote next msg id: 2
 Local req queued: 0 Remote req queued: 2
 Local window: 5 Remote window: 5
 DPD configured for 0 seconds, retry 0
 Fragmentation not configured.
 Extended Authentication not configured.
 NAT-T is detected outside
 Cisco Trust Security SGT is disabled
 Initiator of SA : No

 IPv6 Crypto IKEv2 SA

Spoke1 config:
crypto ikev2 profile IKEV2_PROFILE
 match identity remote address 0.0.0.0
 identity local address 1.2.3.4
 authentication remote pre-share
 authentication local pre-share
 keyring local IKEV2_KEY
 dpd 30 5 on-demand

DMVPN Phase 3 + IKEv2 flapping issue, same pre-NAT IP