cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements
2797
Views
0
Helpful
10
Replies
Highlighted
Beginner

DMVPN Phase 3 + IKEv2 flapping issue, same pre-NAT IP

Hi guys,

we are deploying DMVPN solution and we faced issue with flapping. Two sites were flapping every minute. When spoke1 was up, spoke2 was down, after one minutes it was inverted (I think it was caused by DPD). After some time I found out that it is caused by same private IP on WAN interface from provider. When I changed it, it became stable.

topology.PNG

 

 

 

 

I was thinking that this design is supported so I tried to lab it a bit and play with that. I created simple topology. Firstly I tried it without tunnel protection. It was working and stable. Then I added tunnel protection with ISAKMP. This solution was also working and stable. Finally I changed configuration for tunnel protection with IKEv2 and again it was unstable.

 

Could you please advice me, if this design with same post-nat IP is supported with IKEv2 or if I have there some configuration problem? Configs are attached to this post. 

 

HUB - ASR-1001-X

SPOKEs - ISR4221

 

Thank you a lot for your time!

Martin

 

Crypto config:

 

crypto keyring ISAKMP_KEYRING
  pre-shared-key address 0.0.0.0 0.0.0.0 key cisco123
crypto isakmp policy 10
encr aes 256
hash sha256
authentication pre-share
crypto isakmp profile ISAKMP_PROFILE
keyring ISAKMP_KEYRING
match identity address 0.0.0.0
crypto ikev2 proposal IKE2_PROPOSAL encryption aes-cbc-256 integrity sha256 group 5 crypto ikev2 policy IKE2_POLICY proposal IKE2_PROPOSAL crypto ikev2 keyring IKEV2_KEY peer DMVPN address 0.0.0.0 0.0.0.0 pre-shared-key cisco123 ! crypto ikev2 profile IKEV2_PROFILE match identity remote address 0.0.0.0 authentication remote pre-share authentication local pre-share keyring local IKEV2_KEY dpd 30 5 on-demand crypto ipsec transform-set ESP-AES256-SHA256 esp-aes 256 esp-sha256-hmac mode transport
crypto ipsec profile ISAKMP_IPSEC_PROFILE set transform-set ESP-AES256-SHA256 set isakmp-profile ISAKMP_PROFILE crypto ipsec profile IKEV2_IPSEC_PROFILE set transform-set ESP-AES256-SHA256 set ikev2-profile IKEV2_PROFILE

interface Tunnel10
 tunnel protection ipsec profile ISAKMP_IPSEC_PROFILE/IKEV2_IPSEC_PROFILE

 

 

 

ISAKMP stable:

HUB(config-if)#do sh dmv det
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
        N - NATed, L - Local, X - No Socket
        T1 - Route Installed, T2 - Nexthop-override
        C - CTS Capable
        # Ent --> Number of NHRP entries with same NBMA peer
        NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
        UpDn Time --> Up or Down Time for a Tunnel
==========================================================================

Interface Tunnel10 is up/up, Addr. is 10.208.0.1, VRF ""
   Tunnel Src./Dest. addr: 1.1.1.1/MGRE, Tunnel VRF ""
   Protocol/Transport: "multi-GRE/IP", Protect "ISAKMP_IPSEC_PROFILE"
   Interface State Control: Enabled
   nhrp event-publisher : Disabled
Type:Hub, Total NBMA Peers (v4/v6): 2

# Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb    Target Network
----- --------------- --------------- ----- -------- ----- -----------------
    1 1.1.1.2              10.208.0.2    UP 00:01:16    DN      10.208.0.2/32
               Claimed Addr. 10.0.0.2
    1 1.1.1.3              10.208.0.3    UP 00:01:16    DN      10.208.0.3/32
               Claimed Addr. 10.0.0.2


Crypto Session Details:
--------------------------------------------------------------------------------

Interface: Tunnel10
Session: [0xE5DF97F0]
  Session ID: 0
  IKEv1 SA: local 1.1.1.1/4500 remote 1.1.1.2/4500 Active
          Capabilities:N connid:1001 lifetime:23:58:32
  Crypto Session Status: UP-ACTIVE
  fvrf: (none), Phase1_id: 10.0.0.2
  IPSEC FLOW: permit 47 host 1.1.1.1 host 1.1.1.2
        Active SAs: 2, origin: crypto map
        Inbound:  #pkts dec'ed 13 drop 0 life (KB/Sec) 4373572/3523
        Outbound: #pkts enc'ed 15 drop 0 life (KB/Sec) 4373572/3523
   Outbound SPI : 0xA1091D4F, transform : esp-256-aes esp-sha256-hmac
    Socket State: Open

Interface: Tunnel10
Session: [0xE5DF96F8]
  Session ID: 0
  IKEv1 SA: local 1.1.1.1/4500 remote 1.1.1.3/4500 Active
          Capabilities:N connid:1002 lifetime:23:58:33
  Crypto Session Status: UP-ACTIVE
  fvrf: (none), Phase1_id: 10.0.0.2
  IPSEC FLOW: permit 47 host 1.1.1.1 host 1.1.1.3
        Active SAs: 2, origin: crypto map
        Inbound:  #pkts dec'ed 12 drop 0 life (KB/Sec) 4188666/3523
        Outbound: #pkts enc'ed 15 drop 0 life (KB/Sec) 4188665/3523
   Outbound SPI : 0x10D8DD44, transform : esp-256-aes esp-sha256-hmac
    Socket State: Open

Pending DMVPN Sessions:

 

 

IKEv2 unstable:

 

HUB(config-if)#do sh dmv det
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
        N - NATed, L - Local, X - No Socket
        T1 - Route Installed, T2 - Nexthop-override
        C - CTS Capable
        # Ent --> Number of NHRP entries with same NBMA peer
        NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
        UpDn Time --> Up or Down Time for a Tunnel
==========================================================================

Interface Tunnel10 is up/up, Addr. is 10.208.0.1, VRF ""
   Tunnel Src./Dest. addr: 1.1.1.1/MGRE, Tunnel VRF ""
   Protocol/Transport: "multi-GRE/IP", Protect "IKEV2_IPSEC_PROFILE"
   Interface State Control: Enabled
   nhrp event-publisher : Disabled
Type:Hub, Total NBMA Peers (v4/v6): 2

# Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb    Target Network
----- --------------- --------------- ----- -------- ----- -----------------
    1 UNKNOWN              10.208.0.2  NHRP    never    IX      10.208.0.2/32
    1 1.1.1.3              10.208.0.3    UP 00:00:32    DN      10.208.0.3/32
               Claimed Addr. 10.0.0.2


Crypto Session Details:
--------------------------------------------------------------------------------

Interface: Tunnel10
Session: [0xE5DF97F0]
  Session ID: 5
  IKEv2 SA: local 1.1.1.1/4500 remote 1.1.1.3/4500 Active
          Capabilities:N connid:2 lifetime:23:59:27
  Crypto Session Status: UP-ACTIVE
  fvrf: (none), Phase1_id: 10.0.0.2
  IPSEC FLOW: permit 47 host 1.1.1.1 host 1.1.1.3
        Active SAs: 2, origin: crypto map
        Inbound:  #pkts dec'ed 9 drop 0 life (KB/Sec) 4248630/3567
        Outbound: #pkts enc'ed 11 drop 0 life (KB/Sec) 4248630/3567
   Outbound SPI : 0x12931605, transform : esp-256-aes esp-sha256-hmac
    Socket State: Open

Pending DMVPN Sessions:

HUB(config-if)#

 

 

 

Logs:

 

*Mar 25 13:45:59.341: %DUAL-5-NBRCHANGE: EIGRP-IPv4 400: Neighbor 10.208.0.3 (Tunnel10) is up: new adjacency
*Mar 25 13:46:22.951: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=1.1.1.1, prot=50, spi=0xE6615F3A(3865141050), srcaddr=1.1.1.2, input interface=Ethernet0/0
*Mar 25 13:46:43.541: %DUAL-5-NBRCHANGE: EIGRP-IPv4 400: Neighbor 10.208.0.2 (Tunnel10) is down: holding time expired
*Mar 25 13:47:02.147: %DUAL-5-NBRCHANGE: EIGRP-IPv4 400: Neighbor 10.208.0.2 (Tunnel10) is up: new adjacency
*Mar 25 13:47:39.955: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=1.1.1.1, prot=50, spi=0xC2756891(3262474385), srcaddr=1.1.1.3, input interface=Ethernet0/0
*Mar 25 13:47:43.653: %DUAL-5-NBRCHANGE: EIGRP-IPv4 400: Neighbor 10.208.0.3 (Tunnel10) is down: holding time expired
*Mar 25 13:48:19.962: %DUAL-5-NBRCHANGE: EIGRP-IPv4 400: Neighbor 10.208.0.3 (Tunnel10) is up: new adjacency
*Mar 25 13:48:41.831: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=1.1.1.1, prot=50, spi=0x2C09E01(46177793), srcaddr=1.1.1.2, input interface=Ethernet0/0
*Mar 25 13:49:03.691: %DUAL-5-NBRCHANGE: EIGRP-IPv4 400: Neighbor 10.208.0.2 (Tunnel10) is down: holding time expired
*Mar 25 13:49:22.267: %DUAL-5-NBRCHANGE: EIGRP-IPv4 400: Neighbor 10.208.0.2 (Tunnel10) is up: new adjacency
*Mar 25 13:50:01.207: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=1.1.1.1, prot=50, spi=0x933A4A1C(2470070812), srcaddr=1.1.1.3, input interface=Ethernet0/0
*Mar 25 13:50:03.796: %DUAL-5-NBRCHANGE: EIGRP-IPv4 400: Neighbor 10.208.0.3 (Tunnel10) is down: holding time expired

 

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted
Beginner

Thank you all for your inputs, finally I found out what is the issue during long debug session. IKEv2 is using local IDs to create IKE PAIRs. By default it uses it's outgoing WAN IP (which is in my situation same). When I changed local ID of one peer it became stable.

 

*Mar 26 15:55:07.158: IKEv2:(SESSION ID = 28,SA ID = 1):Session with IKE ID PAIR (10.0.0.2, 1.1.1.1) is UP


HUB#sh cry ikev2 sa detailed
IPv4 Crypto IKEv2 SA

Tunnel-id Local Remote fvrf/ivrf Status
2 1.1.1.1/4500 1.1.1.3/4500 none/none READY
Encr: AES-CBC, keysize: 256, PRF: SHA256, Hash: SHA256, DH Grp:5, Auth sign: PSK, Auth verify: PSK
Life/Active Time: 86400/189 sec
CE id: 1046, Session-id: 40
Status Description: Negotiation done
Local spi: 5419AF3A0493577D Remote spi: 7F53D536C545E790
Local id: 1.1.1.1
Remote id: 10.0.0.2
Local req msg id: 0 Remote req msg id: 2
Local next msg id: 0 Remote next msg id: 2
Local req queued: 0 Remote req queued: 2
Local window: 5 Remote window: 5
DPD configured for 0 seconds, retry 0
Fragmentation not configured.
Extended Authentication not configured.
NAT-T is detected outside
Cisco Trust Security SGT is disabled
Initiator of SA : No

Tunnel-id Local Remote fvrf/ivrf Status
1 1.1.1.1/4500 1.1.1.2/4500 none/none READY
Encr: AES-CBC, keysize: 256, PRF: SHA256, Hash: SHA256, DH Grp:5, Auth sign: PSK, Auth verify: PSK
Life/Active Time: 86400/109 sec
CE id: 1047, Session-id: 41
Status Description: Negotiation done
Local spi: CF10CBB7B38AB072 Remote spi: 62E4C391ED0154C2
Local id: 1.1.1.1
Remote id: 1.2.3.4
Local req msg id: 0 Remote req msg id: 2
Local next msg id: 0 Remote next msg id: 2
Local req queued: 0 Remote req queued: 2
Local window: 5 Remote window: 5
DPD configured for 0 seconds, retry 0
Fragmentation not configured.
Extended Authentication not configured.
NAT-T is detected outside
Cisco Trust Security SGT is disabled
Initiator of SA : No

IPv6 Crypto IKEv2 SA

Spoke1 config:
crypto ikev2 profile IKEV2_PROFILE
match identity remote address 0.0.0.0
identity local address 1.2.3.4
authentication remote pre-share
authentication local pre-share
keyring local IKEV2_KEY
dpd 30 5 on-demand

View solution in original post

10 REPLIES 10
Highlighted
VIP Advisor

You can't have two spokes behind same public IP. That won't work. HUB won't
be able to distinguish between them.
Highlighted

Hi Mohammed,

public IP of spokes are different (according to drawn topology it is 1.1.1.2 and 1.1.1.3). But those ISP owned routers are doing NAT for their private ranges on LAN side and there is connected ISR4221 which from DHCP gets address 10.0.0.2. So both spokes has same private (post-NAT) IP and when they communicate to HUB they have different public IP.

Highlighted

LoL,

 

who said two spokes cant behind same ip public. look on my picture, thats 2 spoke on same ip public.
i look your comment without research.Screen Shot 2018-03-26 at 14.01.56.png

Highlighted

but you are using different NBMA which makes it working. When I said same public IP, that is the NBMA not a natted IP like ADSL. 

Highlighted

NBMA is NBMA
IP PUBLIC is IP PUBLIC

Any alibi again ?
Highlighted

-##I don't need to give you any. I know my capabilities and don't need
someone like you to judge
Highlighted
Beginner

I think's that probelm with Peer NBMA Addr <== may be on spoke 1 and 2 are same. if u use modem from isp, this will gift u ip address on your dhcp interfaces. example. ISP A gift u ip 192.168.1.100 on your spoke router and ISP B gift U ip 192.168.1.100 too. so thats ip will be record on dmvpn HUB as Host IP

 

Peer: 114.125.121.129 port 4500 

  Session ID: 0  

  IKEv1 SA: local X.X.X.28/4500 remote 114.125.121.129/4500 Active 

  IPSEC FLOW: permit 47 host X.X.X.28 host 192.168.1.101 <== this ip local from isp and will be IP NMBA Addr.

My solution u can set static IP on interface Roouter your connect to ISP or u can request ISP to gift u ip local from other network.

Example : ISP A 192.168.1.100/24

                ISP B 192.168.2.100/24

 

If u have probelm, lets me know


Thanks You

Highlighted

Hi Agussang,

Thank you for your reply. For sure it is one of solutions, to ask ISP to readdress LAN side. But it is not prefereble as it needs some interaction with other IT guys (it is not anymore plug-and-play solution). From my point of view there is some issue with CRYPTO IKEv2 configuration. Because when I run the same topology with ISAKMP IPSEC Profile tunnel protection, connection is stable even both spokes have same post-nat IP (you can check CLI output of DMVPN in first post - ISAKMP part).

 

I don't believe that IKEv2 doesn't support some design compared to IKEv1 (ISAKMP). I am convinced that I am missing something in config part what causes unstable IKEv2 deployment.

Highlighted

Last I see your configuration u use  tunnel source Ethernet0/0 <== and u use same ip address 10.0.0.2/24 on spoke 1 and spoke 2. i think nothing wrong with your configuration.

If u want solution plug and play, u can use other modem in up router so u set Static IP on router and DHCP on your modem.

This Simple Topology

ISP A ==> Modem ISP DHCP ==> Modem additional Static IP ==> Router.

Last time i use this toplogy on 20 Branch Office and DMVPN can run normally, to modem additional i use fortigate so include IPS and Firewall.

Highlighted
Beginner

Thank you all for your inputs, finally I found out what is the issue during long debug session. IKEv2 is using local IDs to create IKE PAIRs. By default it uses it's outgoing WAN IP (which is in my situation same). When I changed local ID of one peer it became stable.

 

*Mar 26 15:55:07.158: IKEv2:(SESSION ID = 28,SA ID = 1):Session with IKE ID PAIR (10.0.0.2, 1.1.1.1) is UP


HUB#sh cry ikev2 sa detailed
IPv4 Crypto IKEv2 SA

Tunnel-id Local Remote fvrf/ivrf Status
2 1.1.1.1/4500 1.1.1.3/4500 none/none READY
Encr: AES-CBC, keysize: 256, PRF: SHA256, Hash: SHA256, DH Grp:5, Auth sign: PSK, Auth verify: PSK
Life/Active Time: 86400/189 sec
CE id: 1046, Session-id: 40
Status Description: Negotiation done
Local spi: 5419AF3A0493577D Remote spi: 7F53D536C545E790
Local id: 1.1.1.1
Remote id: 10.0.0.2
Local req msg id: 0 Remote req msg id: 2
Local next msg id: 0 Remote next msg id: 2
Local req queued: 0 Remote req queued: 2
Local window: 5 Remote window: 5
DPD configured for 0 seconds, retry 0
Fragmentation not configured.
Extended Authentication not configured.
NAT-T is detected outside
Cisco Trust Security SGT is disabled
Initiator of SA : No

Tunnel-id Local Remote fvrf/ivrf Status
1 1.1.1.1/4500 1.1.1.2/4500 none/none READY
Encr: AES-CBC, keysize: 256, PRF: SHA256, Hash: SHA256, DH Grp:5, Auth sign: PSK, Auth verify: PSK
Life/Active Time: 86400/109 sec
CE id: 1047, Session-id: 41
Status Description: Negotiation done
Local spi: CF10CBB7B38AB072 Remote spi: 62E4C391ED0154C2
Local id: 1.1.1.1
Remote id: 1.2.3.4
Local req msg id: 0 Remote req msg id: 2
Local next msg id: 0 Remote next msg id: 2
Local req queued: 0 Remote req queued: 2
Local window: 5 Remote window: 5
DPD configured for 0 seconds, retry 0
Fragmentation not configured.
Extended Authentication not configured.
NAT-T is detected outside
Cisco Trust Security SGT is disabled
Initiator of SA : No

IPv6 Crypto IKEv2 SA

Spoke1 config:
crypto ikev2 profile IKEV2_PROFILE
match identity remote address 0.0.0.0
identity local address 1.2.3.4
authentication remote pre-share
authentication local pre-share
keyring local IKEV2_KEY
dpd 30 5 on-demand

View solution in original post

Content for Community-Ad