cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
5500
Views
0
Helpful
12
Replies

DMVPN Spoke Issues after migrating dual hub from ISR2 3925 to ASR-1001X

k.carlbark
Level 1
Level 1

Hello world,

 

After migrating our dual DMVPN hub solution from ISR2 3925 to ASR-1001X (running asr1001x-universalk9.03.12.03.S.154-2.S3-std.SPA.bin) we started having some issues with spokes tunnels flapping (going up and down) and sometime never come up.

Running 'show dmvpn' the spoke is stuck in NHRP state toward our hub. To resolve the issue we run 'shutdown' and then 'no shutdown' on the tunnel interface of the spoke the DMVPN goes up. Also running 'clear crypto session <remote hub nbma>' on the spoke often solves the problem. So it seems the issue has something to do with IPSEC.

When the problem is occurring and when debugging crypto ipsec, crypto socket, crypto isakmp and crypto engine the following can be seen on the hub:

Jun 25 10:01:41 SUMMERT: ISAKMP:(46580):Sending NOTIFY DPD/R_U_THERE protocol 1
    spi 140130067548488, message ID = 629121681
Jun 25 10:01:41 SUMMERT: ISAKMP:(46580): seq. no 0x64B2238C
Jun 25 10:01:41 SUMMERT: ISAKMP:(46580): sending packet to <spoke nbma> my_port 500 peer_port 500 (I) QM_IDLE
Jun 25 10:01:41 SUMMERT: ISAKMP:(46580):Sending an IKE IPv4 Packet.
Jun 25 10:01:41 SUMMERT: ISAKMP:(46580):purging node 629121681
Jun 25 10:01:41 SUMMERT: ISAKMP:(46580):Input = IKE_MESG_FROM_TIMER, IKE_TIMER_IM_ALIVE
Jun 25 10:01:41 SUMMERT: ISAKMP:(46580):Old State = IKE_P1_COMPLETE  New State = IKE_P1_COMPLETE

Jun 25 10:01:41 SUMMERT: ISAKMP (46580): received packet from <spoke nbma> dport 500 sport 500 ISP1-DMVPN (I) QM_IDLE
Jun 25 10:01:41 SUMMERT: ISAKMP: set new node 3442686097 to QM_IDLE
Jun 25 10:01:41 SUMMERT: ISAKMP:(46580): processing HASH payload. message ID = 3442686097
Jun 25 10:01:41 SUMMERT: ISAKMP:(46580): processing NOTIFY DPD/R_U_THERE_ACK protocol 1
    spi 0, message ID = 3442686097, sa = 0x7F72986867D0
Jun 25 10:01:41 SUMMERT: ISAKMP:(46580): DPD/R_U_THERE_ACK received from peer <spoke nbma>, sequence 0x64B2238C
Jun 25 10:01:41 SUMMERT: ISAKMP:(46580):deleting node 3442686097 error FALSE reason "Informational (in) state 1"
Jun 25 10:01:41 SUMMERT: ISAKMP:(46580):Input = IKE_MESG_FROM_PEER, IKE_INFO_NOTIFY
Jun 25 10:01:41 SUMMERT: ISAKMP:(46580):Old State = IKE_P1_COMPLETE  New State = IKE_P1_COMPLETE
Jun 25 10:01:42 SUMMERT: IPSEC: delete incomplete sa: 0x7F729923A438
Jun 25 10:01:42 SUMMERT: IPSEC(send_delete_notify_kmi): not sending KEY_ENGINE_DELETE_SAS
Jun 25 10:01:42 SUMMERT: ISAKMP:(46580):purging node 1111296046
Jun 25 10:01:44 SUMMERT: ISAKMP (46580): received packet from <spoke nbma> dport 500 sport 500 ISP1-DMVPN (I) QM_IDLE
Jun 25 10:01:44 SUMMERT: ISAKMP: set new node 928225319 to QM_IDLE
Jun 25 10:01:44 SUMMERT: ISAKMP:(46580): processing HASH payload. message ID = 928225319
Jun 25 10:01:44 SUMMERT: ISAKMP:(46580): processing SA payload. message ID = 928225319
Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):Checking IPSec proposal 1
Jun 25 10:01:44 SUMMERT: ISAKMP: transform 1, ESP_AES
Jun 25 10:01:44 SUMMERT: ISAKMP:   attributes in transform:
Jun 25 10:01:44 SUMMERT: ISAKMP:      encaps is 2 (Transport)
Jun 25 10:01:44 SUMMERT: ISAKMP:      SA life type in seconds
Jun 25 10:01:44 SUMMERT: ISAKMP:      SA life duration (basic) of 3600
Jun 25 10:01:44 SUMMERT: ISAKMP:      SA life type in kilobytes
Jun 25 10:01:44 SUMMERT: ISAKMP:      SA life duration (VPI) of  0x0 0x46 0x50 0x0
Jun 25 10:01:44 SUMMERT: ISAKMP:      authenticator is HMAC-SHA
Jun 25 10:01:44 SUMMERT: ISAKMP:      key length is 256
Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):atts are acceptable.
Jun 25 10:01:44 SUMMERT: CRYPTO_SS(TUNNEL SEC): Active open, socket info: local <hub nbma> <hub nbma>/255.255.255.255/0, remote <spoke nbma> <spoke nbma>/255.255.255.255/0,  prot 47, ifc Tu3300
Jun 25 10:01:44 SUMMERT: IPSEC(recalculate_mtu): reset sadb_root 7F7292E64990 mtu to 1500
Jun 25 10:01:44 SUMMERT: CRYPTO_SS(TUNNEL SEC): Sending Socket Ready message
Jun 25 10:01:44 SUMMERT: ISAKMP:(46580): processing NONCE payload. message ID = 928225319
Jun 25 10:01:44 SUMMERT: ISAKMP:(46580): processing ID payload. message ID = 928225319
Jun 25 10:01:44 SUMMERT: ISAKMP:(46580): processing ID payload. message ID = 928225319
Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):QM Responder gets spi
Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):Node 928225319, Input = IKE_MESG_FROM_PEER, IKE_QM_EXCH
Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):Old State = IKE_QM_READY  New State = IKE_QM_SPI_STARVE
Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):Node 928225319, Input = IKE_MESG_INTERNAL, IKE_GOT_SPI
Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):Old State = IKE_QM_SPI_STARVE  New State = IKE_QM_IPSEC_INSTALL_AWAIT
Jun 25 10:01:44 SUMMERT: IPSEC(crypto_ipsec_sa_find_ident_head): reconnecting with the same proxies and peer <spoke nbma>
Jun 25 10:01:44 SUMMERT: IPSEC(crypto_ipsec_update_ident_tunnel_decap_oce): updating profile-shared Tunnel3300 ident 7F7298B2BF80 with lookup_oce 7F7296BF5440
Jun 25 10:01:44 SUMMERT: IPSEC(create_sa): sa created,
  (sa) sa_dest= <hub nbma>, sa_proto= 50,
    sa_spi= 0x14F40C56(351538262),
    sa_trans= esp-aes 256 esp-sha-hmac , sa_conn_id= 27873
    sa_lifetime(k/sec)= (4608000/3600),
  (identity) local= <hub nbma>:0, remote= <spoke nbma>:0,
    local_proxy= <hub nbma>/255.255.255.255/47/0,
    remote_proxy= <spoke nbma>/255.255.255.255/47/0
Jun 25 10:01:44 SUMMERT: IPSEC(create_sa): sa created,
  (sa) sa_dest= <spoke nbma>, sa_proto= 50,
    sa_spi= 0x3B4731D7(994521559),
    sa_trans= esp-aes 256 esp-sha-hmac , sa_conn_id= 27874
    sa_lifetime(k/sec)= (4608000/3600),
  (identity) local= <hub nbma>:0, remote= <spoke nbma>:0,
    local_proxy= <hub nbma>/255.255.255.255/47/0,
    remote_proxy= <spoke nbma>/255.255.255.255/47/0
Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):Received IPSec Install callback... proceeding with the negotiation
Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):Successfully installed IPSEC SA (SPI:0x14F40C56) on Tunnel3300
Jun 25 10:01:44 SUMMERT: ISAKMP:(46580): sending packet to <spoke nbma> my_port 500 peer_port 500 (I) QM_IDLE
Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):Sending an IKE IPv4 Packet.
Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):Node 928225319, Input = IKE_MESG_FROM_IPSEC, IPSEC_INSTALL_DONE
Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):Old State = IKE_QM_IPSEC_INSTALL_AWAIT  New State = IKE_QM_R_QM2
Jun 25 10:01:44 SUMMERT: ISAKMP (46580): received packet from <spoke nbma> dport 500 sport 500 ISP1-DMVPN (I) QM_IDLE
Jun 25 10:01:44 SUMMERT: ISAKMP: set new node 1979798297 to QM_IDLE
Jun 25 10:01:44 SUMMERT: ISAKMP:(46580): processing HASH payload. message ID = 1979798297
Jun 25 10:01:44 SUMMERT: ISAKMP:(46580): processing NOTIFY PROPOSAL_NOT_CHOSEN protocol 3
    spi 351538262, message ID = 1979798297, sa = 0x7F72986867D0
Jun 25 10:01:44 SUMMERT: ISAKMP:(46580): deleting spi 351538262 message ID = 928225319
Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):deleting node 928225319 error TRUE reason "Delete Larval"
Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):peer does not do paranoid keepalives.
Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):Enqueued KEY_MGR_DELETE_SAS for IPSEC SA (SPI:0x3B4731D7)
Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):deleting node 1979798297 error FALSE reason "Informational (in) state 1"
Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):Input = IKE_MESG_FROM_PEER, IKE_INFO_NOTIFY
Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):Old State = IKE_P1_COMPLETE  New State = IKE_P1_COMPLETE
Jun 25 10:01:44 SUMMERT: IPSEC: delete incomplete sa: 0x7F729923A340
Jun 25 10:01:44 SUMMERT: IPSEC(key_engine_delete_sas): delete SA with spi 0x3B4731D7 proto 50 for <spoke nbma>
Jun 25 10:01:44 SUMMERT: IPSEC(update_current_outbound_sa): updated peer <spoke nbma> current outbound sa to SPI 0
Jun 25 10:01:44 SUMMERT: IPSEC(send_delete_notify_kmi): not sending KEY_ENGINE_DELETE_SAS
Jun 25 10:01:44 SUMMERT: CRYPTO_SS(TUNNEL SEC): Sending request for CRYPTO SS CLOSE SOCKET<spoke nbma>

 

#sh pl ha qf ac fe ipsec data drop
------------------------------------------------------------------------
Drop Type  Name                                     Packets
------------------------------------------------------------------------
        3  IN_US_V4_PKT_FOUND_IPSEC_NOT_ENABLED                   127672
       19  IN_OCT_ANTI_REPLAY_FAIL                                 13346
       20  IN_UNEXP_OCT_EXCEPTION                                   4224

       33  OUT_V4_PKT_HIT_IKE_START_SP                              1930

       62  IN_OCT_MAC_EXCEPTION                                        9

#sh plat hard qfp act stat drop | e _0_
-------------------------------------------------------------------------
Global Drop Stats                         Packets                  Octets
-------------------------------------------------------------------------
Disabled                                        1                      82
IpFragErr                                  170536               246635169
IpTtlExceeded                                4072                  343853
IpsecIkeIndicate                             1930                  269694
IpsecInput                                 145256                30071488
Ipv4Acl                                   2251965               215240194
Ipv4Martian                                  6248                  692010
Ipv4NoAdj                                   43188                 7627131
Ipv4NoRoute                                   278                   27913
Ipv4Unclassified                                6                     378
MplsNoRoute                                   790                   69130
MplsUnclassified                                1                      60
ReassTimeout                                   63                   10156
ServiceWireHdrErr                            2684                  585112

 

Also, after running 'logging dmvpn rate-limit 20' on the hub

 

 %DMVPN-3-DMVPN_NHRP_ERROR:  Tunnel292: NHRP Encap Error for  Resolution Request , Reason:  protocol generic error (7) on (Tunnel: <hub-tunnel-ip> NBMA: <hub nbma>)

 

On the spoke at the time the following can be seen from debugging as well:

*Jun 25 09:17:26.884: ISAKMP:(1032): sitting IDLE. Starting QM immediately (QM_IDLE      )
*Jun 25 09:17:26.884: ISAKMP:(1032):beginning Quick Mode exchange, M-ID of 1599359281
*Jun 25 09:17:26.884: ISAKMP:(1032):QM Initiator gets spi
*Jun 25 09:17:26.884: ISAKMP:(1032): sending packet to <hub nbma> my_port 500 peer_port 500 (R) QM_IDLE
*Jun 25 09:17:26.884: ISAKMP:(1032):Sending an IKE IPv4 Packet.
*Jun 25 09:17:26.884: ISAKMP:(1032):Node 1599359281, Input = IKE_MESG_INTERNAL, IKE_INIT_QM
*Jun 25 09:17:26.884: ISAKMP:(1032):Old State = IKE_QM_READY  New State = IKE_QM_I_QM1
*Jun 25 09:17:26.940: ISAKMP (1032): received packet from <hub nbma> dport 500 sport 500 Global (R) QM_IDLE
*Jun 25 09:17:26.940: ISAKMP:(1032): processing HASH payload. message ID = 1599359281
*Jun 25 09:17:26.940: ISAKMP:(1032): processing SA payload. message ID = 1599359281
*Jun 25 09:17:26.940: ISAKMP:(1032):Checking IPSec proposal 1
*Jun 25 09:17:26.940: ISAKMP: transform 1, ESP_AES
*Jun 25 09:17:26.940: ISAKMP:   attributes in transform:
*Jun 25 09:17:26.940: ISAKMP:      encaps is 2 (Transport)
*Jun 25 09:17:26.940: ISAKMP:      SA life type in seconds
*Jun 25 09:17:26.940: ISAKMP:      SA life duration (basic) of 3600
*Jun 25 09:17:26.940: ISAKMP:      SA life type in kilobytes
*Jun 25 09:17:26.940: ISAKMP:      SA life duration (VPI) of  0x0 0x46 0x50 0x0
*Jun 25 09:17:26.940: ISAKMP:      authenticator is HMAC-SHA
*Jun 25 09:17:26.940: ISAKMP:      key length is 256
*Jun 25 09:17:26.940: ISAKMP:(1032):atts are acceptable.
*Jun 25 09:17:26.940: IPSEC(ipsec_process_proposal): proxy identities not supported
*Jun 25 09:17:26.940: ISAKMP:(1032): IPSec policy invalidated proposal with error 32
*Jun 25 09:17:26.940: ISAKMP:(1032): phase 2 SA policy not acceptable! (local <spoke nbma> remote <hub nbma>)
*Jun 25 09:17:26.940: ISAKMP: set new node -1745931191 to QM_IDLE
*Jun 25 09:17:26.940: ISAKMP:(1032):Sending NOTIFY PROPOSAL_NOT_CHOSEN protocol 3
    spi 834718720, message ID = 2549036105
*Jun 25 09:17:26.940: ISAKMP:(1032): sending packet to <hub nbma> my_port 500 peer_port 500 (R) QM_IDLE
*Jun 25 09:17:26.940: ISAKMP:(1032):Sending an IKE IPv4 Packet.
*Jun 25 09:17:26.940: ISAKMP:(1032):purging node -1745931191
*Jun 25 09:17:26.940: ISAKMP:(1032):deleting node 1599359281 error TRUE reason "QM rejected"
*Jun 25 09:17:26.940: ISAKMP:(1032):Node 1599359281, Input = IKE_MESG_FROM_PEER, IKE_QM_EXCH
*Jun 25 09:17:26.940: ISAKMP:(1032):Old State = IKE_QM_I_QM1  New State = IKE_QM_I_QM1
*Jun 25 09:17:34.068: ISAKMP (1032): received packet from <hub nbma> dport 500 sport 500 Global (R) QM_IDLE
*Jun 25 09:17:34.068: ISAKMP: set new node 1021264821 to QM_IDLE
*Jun 25 09:17:34.072: ISAKMP:(1032): processing HASH payload. message ID = 1021264821
*Jun 25 09:17:34.072: ISAKMP:(1032): processing NOTIFY DPD/R_U_THERE protocol 1
    spi 0, message ID = 1021264821, sa = 0x32741028
*Jun 25 09:17:34.072: ISAKMP:(1032):deleting node 1021264821 error FALSE reason "Informational (in) state 1"
*Jun 25 09:17:34.072: ISAKMP:(1032):Input = IKE_MESG_FROM_PEER, IKE_INFO_NOTIFY
*Jun 25 09:17:34.072: ISAKMP:(1032):Old State = IKE_P1_COMPLETE  New State = IKE_P1_COMPLETE

*Jun 25 09:17:34.072: ISAKMP:(1032):DPD/R_U_THERE received from peer <hub nbma>, sequence 0x64B2279D
*Jun 25 09:17:34.072: ISAKMP: set new node 716440334 to QM_IDLE
*Jun 25 09:17:34.072: ISAKMP:(1032):Sending NOTIFY DPD/R_U_THERE_ACK protocol 1
    spi 834719464, message ID = 716440334
*Jun 25 09:17:34.072: ISAKMP:(1032): seq. no 0x64B2279D
*Jun 25 09:17:34.072: ISAKMP:(1032): sending packet to <hub nbma> my_port 500 peer_port 500 (R) QM_IDLE
*Jun 25 09:17:34.072: ISAKMP:(1032):Sending an IKE IPv4 Packet.
*Jun 25 09:17:34.072: ISAKMP:(1032):purging node 716440334
*Jun 25 09:17:34.072: ISAKMP:(1032):Input = IKE_MESG_FROM_PEER, IKE_MESG_KEEP_ALIVE
*Jun 25 09:17:34.072: ISAKMP:(1032):Old State = IKE_P1_COMPLETE  New State = IKE_P1_COMPLETE

*Jun 25 09:17:35.356: ISAKMP:(1032):purging node 206299144

 

Obviously something seems to be wrong with Phase 2 not coming up. But why is it going up after either clearing the crypto session or shutting down the tunnel interface and activate the tunnel interface of the spoke?

Very weird. Also, by looking att the debug-messages of the hub it seems the crypto is associated with the wrong tunnel interface Tu3300 when it should be Tu2010. Normal or Bug?

 

The configuration of the hub looks like this:

crypto keyring ISP1-DMVPN vrf ISP1-DMVPN
  pre-shared-key address 0.0.0.0 0.0.0.0 key <PSK>

crypto isakmp policy 10
 encr aes
 authentication pre-share

crypto isakmp keepalive 10 3 periodic
crypto isakmp nat keepalive 10

crypto isakmp profile ISP1-DMVPN
   keyring ISP1-DMVPN
   match identity address 0.0.0.0 ISP1-DMVPN
   keepalive 10 retry 3

crypto ipsec transform-set AES256-MD5 esp-aes 256 esp-md5-hmac
 mode tunnel

crypto ipsec transform-set AES256-SHA-TRANSPORT esp-aes 256 esp-sha-hmac
 mode transport

crypto ipsec profile ISP1-DMVPN
 set transform-set AES256-SHA AES256-SHA-TRANSPORT
 set isakmp-profile ISP1-DMVPN

vrf definition ISP1-DMVPN
 description DMVPN-Outside-ISP1
 rd 65527:10
 !
 address-family ipv4
 exit-address-family
!
!
interface TenGigabitEthernet0/0/0
 no ip address
!
interface TenGigabitEthernet0/0/0.71
 description VPN;ISP1-DMVPN;Outside;VLAN71
 encapsulation dot1Q 71
 vrf forwarding ISP1-DMVPN
 ip address <hub nbma> 255.255.255.128
 no ip proxy-arp
 ip access-group acl_ISP1-DMVPN_IN in
!
ip route vrf ISP1-DMVPN 0.0.0.0 0.0.0.0 <isp1 default gw> name ISP1;Default

ip access-list extended acl_ISP1-DMVPN_IN
 permit icmp any any
 permit esp any host <hub nbma>
 permit gre any host <hub nbma>
 permit udp any host <hub nbma> eq isakmp
 permit udp any host <hub nbma> eq non500-isakmp
 deny   ip any any

vrf definition 2010
 description CUSTA - Customer A
 rd 65527:2010
 route-target export 65527:2010
 route-target import 65527:2010
 !
 address-family ipv4
 exit-address-family
!
!
interface Tunnel2010
 description CUSTA;DMVPN;Failover-secondary
 vrf forwarding 2010
 ip address 10.97.0.34 255.255.255.240
 no ip redirects
 ip mtu 1380
 ip nhrp map multicast dynamic
 ip nhrp network-id 2010
 ip nhrp holdtime 120
 ip nhrp server-only
 ip nhrp max-send 1000 every 10
 ip tcp adjust-mss 1340
 tunnel source TenGigabitEthernet0/0/0.71
 tunnel mode gre multipoint
 tunnel key 2010
 tunnel vrf ISP1-DMVPN
 tunnel protection ipsec profile ISP1-DMVPN shared

router bgp 65527
 !
 address-family ipv4 vrf 2010
  redistribute connected metric 10
  redistribute static metric 15

  neighbor 10.97.0.39 remote-as 65028
  neighbor 10.97.0.39 description spokerouter;Tunnel1
  neighbor 10.97.0.39 update-source Tunnel2010
  neighbor 10.97.0.39 activate
  neighbor 10.97.0.39 soft-reconfiguration inbound
  neighbor 10.97.0.39 prefix-list EXPORT-IVPN-VRF2010 out
  neighbor 10.97.0.39 route-map AllVRF-LocalPref-80 in
  neighbor 10.97.0.39 maximum-prefix 5000 80
  default-information originate
 exit-address-family

 

The spoke configuration:

 

crypto keyring DMVPN01
  pre-shared-key address 0.0.0.0 0.0.0.0 key <PSK>

crypto isakmp policy 10
 encr aes
 authentication pre-share

crypto isakmp invalid-spi-recovery

crypto isakmp profile DMVPN01
   keyring DMVPN01
   match identity address 0.0.0.0
   keepalive 10 retry 3

crypto ipsec transform-set AES256-SHA esp-aes 256 esp-sha-hmac
 mode tunnel
crypto ipsec transform-set AES256-SHA-TRANSPORT esp-aes 256 esp-sha-hmac
 mode transport

crypto ipsec profile DMVPN01
 set transform-set AES256-SHA-TRANSPORT
 set isakmp-profile DMVPN01

vrf definition inside
 rd 65028:1
 route-target export 65028:1
 route-target import 65028:1
 !
 address-family ipv4
 exit-address-family
!

interface Tunnel1
 description DMVPN to HUB
 vrf forwarding inside
 ip address 10.97.0.39 255.255.255.240
 no ip redirects
 ip mtu 1380
 ip nhrp map 10.97.0.33 <hub1 nbma>
 ip nhrp map multicast <hub1 nbma>
 ip nhrp map 10.97.0.34 <hub2 nbma>
 ip nhrp map multicast <hub2 nbma>
 ip nhrp network-id 1
 ip nhrp holdtime 120
 ip nhrp nhs 10.97.0.33
 ip nhrp nhs 10.97.0.34
 ip nhrp registration no-unique
 ip nhrp registration timeout 60
 ip tcp adjust-mss 1340
 tunnel source GigabitEthernet0/0
 tunnel mode gre multipoint
 tunnel key 2010
 tunnel protection ipsec profile DMVPN01 shared

router bgp 65028
 !
 address-family ipv4 vrf inside
  bgp router-id 172.28.5.137
  network 10.97.20.128 mask 255.255.255.128
  network 10.97.21.0 mask 255.255.255.0
  network 10.97.22.0 mask 255.255.255.0
  network 10.97.23.0 mask 255.255.255.0
  network 172.28.5.137 mask 255.255.255.255
  neighbor 10.97.0.33 remote-as 65527
  neighbor 10.97.0.33 description HUB1;Tunnel2010
  neighbor 10.97.0.33 update-source Tunnel1
  neighbor 10.97.0.33 timers 10 30
  neighbor 10.97.0.33 activate
  neighbor 10.97.0.33 send-community both
  neighbor 10.97.0.33 soft-reconfiguration inbound
  neighbor 10.97.0.33 prefix-list IROUTE-EXPORT out
  neighbor 10.97.0.33 maximum-prefix 5000 80
  neighbor 10.97.0.34 remote-as 65527
  neighbor 10.97.0.34 description HUB2;tunnel2010
  neighbor 10.97.0.34 update-source Tunnel1
  neighbor 10.97.0.34 timers 10 30
  neighbor 10.97.0.34 activate
  neighbor 10.97.0.34 send-community both
  neighbor 10.97.0.34 soft-reconfiguration inbound
  neighbor 10.97.0.34 prefix-list IROUTE-EXPORT out
  neighbor 10.97.0.34 route-map AllVRF-LocalPref-80 in
  neighbor 10.97.0.34 maximum-prefix 5000 80
 exit-address-family

If more information is needed please say so.

Any help or guidance would be greatly appreciated!

Thanks!

1 Accepted Solution

Accepted Solutions

Marcin Latosiewicz
Cisco Employee
Cisco Employee

It's possible you're hitting this one - the phase 2 negotiation failure:

https://tools.cisco.com/bugsearch/bug/CSCup72039/?reffering_site=dumpcr

Too few details to say for sure :]

M.

View solution in original post

12 Replies 12

Marcin Latosiewicz
Cisco Employee
Cisco Employee

It's possible you're hitting this one - the phase 2 negotiation failure:

https://tools.cisco.com/bugsearch/bug/CSCup72039/?reffering_site=dumpcr

Too few details to say for sure :]

M.

Hmm, interesting. I will disable periodic DPD and see if that solves things.

What more information can I give you?

 

Thanks!

Try IKEv2? :-) You know it's good for you! 

 

Check whether you may or may not have multiple IKE sessions. 

And remember even changing DPDs configuration will not take immediate effect - you need to still re-key IKE sessions. In case you were thinking it's just as easy as changing DPD config. 

Hehe, will sure do! :)

Though we have around 200 spokes that needs to be reconfigured. It would be more satisfactory to solve the current issues without ikev2 :)

As a matter of fact there are multiple IKE sessions to this particular spoke and other as well. 64 as of this writing. Is this normal?

On the hub:

#show crypto ipsec sa peer <spoke nbma> | count current_peer
Number of lines which match regexp = 64
#

Also the IKE seems to be associated with the wrong tunnel interface.

Those are IPsec SA count, probably a bit too much, but most likely not all of those are active - a couple could have been introduced because of the problem we see in debugs (PROPOSAL_NOT_CHOSEN). Hard to say without actually seeing what they are ... but indeed it's uncommon for a spoke to have so many IPsec SAs (in DMVPN).

You'd need have a look at "show cry isa sa" output to check what IKE sessions exist. 

By looking at the "Status:" on the IPSEC SA's with this spoke the 64 are all active. Both inbound and outbound.

They say:

Status: ACTIVE(ACTIVE)

By looking at

show crypto ipsec sa peer <spoke nbma>

It seems it creates one IPSEC sa per tunnel interface on the hub (almost), when looking at the output of "Interface:"? Even though this particular spoke should be associated with just one tunnel interface (Tunnel2010). Is this normal when sharing the ipsec profile, perhaps?

#show crypto ipsec sa peer <spoke nbma> | inc interface:
interface: Tunnel3300
interface: Tunnel3260
interface: Tunnel3160
interface: Tunnel2931
interface: Tunnel2930
interface: Tunnel2818
interface: Tunnel2817
interface: Tunnel2816
interface: Tunnel2815
interface: Tunnel2814
interface: Tunnel2813
interface: Tunnel2812
interface: Tunnel2811
interface: Tunnel2750
interface: Tunnel2661
interface: Tunnel2660
interface: Tunnel2590
interface: Tunnel2470
interface: Tunnel2342
interface: Tunnel2341
interface: Tunnel2340
interface: Tunnel2220
interface: Tunnel2190
interface: Tunnel2180
interface: Tunnel2160
interface: Tunnel2011
interface: Tunnel2010
interface: Tunnel1933
interface: Tunnel1932
interface: Tunnel1931
interface: Tunnel1930
interface: Tunnel1820
interface: Tunnel1720
interface: Tunnel1700
interface: Tunnel1680
interface: Tunnel1670
interface: Tunnel1450
interface: Tunnel1370
interface: Tunnel1360
interface: Tunnel1050
interface: Tunnel871
interface: Tunnel870
interface: Tunnel850
interface: Tunnel710
interface: Tunnel550
interface: Tunnel545
interface: Tunnel542
interface: Tunnel540
interface: Tunnel490
interface: Tunnel480
interface: Tunnel470
interface: Tunnel469
interface: Tunnel468
interface: Tunnel461
interface: Tunnel460
interface: Tunnel430
interface: Tunnel410
interface: Tunnel380
interface: Tunnel360
interface: Tunnel310
interface: Tunnel292
interface: Tunnel250
interface: Tunnel2591

 

When looking at "show crypto isakmp sa" I saw a couple of duplicated entries to this spoke. This was fixed by clearing the crypto session. I've not seen it since then.

 

 

 

 

Ha, indeed I only saw one tunnel in the config you've outlined. 

There are a couple of interested (cosmetic and not) side effects when using "shared" ... that's why I prefer FlexVPN for those scenarios. 

I think some enhancements came to "shared" code relatively recently (1.5 years ago...) 

 

Regardless... any changes in behavior since moving to non-periodic DPDs? 

Yeah, sorry for leaving that "little" part out :)

I see. FlexVPN looks interesting. From what i've read it is difficult to implement this on an already DMVPN enabled device?

 

Yes actually, from what I can see the spokes have been more stable since moving to non-periodic DPD's and renegotiating the tunnels. I will still keep an eye on this though. Do you see anything else that I can improve or adjust regarding the configuration?

Thank you very much this far, Marcin. 

 

 

Well hard to say, I think you've for a pretty cool thing going. 

BGP as routing protocol already a big plus. 

VRF separation for ISPs and inside, I wish more people had that. 

Periodic DPDs are also a pretty good idea (overall). 

 

How many spokes are you terminating per ASR? (I assume you have two)

 

Maybe one suggestion is to use certificates instead of PSK ... there's a slight learning curve, but IOS devices can act as both DMVPN nodes and CAs ... I think there was/is a limitation that ASRs could/can not act as CAs.

Another ... your encryption suite is fine, but I'd probably look into updating it, should spokes support it ... AES-GCM is pretty sweet. Vide:

http://www.cisco.com/web/about/security/intelligence/nextgen_crypto.html#2

 

A couple of years back I wrote DM to Flex migration guide:

http://www.cisco.com/c/en/us/support/docs/security/flexvpn/116678-configure-product-00.html

Obviously it's for a very simple example. 

But what's good with newer software is that you can have co-existence of FlexVPN and DMVPN on same boxes (you'd need something newer than IOS XE 10.3 and IOS 15... I want to say 15.3T ). You can move between FlexVPN and DMVPN tunnels using BGP. 

Oh I wish we had unlimited time in the lab to play with things ;]

Cool.

Um, by running sh dmvpn and counting NBMA Peers there are around 200+ spokes per ASR.

 

Really cool, I will definitely look into this. Yeah, I agree with you. If you only had more time.. :)

What do you think of implementing if-state nhrp on the spokes? Would this help me in this scenario?

Lastly, what would you prefer to accomplish DMVPN-redundancy on the spoke, two tunnel interfaces to each hub with different tunnel subnets or just one tunnel interface with configuration for both hubs using same tunnel subnet?

Thanks,

 

 

 

 

 

 

200-300 per ASR seems decent, lots of possibility to scale up on most ASR1ks.

if-state nhrp ...  it's good for monitoring, and if you're running only spoke to hub traffic. If you do have some spoke to spoke you'll start losing the benefits (especially in case of failure). 

For redundancy on spokes I'm definitely a proponent of multi-cloud. But I do like things compartmentalized. It's easier to manage and make changes on - you know that changing X on tunnel Y will not impact tunnel Z --- or at least should not :}

It does take a bit more implementation effort but I find that it's easier to understand. 

Hey, a thought, even if you FlexVPN is not for you at the moment, don't write off IKEv2 ... DMVPN integrates with IKEv2! If you're thinking about implementing redundancy on spokes ... have on tunnel interface using IKEv1 another using IKEv2 - at least it out. :-)

All right, thanks for your input!

Regarding IKEv2 I will check this out!

Thanks for all the help and input. Greatly appreciate it. :)