06-25-2015 05:34 AM - edited 02-21-2020 08:18 PM
Hello world,
After migrating our dual DMVPN hub solution from ISR2 3925 to ASR-1001X (running asr1001x-universalk9.03.12.03.S.154-2.S3-std.SPA.bin) we started having some issues with spokes tunnels flapping (going up and down) and sometime never come up.
Running 'show dmvpn' the spoke is stuck in NHRP state toward our hub. To resolve the issue we run 'shutdown' and then 'no shutdown' on the tunnel interface of the spoke the DMVPN goes up. Also running 'clear crypto session <remote hub nbma>' on the spoke often solves the problem. So it seems the issue has something to do with IPSEC.
When the problem is occurring and when debugging crypto ipsec, crypto socket, crypto isakmp and crypto engine the following can be seen on the hub:
Jun 25 10:01:41 SUMMERT: ISAKMP:(46580):Sending NOTIFY DPD/R_U_THERE protocol 1 spi 140130067548488, message ID = 629121681 Jun 25 10:01:41 SUMMERT: ISAKMP:(46580): seq. no 0x64B2238C Jun 25 10:01:41 SUMMERT: ISAKMP:(46580): sending packet to <spoke nbma> my_port 500 peer_port 500 (I) QM_IDLE Jun 25 10:01:41 SUMMERT: ISAKMP:(46580):Sending an IKE IPv4 Packet. Jun 25 10:01:41 SUMMERT: ISAKMP:(46580):purging node 629121681 Jun 25 10:01:41 SUMMERT: ISAKMP:(46580):Input = IKE_MESG_FROM_TIMER, IKE_TIMER_IM_ALIVE Jun 25 10:01:41 SUMMERT: ISAKMP:(46580):Old State = IKE_P1_COMPLETE New State = IKE_P1_COMPLETE Jun 25 10:01:41 SUMMERT: ISAKMP (46580): received packet from <spoke nbma> dport 500 sport 500 ISP1-DMVPN (I) QM_IDLE Jun 25 10:01:41 SUMMERT: ISAKMP: set new node 3442686097 to QM_IDLE Jun 25 10:01:41 SUMMERT: ISAKMP:(46580): processing HASH payload. message ID = 3442686097 Jun 25 10:01:41 SUMMERT: ISAKMP:(46580): processing NOTIFY DPD/R_U_THERE_ACK protocol 1 spi 0, message ID = 3442686097, sa = 0x7F72986867D0 Jun 25 10:01:41 SUMMERT: ISAKMP:(46580): DPD/R_U_THERE_ACK received from peer <spoke nbma>, sequence 0x64B2238C Jun 25 10:01:41 SUMMERT: ISAKMP:(46580):deleting node 3442686097 error FALSE reason "Informational (in) state 1" Jun 25 10:01:41 SUMMERT: ISAKMP:(46580):Input = IKE_MESG_FROM_PEER, IKE_INFO_NOTIFY Jun 25 10:01:41 SUMMERT: ISAKMP:(46580):Old State = IKE_P1_COMPLETE New State = IKE_P1_COMPLETE Jun 25 10:01:42 SUMMERT: IPSEC: delete incomplete sa: 0x7F729923A438 Jun 25 10:01:42 SUMMERT: IPSEC(send_delete_notify_kmi): not sending KEY_ENGINE_DELETE_SAS Jun 25 10:01:42 SUMMERT: ISAKMP:(46580):purging node 1111296046 Jun 25 10:01:44 SUMMERT: ISAKMP (46580): received packet from <spoke nbma> dport 500 sport 500 ISP1-DMVPN (I) QM_IDLE Jun 25 10:01:44 SUMMERT: ISAKMP: set new node 928225319 to QM_IDLE Jun 25 10:01:44 SUMMERT: ISAKMP:(46580): processing HASH payload. message ID = 928225319 Jun 25 10:01:44 SUMMERT: ISAKMP:(46580): processing SA payload. message ID = 928225319 Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):Checking IPSec proposal 1 Jun 25 10:01:44 SUMMERT: ISAKMP: transform 1, ESP_AES Jun 25 10:01:44 SUMMERT: ISAKMP: attributes in transform: Jun 25 10:01:44 SUMMERT: ISAKMP: encaps is 2 (Transport) Jun 25 10:01:44 SUMMERT: ISAKMP: SA life type in seconds Jun 25 10:01:44 SUMMERT: ISAKMP: SA life duration (basic) of 3600 Jun 25 10:01:44 SUMMERT: ISAKMP: SA life type in kilobytes Jun 25 10:01:44 SUMMERT: ISAKMP: SA life duration (VPI) of 0x0 0x46 0x50 0x0 Jun 25 10:01:44 SUMMERT: ISAKMP: authenticator is HMAC-SHA Jun 25 10:01:44 SUMMERT: ISAKMP: key length is 256 Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):atts are acceptable. Jun 25 10:01:44 SUMMERT: CRYPTO_SS(TUNNEL SEC): Active open, socket info: local <hub nbma> <hub nbma>/255.255.255.255/0, remote <spoke nbma> <spoke nbma>/255.255.255.255/0, prot 47, ifc Tu3300 Jun 25 10:01:44 SUMMERT: IPSEC(recalculate_mtu): reset sadb_root 7F7292E64990 mtu to 1500 Jun 25 10:01:44 SUMMERT: CRYPTO_SS(TUNNEL SEC): Sending Socket Ready message Jun 25 10:01:44 SUMMERT: ISAKMP:(46580): processing NONCE payload. message ID = 928225319 Jun 25 10:01:44 SUMMERT: ISAKMP:(46580): processing ID payload. message ID = 928225319 Jun 25 10:01:44 SUMMERT: ISAKMP:(46580): processing ID payload. message ID = 928225319 Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):QM Responder gets spi Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):Node 928225319, Input = IKE_MESG_FROM_PEER, IKE_QM_EXCH Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):Old State = IKE_QM_READY New State = IKE_QM_SPI_STARVE Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):Node 928225319, Input = IKE_MESG_INTERNAL, IKE_GOT_SPI Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):Old State = IKE_QM_SPI_STARVE New State = IKE_QM_IPSEC_INSTALL_AWAIT Jun 25 10:01:44 SUMMERT: IPSEC(crypto_ipsec_sa_find_ident_head): reconnecting with the same proxies and peer <spoke nbma> Jun 25 10:01:44 SUMMERT: IPSEC(crypto_ipsec_update_ident_tunnel_decap_oce): updating profile-shared Tunnel3300 ident 7F7298B2BF80 with lookup_oce 7F7296BF5440 Jun 25 10:01:44 SUMMERT: IPSEC(create_sa): sa created, (sa) sa_dest= <hub nbma>, sa_proto= 50, sa_spi= 0x14F40C56(351538262), sa_trans= esp-aes 256 esp-sha-hmac , sa_conn_id= 27873 sa_lifetime(k/sec)= (4608000/3600), (identity) local= <hub nbma>:0, remote= <spoke nbma>:0, local_proxy= <hub nbma>/255.255.255.255/47/0, remote_proxy= <spoke nbma>/255.255.255.255/47/0 Jun 25 10:01:44 SUMMERT: IPSEC(create_sa): sa created, (sa) sa_dest= <spoke nbma>, sa_proto= 50, sa_spi= 0x3B4731D7(994521559), sa_trans= esp-aes 256 esp-sha-hmac , sa_conn_id= 27874 sa_lifetime(k/sec)= (4608000/3600), (identity) local= <hub nbma>:0, remote= <spoke nbma>:0, local_proxy= <hub nbma>/255.255.255.255/47/0, remote_proxy= <spoke nbma>/255.255.255.255/47/0 Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):Received IPSec Install callback... proceeding with the negotiation Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):Successfully installed IPSEC SA (SPI:0x14F40C56) on Tunnel3300 Jun 25 10:01:44 SUMMERT: ISAKMP:(46580): sending packet to <spoke nbma> my_port 500 peer_port 500 (I) QM_IDLE Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):Sending an IKE IPv4 Packet. Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):Node 928225319, Input = IKE_MESG_FROM_IPSEC, IPSEC_INSTALL_DONE Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):Old State = IKE_QM_IPSEC_INSTALL_AWAIT New State = IKE_QM_R_QM2 Jun 25 10:01:44 SUMMERT: ISAKMP (46580): received packet from <spoke nbma> dport 500 sport 500 ISP1-DMVPN (I) QM_IDLE Jun 25 10:01:44 SUMMERT: ISAKMP: set new node 1979798297 to QM_IDLE Jun 25 10:01:44 SUMMERT: ISAKMP:(46580): processing HASH payload. message ID = 1979798297 Jun 25 10:01:44 SUMMERT: ISAKMP:(46580): processing NOTIFY PROPOSAL_NOT_CHOSEN protocol 3 spi 351538262, message ID = 1979798297, sa = 0x7F72986867D0 Jun 25 10:01:44 SUMMERT: ISAKMP:(46580): deleting spi 351538262 message ID = 928225319 Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):deleting node 928225319 error TRUE reason "Delete Larval" Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):peer does not do paranoid keepalives. Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):Enqueued KEY_MGR_DELETE_SAS for IPSEC SA (SPI:0x3B4731D7) Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):deleting node 1979798297 error FALSE reason "Informational (in) state 1" Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):Input = IKE_MESG_FROM_PEER, IKE_INFO_NOTIFY Jun 25 10:01:44 SUMMERT: ISAKMP:(46580):Old State = IKE_P1_COMPLETE New State = IKE_P1_COMPLETE Jun 25 10:01:44 SUMMERT: IPSEC: delete incomplete sa: 0x7F729923A340 Jun 25 10:01:44 SUMMERT: IPSEC(key_engine_delete_sas): delete SA with spi 0x3B4731D7 proto 50 for <spoke nbma> Jun 25 10:01:44 SUMMERT: IPSEC(update_current_outbound_sa): updated peer <spoke nbma> current outbound sa to SPI 0 Jun 25 10:01:44 SUMMERT: IPSEC(send_delete_notify_kmi): not sending KEY_ENGINE_DELETE_SAS Jun 25 10:01:44 SUMMERT: CRYPTO_SS(TUNNEL SEC): Sending request for CRYPTO SS CLOSE SOCKET<spoke nbma>
#sh pl ha qf ac fe ipsec data drop ------------------------------------------------------------------------ Drop Type Name Packets ------------------------------------------------------------------------ 3 IN_US_V4_PKT_FOUND_IPSEC_NOT_ENABLED 127672 19 IN_OCT_ANTI_REPLAY_FAIL 13346 20 IN_UNEXP_OCT_EXCEPTION 4224 33 OUT_V4_PKT_HIT_IKE_START_SP 1930 62 IN_OCT_MAC_EXCEPTION 9 #sh plat hard qfp act stat drop | e _0_ ------------------------------------------------------------------------- Global Drop Stats Packets Octets ------------------------------------------------------------------------- Disabled 1 82 IpFragErr 170536 246635169 IpTtlExceeded 4072 343853 IpsecIkeIndicate 1930 269694 IpsecInput 145256 30071488 Ipv4Acl 2251965 215240194 Ipv4Martian 6248 692010 Ipv4NoAdj 43188 7627131 Ipv4NoRoute 278 27913 Ipv4Unclassified 6 378 MplsNoRoute 790 69130 MplsUnclassified 1 60 ReassTimeout 63 10156 ServiceWireHdrErr 2684 585112
Also, after running 'logging dmvpn rate-limit 20' on the hub
%DMVPN-3-DMVPN_NHRP_ERROR: Tunnel292: NHRP Encap Error for Resolution Request , Reason: protocol generic error (7) on (Tunnel: <hub-tunnel-ip> NBMA: <hub nbma>)
On the spoke at the time the following can be seen from debugging as well:
*Jun 25 09:17:26.884: ISAKMP:(1032): sitting IDLE. Starting QM immediately (QM_IDLE ) *Jun 25 09:17:26.884: ISAKMP:(1032):beginning Quick Mode exchange, M-ID of 1599359281 *Jun 25 09:17:26.884: ISAKMP:(1032):QM Initiator gets spi *Jun 25 09:17:26.884: ISAKMP:(1032): sending packet to <hub nbma> my_port 500 peer_port 500 (R) QM_IDLE *Jun 25 09:17:26.884: ISAKMP:(1032):Sending an IKE IPv4 Packet. *Jun 25 09:17:26.884: ISAKMP:(1032):Node 1599359281, Input = IKE_MESG_INTERNAL, IKE_INIT_QM *Jun 25 09:17:26.884: ISAKMP:(1032):Old State = IKE_QM_READY New State = IKE_QM_I_QM1 *Jun 25 09:17:26.940: ISAKMP (1032): received packet from <hub nbma> dport 500 sport 500 Global (R) QM_IDLE *Jun 25 09:17:26.940: ISAKMP:(1032): processing HASH payload. message ID = 1599359281 *Jun 25 09:17:26.940: ISAKMP:(1032): processing SA payload. message ID = 1599359281 *Jun 25 09:17:26.940: ISAKMP:(1032):Checking IPSec proposal 1 *Jun 25 09:17:26.940: ISAKMP: transform 1, ESP_AES *Jun 25 09:17:26.940: ISAKMP: attributes in transform: *Jun 25 09:17:26.940: ISAKMP: encaps is 2 (Transport) *Jun 25 09:17:26.940: ISAKMP: SA life type in seconds *Jun 25 09:17:26.940: ISAKMP: SA life duration (basic) of 3600 *Jun 25 09:17:26.940: ISAKMP: SA life type in kilobytes *Jun 25 09:17:26.940: ISAKMP: SA life duration (VPI) of 0x0 0x46 0x50 0x0 *Jun 25 09:17:26.940: ISAKMP: authenticator is HMAC-SHA *Jun 25 09:17:26.940: ISAKMP: key length is 256 *Jun 25 09:17:26.940: ISAKMP:(1032):atts are acceptable. *Jun 25 09:17:26.940: IPSEC(ipsec_process_proposal): proxy identities not supported *Jun 25 09:17:26.940: ISAKMP:(1032): IPSec policy invalidated proposal with error 32 *Jun 25 09:17:26.940: ISAKMP:(1032): phase 2 SA policy not acceptable! (local <spoke nbma> remote <hub nbma>) *Jun 25 09:17:26.940: ISAKMP: set new node -1745931191 to QM_IDLE *Jun 25 09:17:26.940: ISAKMP:(1032):Sending NOTIFY PROPOSAL_NOT_CHOSEN protocol 3 spi 834718720, message ID = 2549036105 *Jun 25 09:17:26.940: ISAKMP:(1032): sending packet to <hub nbma> my_port 500 peer_port 500 (R) QM_IDLE *Jun 25 09:17:26.940: ISAKMP:(1032):Sending an IKE IPv4 Packet. *Jun 25 09:17:26.940: ISAKMP:(1032):purging node -1745931191 *Jun 25 09:17:26.940: ISAKMP:(1032):deleting node 1599359281 error TRUE reason "QM rejected" *Jun 25 09:17:26.940: ISAKMP:(1032):Node 1599359281, Input = IKE_MESG_FROM_PEER, IKE_QM_EXCH *Jun 25 09:17:26.940: ISAKMP:(1032):Old State = IKE_QM_I_QM1 New State = IKE_QM_I_QM1 *Jun 25 09:17:34.068: ISAKMP (1032): received packet from <hub nbma> dport 500 sport 500 Global (R) QM_IDLE *Jun 25 09:17:34.068: ISAKMP: set new node 1021264821 to QM_IDLE *Jun 25 09:17:34.072: ISAKMP:(1032): processing HASH payload. message ID = 1021264821 *Jun 25 09:17:34.072: ISAKMP:(1032): processing NOTIFY DPD/R_U_THERE protocol 1 spi 0, message ID = 1021264821, sa = 0x32741028 *Jun 25 09:17:34.072: ISAKMP:(1032):deleting node 1021264821 error FALSE reason "Informational (in) state 1" *Jun 25 09:17:34.072: ISAKMP:(1032):Input = IKE_MESG_FROM_PEER, IKE_INFO_NOTIFY *Jun 25 09:17:34.072: ISAKMP:(1032):Old State = IKE_P1_COMPLETE New State = IKE_P1_COMPLETE *Jun 25 09:17:34.072: ISAKMP:(1032):DPD/R_U_THERE received from peer <hub nbma>, sequence 0x64B2279D *Jun 25 09:17:34.072: ISAKMP: set new node 716440334 to QM_IDLE *Jun 25 09:17:34.072: ISAKMP:(1032):Sending NOTIFY DPD/R_U_THERE_ACK protocol 1 spi 834719464, message ID = 716440334 *Jun 25 09:17:34.072: ISAKMP:(1032): seq. no 0x64B2279D *Jun 25 09:17:34.072: ISAKMP:(1032): sending packet to <hub nbma> my_port 500 peer_port 500 (R) QM_IDLE *Jun 25 09:17:34.072: ISAKMP:(1032):Sending an IKE IPv4 Packet. *Jun 25 09:17:34.072: ISAKMP:(1032):purging node 716440334 *Jun 25 09:17:34.072: ISAKMP:(1032):Input = IKE_MESG_FROM_PEER, IKE_MESG_KEEP_ALIVE *Jun 25 09:17:34.072: ISAKMP:(1032):Old State = IKE_P1_COMPLETE New State = IKE_P1_COMPLETE *Jun 25 09:17:35.356: ISAKMP:(1032):purging node 206299144
Obviously something seems to be wrong with Phase 2 not coming up. But why is it going up after either clearing the crypto session or shutting down the tunnel interface and activate the tunnel interface of the spoke?
Very weird. Also, by looking att the debug-messages of the hub it seems the crypto is associated with the wrong tunnel interface Tu3300 when it should be Tu2010. Normal or Bug?
The configuration of the hub looks like this:
crypto keyring ISP1-DMVPN vrf ISP1-DMVPN pre-shared-key address 0.0.0.0 0.0.0.0 key <PSK> crypto isakmp policy 10 encr aes authentication pre-share crypto isakmp keepalive 10 3 periodic crypto isakmp nat keepalive 10 crypto isakmp profile ISP1-DMVPN keyring ISP1-DMVPN match identity address 0.0.0.0 ISP1-DMVPN keepalive 10 retry 3 crypto ipsec transform-set AES256-MD5 esp-aes 256 esp-md5-hmac mode tunnel crypto ipsec transform-set AES256-SHA-TRANSPORT esp-aes 256 esp-sha-hmac mode transport crypto ipsec profile ISP1-DMVPN set transform-set AES256-SHA AES256-SHA-TRANSPORT set isakmp-profile ISP1-DMVPN vrf definition ISP1-DMVPN description DMVPN-Outside-ISP1 rd 65527:10 ! address-family ipv4 exit-address-family ! ! interface TenGigabitEthernet0/0/0 no ip address ! interface TenGigabitEthernet0/0/0.71 description VPN;ISP1-DMVPN;Outside;VLAN71 encapsulation dot1Q 71 vrf forwarding ISP1-DMVPN ip address <hub nbma> 255.255.255.128 no ip proxy-arp ip access-group acl_ISP1-DMVPN_IN in ! ip route vrf ISP1-DMVPN 0.0.0.0 0.0.0.0 <isp1 default gw> name ISP1;Default ip access-list extended acl_ISP1-DMVPN_IN permit icmp any any permit esp any host <hub nbma> permit gre any host <hub nbma> permit udp any host <hub nbma> eq isakmp permit udp any host <hub nbma> eq non500-isakmp deny ip any any vrf definition 2010 description CUSTA - Customer A rd 65527:2010 route-target export 65527:2010 route-target import 65527:2010 ! address-family ipv4 exit-address-family ! ! interface Tunnel2010 description CUSTA;DMVPN;Failover-secondary vrf forwarding 2010 ip address 10.97.0.34 255.255.255.240 no ip redirects ip mtu 1380 ip nhrp map multicast dynamic ip nhrp network-id 2010 ip nhrp holdtime 120 ip nhrp server-only ip nhrp max-send 1000 every 10 ip tcp adjust-mss 1340 tunnel source TenGigabitEthernet0/0/0.71 tunnel mode gre multipoint tunnel key 2010 tunnel vrf ISP1-DMVPN tunnel protection ipsec profile ISP1-DMVPN shared router bgp 65527 ! address-family ipv4 vrf 2010 redistribute connected metric 10 redistribute static metric 15 neighbor 10.97.0.39 remote-as 65028 neighbor 10.97.0.39 description spokerouter;Tunnel1 neighbor 10.97.0.39 update-source Tunnel2010 neighbor 10.97.0.39 activate neighbor 10.97.0.39 soft-reconfiguration inbound neighbor 10.97.0.39 prefix-list EXPORT-IVPN-VRF2010 out neighbor 10.97.0.39 route-map AllVRF-LocalPref-80 in neighbor 10.97.0.39 maximum-prefix 5000 80 default-information originate exit-address-family
The spoke configuration:
crypto keyring DMVPN01 pre-shared-key address 0.0.0.0 0.0.0.0 key <PSK> crypto isakmp policy 10 encr aes authentication pre-share crypto isakmp invalid-spi-recovery crypto isakmp profile DMVPN01 keyring DMVPN01 match identity address 0.0.0.0 keepalive 10 retry 3 crypto ipsec transform-set AES256-SHA esp-aes 256 esp-sha-hmac mode tunnel crypto ipsec transform-set AES256-SHA-TRANSPORT esp-aes 256 esp-sha-hmac mode transport crypto ipsec profile DMVPN01 set transform-set AES256-SHA-TRANSPORT set isakmp-profile DMVPN01 vrf definition inside rd 65028:1 route-target export 65028:1 route-target import 65028:1 ! address-family ipv4 exit-address-family ! interface Tunnel1 description DMVPN to HUB vrf forwarding inside ip address 10.97.0.39 255.255.255.240 no ip redirects ip mtu 1380 ip nhrp map 10.97.0.33 <hub1 nbma> ip nhrp map multicast <hub1 nbma> ip nhrp map 10.97.0.34 <hub2 nbma> ip nhrp map multicast <hub2 nbma> ip nhrp network-id 1 ip nhrp holdtime 120 ip nhrp nhs 10.97.0.33 ip nhrp nhs 10.97.0.34 ip nhrp registration no-unique ip nhrp registration timeout 60 ip tcp adjust-mss 1340 tunnel source GigabitEthernet0/0 tunnel mode gre multipoint tunnel key 2010 tunnel protection ipsec profile DMVPN01 shared router bgp 65028 ! address-family ipv4 vrf inside bgp router-id 172.28.5.137 network 10.97.20.128 mask 255.255.255.128 network 10.97.21.0 mask 255.255.255.0 network 10.97.22.0 mask 255.255.255.0 network 10.97.23.0 mask 255.255.255.0 network 172.28.5.137 mask 255.255.255.255 neighbor 10.97.0.33 remote-as 65527 neighbor 10.97.0.33 description HUB1;Tunnel2010 neighbor 10.97.0.33 update-source Tunnel1 neighbor 10.97.0.33 timers 10 30 neighbor 10.97.0.33 activate neighbor 10.97.0.33 send-community both neighbor 10.97.0.33 soft-reconfiguration inbound neighbor 10.97.0.33 prefix-list IROUTE-EXPORT out neighbor 10.97.0.33 maximum-prefix 5000 80 neighbor 10.97.0.34 remote-as 65527 neighbor 10.97.0.34 description HUB2;tunnel2010 neighbor 10.97.0.34 update-source Tunnel1 neighbor 10.97.0.34 timers 10 30 neighbor 10.97.0.34 activate neighbor 10.97.0.34 send-community both neighbor 10.97.0.34 soft-reconfiguration inbound neighbor 10.97.0.34 prefix-list IROUTE-EXPORT out neighbor 10.97.0.34 route-map AllVRF-LocalPref-80 in neighbor 10.97.0.34 maximum-prefix 5000 80 exit-address-family
If more information is needed please say so.
Any help or guidance would be greatly appreciated!
Thanks!
Solved! Go to Solution.
06-25-2015 09:02 AM
It's possible you're hitting this one - the phase 2 negotiation failure:
https://tools.cisco.com/bugsearch/bug/CSCup72039/?reffering_site=dumpcr
Too few details to say for sure :]
M.
06-25-2015 09:02 AM
It's possible you're hitting this one - the phase 2 negotiation failure:
https://tools.cisco.com/bugsearch/bug/CSCup72039/?reffering_site=dumpcr
Too few details to say for sure :]
M.
06-25-2015 09:35 AM
Hmm, interesting. I will disable periodic DPD and see if that solves things.
What more information can I give you?
Thanks!
06-25-2015 10:16 AM
Try IKEv2? :-) You know it's good for you!
Check whether you may or may not have multiple IKE sessions.
And remember even changing DPDs configuration will not take immediate effect - you need to still re-key IKE sessions. In case you were thinking it's just as easy as changing DPD config.
06-25-2015 12:20 PM
Hehe, will sure do! :)
Though we have around 200 spokes that needs to be reconfigured. It would be more satisfactory to solve the current issues without ikev2 :)
As a matter of fact there are multiple IKE sessions to this particular spoke and other as well. 64 as of this writing. Is this normal?
On the hub:
#show crypto ipsec sa peer <spoke nbma> | count current_peer Number of lines which match regexp = 64 #
Also the IKE seems to be associated with the wrong tunnel interface.
06-25-2015 11:53 PM
Those are IPsec SA count, probably a bit too much, but most likely not all of those are active - a couple could have been introduced because of the problem we see in debugs (PROPOSAL_NOT_CHOSEN). Hard to say without actually seeing what they are ... but indeed it's uncommon for a spoke to have so many IPsec SAs (in DMVPN).
You'd need have a look at "show cry isa sa" output to check what IKE sessions exist.
06-26-2015 04:51 AM
By looking at the "Status:" on the IPSEC SA's with this spoke the 64 are all active. Both inbound and outbound.
They say:
Status: ACTIVE(ACTIVE)
By looking at
show crypto ipsec sa peer <spoke nbma>
It seems it creates one IPSEC sa per tunnel interface on the hub (almost), when looking at the output of "Interface:"? Even though this particular spoke should be associated with just one tunnel interface (Tunnel2010). Is this normal when sharing the ipsec profile, perhaps?
#show crypto ipsec sa peer <spoke nbma> | inc interface: interface: Tunnel3300 interface: Tunnel3260 interface: Tunnel3160 interface: Tunnel2931 interface: Tunnel2930 interface: Tunnel2818 interface: Tunnel2817 interface: Tunnel2816 interface: Tunnel2815 interface: Tunnel2814 interface: Tunnel2813 interface: Tunnel2812 interface: Tunnel2811 interface: Tunnel2750 interface: Tunnel2661 interface: Tunnel2660 interface: Tunnel2590 interface: Tunnel2470 interface: Tunnel2342 interface: Tunnel2341 interface: Tunnel2340 interface: Tunnel2220 interface: Tunnel2190 interface: Tunnel2180 interface: Tunnel2160 interface: Tunnel2011 interface: Tunnel2010 interface: Tunnel1933 interface: Tunnel1932 interface: Tunnel1931 interface: Tunnel1930 interface: Tunnel1820 interface: Tunnel1720 interface: Tunnel1700 interface: Tunnel1680 interface: Tunnel1670 interface: Tunnel1450 interface: Tunnel1370 interface: Tunnel1360 interface: Tunnel1050 interface: Tunnel871 interface: Tunnel870 interface: Tunnel850 interface: Tunnel710 interface: Tunnel550 interface: Tunnel545 interface: Tunnel542 interface: Tunnel540 interface: Tunnel490 interface: Tunnel480 interface: Tunnel470 interface: Tunnel469 interface: Tunnel468 interface: Tunnel461 interface: Tunnel460 interface: Tunnel430 interface: Tunnel410 interface: Tunnel380 interface: Tunnel360 interface: Tunnel310 interface: Tunnel292 interface: Tunnel250 interface: Tunnel2591
When looking at "show crypto isakmp sa" I saw a couple of duplicated entries to this spoke. This was fixed by clearing the crypto session. I've not seen it since then.
06-26-2015 04:51 AM
Ha, indeed I only saw one tunnel in the config you've outlined.
There are a couple of interested (cosmetic and not) side effects when using "shared" ... that's why I prefer FlexVPN for those scenarios.
I think some enhancements came to "shared" code relatively recently (1.5 years ago...)
Regardless... any changes in behavior since moving to non-periodic DPDs?
06-26-2015 05:10 AM
Yeah, sorry for leaving that "little" part out :)
I see. FlexVPN looks interesting. From what i've read it is difficult to implement this on an already DMVPN enabled device?
Yes actually, from what I can see the spokes have been more stable since moving to non-periodic DPD's and renegotiating the tunnels. I will still keep an eye on this though. Do you see anything else that I can improve or adjust regarding the configuration?
Thank you very much this far, Marcin.
06-26-2015 05:41 AM
Well hard to say, I think you've for a pretty cool thing going.
BGP as routing protocol already a big plus.
VRF separation for ISPs and inside, I wish more people had that.
Periodic DPDs are also a pretty good idea (overall).
How many spokes are you terminating per ASR? (I assume you have two)
Maybe one suggestion is to use certificates instead of PSK ... there's a slight learning curve, but IOS devices can act as both DMVPN nodes and CAs ... I think there was/is a limitation that ASRs could/can not act as CAs.
Another ... your encryption suite is fine, but I'd probably look into updating it, should spokes support it ... AES-GCM is pretty sweet. Vide:
http://www.cisco.com/web/about/security/intelligence/nextgen_crypto.html#2
A couple of years back I wrote DM to Flex migration guide:
http://www.cisco.com/c/en/us/support/docs/security/flexvpn/116678-configure-product-00.html
Obviously it's for a very simple example.
But what's good with newer software is that you can have co-existence of FlexVPN and DMVPN on same boxes (you'd need something newer than IOS XE 10.3 and IOS 15... I want to say 15.3T ). You can move between FlexVPN and DMVPN tunnels using BGP.
Oh I wish we had unlimited time in the lab to play with things ;]
06-26-2015 06:55 AM
Cool.
Um, by running sh dmvpn and counting NBMA Peers there are around 200+ spokes per ASR.
Really cool, I will definitely look into this. Yeah, I agree with you. If you only had more time.. :)
What do you think of implementing if-state nhrp on the spokes? Would this help me in this scenario?
Lastly, what would you prefer to accomplish DMVPN-redundancy on the spoke, two tunnel interfaces to each hub with different tunnel subnets or just one tunnel interface with configuration for both hubs using same tunnel subnet?
Thanks,
06-26-2015 07:41 AM
200-300 per ASR seems decent, lots of possibility to scale up on most ASR1ks.
if-state nhrp ... it's good for monitoring, and if you're running only spoke to hub traffic. If you do have some spoke to spoke you'll start losing the benefits (especially in case of failure).
For redundancy on spokes I'm definitely a proponent of multi-cloud. But I do like things compartmentalized. It's easier to manage and make changes on - you know that changing X on tunnel Y will not impact tunnel Z --- or at least should not :}
It does take a bit more implementation effort but I find that it's easier to understand.
Hey, a thought, even if you FlexVPN is not for you at the moment, don't write off IKEv2 ... DMVPN integrates with IKEv2! If you're thinking about implementing redundancy on spokes ... have on tunnel interface using IKEv1 another using IKEv2 - at least it out. :-)
06-30-2015 03:58 AM
All right, thanks for your input!
Regarding IKEv2 I will check this out!
Thanks for all the help and input. Greatly appreciate it. :)
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide