02-21-2019 06:12 PM
Hi,
So I have hundreds of sites configured the same as this site with the weird issue. It's not the DMVPN configs.
Issue is, I have two DMVPN tunnels configured to two different hubs on the router. When both DMVPNs are no shut, they will stay up for an hour, two, four, it varies, but they will both drop at separate times. Sometimes they stay down for a bit, sometimes only briefly. The WAN connection is not dropping packets (verified via SLAs). What I discovered is that if you shut one of the DMVPN tunnels, the other one will no longer drop intermittently, doesn't matter which tunnel is shut. So obviously either the router or the ISP DSL modem is having issues handling the two DMVPNs at the same time. When DMVPNs are down they are stuck in the NHRP state.
Questions are:
1) Has anyone experienced this and if so, did it turn out to be the modem or router (latest IOS c800-universalk9-mz.SPA.154-3.M9.bin)?
2) Is it possible to put one DMVPN in a VRF without configs on the Hub side (no VRF configs anywhere else)?
I suspect the AT&T DSL modem (1.5Mb DL 768k UL), but we've had this setup elsewhere and it worked here for a long time too, so I don't think it's the specs of the modem, I think there is something wrong with it.
02-21-2019 08:46 PM - edited 02-21-2019 09:13 PM
Hi,
1) Has anyone experienced this and if so, did it turn out to be the modem or router (latest IOS c800-universalk9-mz.SPA.154-3.M9.bin)?
Ans: No, I never faced this issue. Is modem configured in the NAT mode or bridge mode? Is it possible to share the router (Spoke) configuration? Have you applied IPsec on the tunnels?
2) Is it possible to put one DMVPN in a VRF without configs on the Hub side (no VRF configs anywhere else)?
Ans: Yes, It is possible but may require some more efforts. If your modem in the NAT mode and you have assigned private IP on the router WAN interface then You can connect two cables between Modem and router. The router's one interface you can keep in the global VRF and secondary wan interface can keep in the new VRF. Your configuration changes may like
ip vrf TEST
!
interface gig 0/1
desc WAN interface Global VRF
IP add 192.168.1.1 255.255.255.0
no shut
!
interface gig 0/2
desc WAN interface TEST VRF
ip forwarding vrf TEST
IP add 192.168.1.2 255.255.255.0
no shut
!
interface tunnel 100
des Tunnel interface from TEST VRF
tunnel vrf TEST
!
!
ip route vrf TEST 0.0.0.0 0.0.0.0 gig 0/2
But I advised sharing the Router configuration for a better understanding of the issue.
Regards,
Deepak Kumar
02-22-2019 10:14 AM - edited 02-22-2019 10:15 AM
Thanks for the reply Deepak.
1)Modem is in bridge mode. (so not possible to do VRF for one of the DMVPNs on the single public IP interface?)
2)IPsec is applied
3) Spoke configs below.
interface Tunnel118
description DMVPN
ip address 10.90.118.120 255.255.254.0
no ip redirects
no ip unreachables
no ip proxy-arp
ip mtu 1400
ip nhrp authentication xxx
ip nhrp map multicast dynamic
ip nhrp map 10.90.118.1 x.x.x.x
ip nhrp map multicast x.x.x.x
ip nhrp network-id xxx
ip nhrp holdtime 120
ip nhrp nhs 10.90.118.1
ip tcp adjust-mss 1360
delay 1000
shutdown
tunnel source FastEthernet4
tunnel mode gre multipoint
tunnel key xxxxxxx
tunnel protection ipsec profile xxxxxx shared
interface Tunnel128
description DMVPN
ip address 10.90.129.67 255.255.252.0
no ip redirects
no ip unreachables
no ip proxy-arp
ip mtu 1400
ip nhrp authentication xxxx
ip nhrp map multicast dynamic
ip nhrp map 10.90.128.1 x.x.x.x
ip nhrp map multicast x.x.x.x
ip nhrp network-id xxx
ip nhrp holdtime 120
ip nhrp nhs 10.90.128.1
ip tcp adjust-mss 1360
tunnel source FastEthernet4
tunnel mode gre multipoint
tunnel key xxxxxx
tunnel protection ipsec profile xxxxxx shared
02-22-2019 01:54 PM
Since you masked both the tunnel keys and network-ids, I have to ask you to confirm if the tunnel keys and network-ids are unique for Tunnel 118 and 128.
Also, you don't need ip nhrp map multicast dynamic on the Spoke. That should only be at the Hub.
02-22-2019 02:06 PM
Yes, they are unique, and the ip nhrp map multicast dynamic are old configs from whoever implemented the tunnels. They don't really affect anything so they are left in place. Currently I reduced the IP MTU and ip tcp adjust-mss to minimums and the tunnels have been stable since. I'll need another several hours to be for sure, but this leads me to believe it is an issue with the modem.
02-22-2019 11:03 PM
Hi,
As you have defined the MTU 1400 and it is recommended MTU from the Cisco Design guide for DMVPN. But IPSec transfer set must be in TRANSPORT MODE because spoke or hub is behind the nat.
Regards,
Deepak Kumar
02-23-2019 04:13 PM - edited 02-23-2019 04:21 PM
As a continuation from my previous post, issue is still occurring. And to Deepak, we have literally a couple hundred sites working fine in tunnel mode. I will attempt to put this site in transport mode though and see how that goes. Thanks.
Edit: Won't work without changing hub side too, so not worth taking down all these tunnels because one site isn't working.
02-23-2019 08:57 PM
Hi,
If possible share debugs logs if we will found something from debugs.
Regards,
Deepak Kumar
03-09-2019 02:26 PM
So I got the modem replaced and it stabilized for several days and then the issue returned. DMVPN/BGP goes down for a bit then comes back.
Mar 9 14:21:22: IPSEC(send_delete_notify_kmi): not sending KEY_ENGINE_DELETE_SAS
Mar 9 14:21:22: IPSEC(ERROR): [ident_update_final_flow_stats] Peer index node NULL for peer index 0 when corresponding flow id 0x140001FD was completed
Mar 9 14:21:23: IPSEC(ERROR): crypto_notify_rp Rejected notify RP, elapse time 0 < 1000
Mar 9 14:21:23: IPSEC(ERROR): crypto_notify_rp Rejected notify RP, elapse time 0 < 1000
Mar 9 14:21:23: IPSEC(ipsec_get_crypto_session_id):
Invalid Payload Id
Mar 9 14:21:57: IPSEC(sibling_delete_notify_ident_action): Ident down, not sending DECR/DELETE
Mar 9 14:21:57: IPSEC(ERROR): [ident_update_final_flow_stats] Peer index node NULL for peer index 0 when corresponding flow id 0x140001FF was completed
Mar 9 14:21:57: IPSEC(ERROR): Failed to remove SADB 0x1080A318 from sadb_root_avl_tree
Mar 9 14:21:59: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF
Mar 9 14:21:59: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is ON
Mar 9 14:21:59: insert of map into mapdb AVL failed, map + ace pair already exists on the mapdb
Mar 9 14:21:59: IPSEC(ipsec_get_crypto_session_id):
Invalid Payload Id
Mar 9 14:23:12: IPSEC(send_delete_notify_kmi): not sending KEY_ENGINE_DELETE_SAS
Mar 9 14:23:12: IPSEC(ERROR): [ident_update_final_flow_stats] Peer index node NULL for peer index 0 when corresponding flow id 0x14000201 was completed
Mar 9 14:23:19: IPSEC(ERROR): crypto_notify_rp Rejected notify RP, elapse time 0 < 1000
Mar 9 14:23:19: IPSEC(ipsec_get_crypto_session_id):
Invalid Payload Id
Mar 9 14:23:29: %BGP-5-ADJCHANGE: neighbor x.x.x.x Up
03-09-2019 06:54 PM
Hi,
Please confirm that why underlying routing (BGP) is getting down?
Mar 9 14:23:29: %BGP-5-ADJCHANGE: neighbor x.x.x.x Up
Regards,
Deepak Kumar
03-12-2019 01:54 PM - edited 03-12-2019 01:57 PM
Deepak,
BGP is reliant upon the DMVPN so is on top of it. DMVPN goes down, then yes, the BGP goes down as well. What you are seeing is BGP returning after it and DMVPN being down due to mentioned issue. At this point my suggested next step is to upgrade the DSL speeds (1.5 Mbps DL and 768Kbps UL). We had a similar issue with another site that got resolved by upgrading speeds. If that doesn't work then I'm going to suggest a new provider. I will update with results of speed upgrade.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide