cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2365
Views
5
Helpful
10
Replies

Weird Dual DMVPN bouncing issue.

Larry Sullivan
Level 3
Level 3

Hi,

 

So I have hundreds of sites configured the same as this site with the weird issue.  It's not the DMVPN configs.

Issue is, I have two DMVPN tunnels configured to two different hubs on the router.  When both DMVPNs are no shut, they will stay up for an hour, two, four, it varies, but they will both drop at separate times.  Sometimes they stay down for a bit, sometimes only briefly.  The WAN connection is not dropping packets (verified via SLAs).  What I discovered is that if you shut one of the DMVPN tunnels, the other one will no longer drop intermittently, doesn't matter which tunnel is shut.  So obviously either the router or the ISP DSL modem is having issues handling the two DMVPNs at the same time.  When DMVPNs are down they are stuck in the NHRP state.

 

Questions are:

1) Has anyone experienced this and if so, did it turn out to be the modem or router (latest IOS c800-universalk9-mz.SPA.154-3.M9.bin)?

2) Is it possible to put one DMVPN in a VRF without configs on the Hub side (no VRF configs anywhere else)?

 

I suspect the AT&T DSL modem (1.5Mb DL 768k UL), but we've had this setup elsewhere and it worked here for a long time too, so I don't think it's the specs of the modem, I think there is something wrong with it.

10 Replies 10

Deepak Kumar
VIP Alumni
VIP Alumni

Hi,

1) Has anyone experienced this and if so, did it turn out to be the modem or router (latest IOS c800-universalk9-mz.SPA.154-3.M9.bin)?

Ans: No, I never faced this issue. Is modem configured in the NAT mode or bridge mode? Is it possible to share the router (Spoke) configuration? Have you applied IPsec on the tunnels?

2) Is it possible to put one DMVPN in a VRF without configs on the Hub side (no VRF configs anywhere else)?

Ans: Yes, It is possible but may require some more efforts. If your modem in the NAT mode and you have assigned private IP on the router WAN interface then You can connect two cables between Modem and router. The router's one interface you can keep in the global VRF and secondary wan interface can keep in the new VRF. Your configuration changes may like 

 

ip vrf TEST

!

interface gig 0/1

desc WAN interface Global VRF

IP add 192.168.1.1 255.255.255.0

no shut

!

interface gig 0/2

desc WAN interface TEST VRF

ip forwarding vrf TEST

IP add 192.168.1.2 255.255.255.0

no shut

!

interface tunnel 100

des Tunnel interface from TEST VRF

tunnel vrf TEST

!
!
ip route vrf TEST 0.0.0.0 0.0.0.0 gig 0/2

 

But I advised sharing the Router configuration for a better understanding of the issue.

 

Regards,

Deepak Kumar

 

 

Regards,
Deepak Kumar,
Don't forget to vote and accept the solution if this comment will help you!

Thanks for the reply Deepak.

1)Modem is in bridge mode. (so not possible to do VRF for one of the DMVPNs on the single public IP interface?)

2)IPsec is applied

3) Spoke configs below.

 

interface Tunnel118
description DMVPN
ip address 10.90.118.120 255.255.254.0
no ip redirects
no ip unreachables
no ip proxy-arp
ip mtu 1400
ip nhrp authentication xxx
ip nhrp map multicast dynamic
ip nhrp map 10.90.118.1 x.x.x.x
ip nhrp map multicast x.x.x.x
ip nhrp network-id xxx
ip nhrp holdtime 120
ip nhrp nhs 10.90.118.1
ip tcp adjust-mss 1360
delay 1000
shutdown
tunnel source FastEthernet4
tunnel mode gre multipoint
tunnel key xxxxxxx
tunnel protection ipsec profile xxxxxx shared

 

interface Tunnel128
description DMVPN
ip address 10.90.129.67 255.255.252.0
no ip redirects
no ip unreachables
no ip proxy-arp
ip mtu 1400
ip nhrp authentication xxxx
ip nhrp map multicast dynamic
ip nhrp map 10.90.128.1 x.x.x.x
ip nhrp map multicast x.x.x.x
ip nhrp network-id xxx
ip nhrp holdtime 120
ip nhrp nhs 10.90.128.1
ip tcp adjust-mss 1360
tunnel source FastEthernet4
tunnel mode gre multipoint
tunnel key xxxxxx
tunnel protection ipsec profile xxxxxx shared

Since you masked both the tunnel keys and network-ids, I have to ask you to confirm if the tunnel keys and network-ids are unique for Tunnel 118 and 128.

 

Also, you don't need ip nhrp map multicast dynamic on the Spoke. That should only be at the Hub.

Yes, they are unique, and the ip nhrp map multicast dynamic are old configs from whoever implemented the tunnels.  They don't really affect anything so they are left in place.  Currently I reduced the IP MTU and ip tcp adjust-mss to minimums and the tunnels have been stable since.  I'll need another several hours to be for sure, but this leads me to believe it is an issue with the modem.

Hi,

As you have defined the MTU 1400 and it is recommended MTU from the Cisco Design guide for DMVPN. But IPSec transfer set must be in TRANSPORT MODE  because spoke or hub is behind the nat. 

 

Regards,

Deepak Kumar 

Regards,
Deepak Kumar,
Don't forget to vote and accept the solution if this comment will help you!

As a continuation from my previous post, issue is still occurring.  And to Deepak, we have literally a couple hundred sites working fine in tunnel mode.  I will attempt to put this site in transport mode though and see how that goes.  Thanks.

 

Edit: Won't work without changing hub side too, so not worth taking down all these tunnels because one site isn't working.

Hi,

If possible share debugs logs if we will found something from debugs.

 

Regards,

Deepak Kumar

 

 

Regards,
Deepak Kumar,
Don't forget to vote and accept the solution if this comment will help you!

So I got the modem replaced and it stabilized for several days and then the issue returned.  DMVPN/BGP goes down for a bit then comes back.

 

Mar 9 14:21:22: IPSEC(send_delete_notify_kmi): not sending KEY_ENGINE_DELETE_SAS
Mar 9 14:21:22: IPSEC(ERROR): [ident_update_final_flow_stats] Peer index node NULL for peer index 0 when corresponding flow id 0x140001FD was completed

Mar 9 14:21:23: IPSEC(ERROR): crypto_notify_rp Rejected notify RP, elapse time 0 < 1000

Mar 9 14:21:23: IPSEC(ERROR): crypto_notify_rp Rejected notify RP, elapse time 0 < 1000

Mar 9 14:21:23: IPSEC(ipsec_get_crypto_session_id):
Invalid Payload Id
Mar 9 14:21:57: IPSEC(sibling_delete_notify_ident_action): Ident down, not sending DECR/DELETE
Mar 9 14:21:57: IPSEC(ERROR): [ident_update_final_flow_stats] Peer index node NULL for peer index 0 when corresponding flow id 0x140001FF was completed

Mar 9 14:21:57: IPSEC(ERROR): Failed to remove SADB 0x1080A318 from sadb_root_avl_tree
Mar 9 14:21:59: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF
Mar 9 14:21:59: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is ON
Mar 9 14:21:59: insert of map into mapdb AVL failed, map + ace pair already exists on the mapdb
Mar 9 14:21:59: IPSEC(ipsec_get_crypto_session_id):
Invalid Payload Id
Mar 9 14:23:12: IPSEC(send_delete_notify_kmi): not sending KEY_ENGINE_DELETE_SAS
Mar 9 14:23:12: IPSEC(ERROR): [ident_update_final_flow_stats] Peer index node NULL for peer index 0 when corresponding flow id 0x14000201 was completed

Mar 9 14:23:19: IPSEC(ERROR): crypto_notify_rp Rejected notify RP, elapse time 0 < 1000

Mar 9 14:23:19: IPSEC(ipsec_get_crypto_session_id):
Invalid Payload Id

Mar 9 14:23:29: %BGP-5-ADJCHANGE: neighbor x.x.x.x Up

Hi,

Please confirm that why underlying routing (BGP) is getting down? 

Mar 9 14:23:29: %BGP-5-ADJCHANGE: neighbor x.x.x.x Up

Regards,

Deepak Kumar

Regards,
Deepak Kumar,
Don't forget to vote and accept the solution if this comment will help you!

Deepak,

BGP is reliant upon the DMVPN so is on top of it.  DMVPN goes down, then yes, the BGP goes down as well.  What you are seeing is BGP returning after it and DMVPN being down due to mentioned issue.  At this point my suggested next step is to upgrade the DSL speeds (1.5 Mbps DL and 768Kbps UL).  We had a similar issue with another site that got resolved by upgrading speeds.  If that doesn't work then I'm going to suggest a new provider.  I will update with results of speed upgrade.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Innovations in Cisco Full Stack Observability - A new webinar from Cisco