DMVPN - EIGRP Neighbors

[Spyridon Katrivesis]

Hi,

I run a DMVPN solution in Dual hub mode. I use EIGRP as a routing protocol between the HUb and Spokes.

I know that gre is pain most of the times but we have to live with that. Although I had EIGRP spoke neighbors

stable for 8-9 weeks and someothers dropping every few weeks I realised 2 days ago that all the EIGRP neighbors dropped the same time

from both Hubs.

On each spoke I run a common phase 1 for the VPN but different phase 2 people who know th DMVPN well know what I mean.

The HUBs located in different areas and there was not bandwidth issue to affect both Hubs the same time. Its definitely something

with the protocols that the DMVPN uses or with EIGRP.

I didnt see any DMVPN drops I saw only EIGRP neighborship dropped for all spokes from both Hubs the same time. Any suggestions

why EIGRP failed ?

It could be something with NHRP or an IOS bug;

ios c800-universalk9-mz.spa.153-3.m.bin

Please do not ask me for basic troubleshooting or connectivity or timers . I am looking for an advanced suggestion as I resolved many DMVPN issues

which even cisco couldn't find.

I am looking forward for any good suggestion and thanks for taking time to look into that.

Regards,

Spyros

[Spyridon Katrivesis]

I forgot to say that I run syslog but for some reason TFTPd32 crashed so I was not able to see what happend

[paul driver]

Hello

"Please do not ask me for basic troubleshooting or connectivity or timers . I am looking for an advanced suggestion as I resolved many DMVPN issues which even cisco couldn't find"

personally I find that rather conceited statement given the experienced engineers these forums have as members.

Also that really isn't a good statement to start a discussion off with -You may be familiar with your setup but no one else would -Also we wouldn't be aware of what troubleshooting steps you have already performed - so if your not interested in being asked basic troubleshooting steps until we are also familiar with you setup then I guess you will not get many responses to your query

Now regards your issue. Can post your VPN configuration and any logs pertaining these protocols if applicable

Res
Paul


Sent from Cisco Technical Support iPad App

[karthik2000]

Hi Paul,

I am facing one issue in dmvpn.

For hub location I have 2 isp which is nat with our own public ip 103.x.x.x and advertised in both isp. I primary isp up I am able reach hub to spoke.

If primary is down I am unable to reach hub to spoke. I have observed IPsec tunnel is I.e qm_idel on both hub and spoke but not receiving eigrp routes when primary internet is down. Pls help what need to do.

[Spyridon Katrivesis]

Hi Paul,

thank you for your reply. I was sure that what I mentioned regarding "..." will annoy some people but people who go through tickets like that and are here regularly know what I meant. This is to avoid people asking for connectivity issues or backup applications running the same time affecting the links etc or anything involved the default timers.

DMVPN is a solutions that runs multiple protocols and technologies. So we have gre , ipsec, nhrp etc and is not easy to identify where the problem is unless you run debuging.

So here we are with the configuration;

MAIN HUB

---------------------

interface Tunnel0

description

bandwidth 1000000

ip address x.x.x.1.254

no ip redirects

ip mtu 1440

ip authentication mode eigrp 90 md5

ip authentication key-chain eigrp 90 xxxxxxxxxxxxxxxxxx

no ip next-hop-self eigrp 90

no ip split-horizon eigrp 90

ip nhrp authentication xxxxxxxxxxxxxxxxx

ip nhrp map multicast dynamic

ip nhrp network-id 1

ip nhrp holdtime 3600

ip tcp adjust-mss 1360

tunnel source xxxxxxxxxxxxxxx

tunnel mode gre multipoint

tunnel key 1

tunnel protection ipsec profile xxxxxxxxxxxxxxxxxxx

router eigrp 90

distribute-list prefix xxxxxxx

distribute-list prefix xxxxxxxxxxx

network y.y.y.y

network x.x.x.x -----> this is the tunnel 0 subnet

redistribute static route-map xxxxxxx

passive-interface xxxxxxxxxxxxxxxxxxx

Spoke

--------------

crypto isakmp policy 10

hash md5

authentication pre-share

crypto isakmp key xxxxxxxx address 0.0.0.0       

!

!

crypto ipsec transform-set stone esp-3des esp-md5-hmac

mode tunnel

crypto ipsec transform-set strone2 esp-3des esp-md5-hmac

mode tunnel

!

crypto ipsec profile test

set security-association lifetime seconds 120

set transform-set strong

!

crypto ipsec profile test2

set security-association lifetime seconds 120

set transform-set strong2

!        

!

!

!

!

!

!

!

interface Tunnel0

description Tunnel to the main Hub

bandwidth 100000

ip address x.x.x.x 255.255.255.0

no ip redirects

ip mtu 1440

ip authentication mode eigrp 90 md5

ip authentication key-chain eigrp 90 xxxxxxx

no ip next-hop-self eigrp 90

no ip split-horizon eigrp 90

ip nhrp authentication xxxxxxxx

ip nhrp map multicast dynamic

ip nhrp map multicaste.e.e.e ( public Ip for the main hub)

ip nhrp map  s.s.s.s e.e.e.e ( tunnel IP to the public IP for the main hub)

ip nhrp network-id 1

ip nhrp holdtime 3600

ip nhrp nhs s.s.s.s (tunnel Ip fo rthe main hub)

ip tcp adjust-mss 1360

delay 50000

tunnel source Dialer1

tunnel mode gre multipoint

tunnel key 1

tunnel protection ipsec profile test shared

!

interface Tunnel1

description Tunnel to the backup Hub

bandwidth 10000

ip address x.x.y.x 255.255.255.0

no ip redirects

ip mtu 1440

ip authentication mode eigrp 90 md5

ip authentication key-chain eigrp 90 xxxxxxx

no ip next-hop-self eigrp 90

no ip split-horizon eigrp 90

ip nhrp authentication xxxxxxx

ip nhrp map multicast dynamic

ip nhrp map multicast  ............

ip nhrp map s.s.s.s e.e.e.e

ip nhrp network-id 100

ip nhrp holdtime 3600

ip nhrp nhs x.x.x.x

ip tcp adjust-mss 1360

delay 50000

tunnel source Dialer1

tunnel mode gre multipoint

tunnel key 100

tunnel protection ipsec profile test2 shared

router eigrp 90

network local network

network x.x.x.x (tunnel 0 subnet)

network y.y.y.y (tunnel 1 subnet)

passive-interface GigabitEthernet0

passive-interface GigabitEthernet1

passive-interface GigabitEthernet2

passive-interface GigabitEthernet3

passive-interface GigabitEthernet4

passive-interface GigabitEthernet5

passive-interface GigabitEthernet6

passive-interface GigabitEthernet7

eigrp stub connected

I hope that helps.

My main question is fomeone had the same problem seen eigrp dropping from both hubs the same time for all the spokes.

The config for the backup Up is configured the same way what only changesare the Ip addresses.

Thanks,

Spyros

[Spyridon Katrivesis]

when  said backup in the last line I meant the Backup DMVPN Hub.

[Vasilii Mikhailovskii]

Hello.

Whenever you observe massive connection (EIGRP nei) drops, I personally suspect ISP issue first.

If you are using same ISP in both locations (or for all the affected Spokes), then it could be some issue with ISP routing.

Do you have any monitoring tool for WAN (google for example) reachability, that could prove that Internet access was fine that time?

Do you have any link load diagrams for the moment you faced an issue?

PS: just out of curiosity:

a) why do you use "ip nhrp map multicast dynamic", "no ip next-hop-self eigrp 90" and "no ip split-horizon eigrp 90" on your spokes?

b) why do you use ip mtu 1440 + MSS=1360?

c) why don't you tune timers for EIGRP over tunnels?

d) why do you use EIGRP authentication over IPSec?

[paul driver]

Hello

I must agree MikhalioyskyVV that to drop connectivity to both hubs to all spoke at the sametime then something drastic seems to have occured regards NLRI connectivy?

However saying that what do the logs state regrads your eigp neigbour states at the time when connectiviity was lost -

I assume this is a Hub &Spoke setup?

res

Paul

Regards you DWVPN eigrp proccess

Please don't forget to rate any posts that have been helpful.

Thanks.

[Spyridon Katrivesis]

Hii MikhailovskyVV/Paul,

the idea of DMVPN is a Hub and spoke design so it couldn't be anything else other that. Then you deside how that

Hub and spoke design works . For my design its a dual Hub and Spoke solution.  This is to replace MPLS.

Reagrding the ISP routing issue that you suggest can't be. I run IP sla on every router and if there is a routing issue the IP sla will kick in. The IP sla run over the dialers and not through the tunnels monitoring public IPs which means as long as the routers can ping the public IPs over the internet then there is no routing.

I monitor the lines through solarwinds so even through to that I can pick up any disconnection. This is  the reason I mentioned in my original email its poinless to ask me for connectivity issues or basic troubleshooting.

I will not answer a,b,c because  this is basic with DMVPN using EIGRP as a routing protocol.

Regarding question d . It simple stressing a bit more your router is not a big issue. Its good if you can make your network more secure especially if that goes via the public internet. Expecting the unexpected if someone breaks IPsec then he has to break EIGRP authentication as well.

Reagrding tunning the timers is something that you shouldn't do unless you know why you do it especially in my schenario where the Hubs are over 1 Gb link and there is no bottleneck to slow down Eigrp hello packets.

Any other suggestions?

Regards,

Spyros

[paul driver]

Hello

Regards basic DWVPn setup,

a) why do you use "no ip next-hop-self eigrp 90" and "no ip split-horizon eigrp 90" on your spokes?

II think MikhalioyskyVV was querying why you have these above commands on the spoke routers as these are really for the DWVPN HUBs.

"My main question is fomeone had the same problem seen eigrp dropping from both hubs the same time for all the spokes"

You are peering on the tunnel address are you not?,

So if the tunnel connectivity to these address are lost then you will lose peering. correct?

You say you have no underlying ISP issues or the basic troubleshooting steps you have performed dont show any issues and you are not willing to provide these details either?

So we need to ssume the nhrp and crypto iskamp/ipsec sa connection stats are all fine, and routing table is as it should be?

I have asked before can you post ( if applicable)  the logs relating to the loss of eigrp connectivity - it should tell us an indication as to why eigrp dropped?

One thing I did notice it that you have auto summarization enable in eigrp process- this requires to be turned off?

res

Paul

So

Please don't forget to rate any posts that have been helpful.

Thanks.

[Spyridon Katrivesis]

Hi Paul,

thank you for keep helping on that.

This commands are necessary . I found cisco documents where these commands where and wheren't.

Do not forget that this is a spoke to spoke design as well. Spoke to spoke communication goes staright away. DMVPN creates a dynamic tunnel between them and traffic doesn't go via the HUB.

If you are talking for eigrp peering of course the peering is the tunnel IP which means the tunnel subnet. What else could be.....

To loose tunnel connectivity means  generally connectivity lose otherwise GRE has no reason to go down as long as the

Public IP's are UP.

I am not sure if I answered your question regarding peering but if you provide more information I can be more specific.

I guess you mean ISP issues when you say IPS. What kind of information you need me to provide you while all the links ATM interfaces(xDSL) and Fiber links where up?

Where I believe is the problem  could be the vpn. I was dealing for that for months and even the carrier couldn't resolve it. SO thats the reason I created different phase2 from the vpn for each Hub to avoid that failure. Cisco said that there were 4 bugs for the specific IOS I use on spokes but I managed to stop that failure. I had eigrp peers up for 10 weeks which means the DMVPN works fine.

What only worries me is that it dropped on both Hubs. If it was only on the main one I wouldn't be bothered.

Syslog didn't work that day and I noticed that drop when I came back from holidays so  it was really late.

Also I dont want to enable debugging and send that to the syslog. I need to enable debugging step by step

troubleshhoting every time one technology. Its p[ointless to enable eigrp packet debugging if teh vpn fails or NHRP.

I undertsand that you dont have enough information regarding logs but I dont have either. 

Perhaps the problem is either NHRP somehow or vpn phase 1.

EIGRP auto summarazation is desabled by default in the IOS firmware I run.

Thanks,

Spyros

[paul driver · Accepted solution]

Hello

"Do not forget that this is a spoke to spoke design as well. Spoke to spoke communication goes staright away. DMVPN creates a dynamic tunnel between them and traffic doesn't go via the HUB."

I think I have to disagree with you here regards having those eigrp next-hop and split-horizon statements on the spokes

Spokes do indeed establish tunnels between each other however I am on the understanding that the nhrp spokes first need to query the nhrp server cache for the "inside" ip of the spoke it wants to connect to so to verify reachability of the tunnel address- I cannot see or understand at present why this requirement is also needed on the spokes.

When you say of eigrp adjacencies all drooped at the same time- We are still not sure this due to some partial outage that has been found at present but I am thinking for any dynamic failover between the eigrp hubs to work they need to have feasible successors so do these show up in the topology tables? -Maybe you had a situation where both Hubs became SIA state and dropped?

One last thing for a mesh DWVPN (spoke to spoke) isnt PKI is required and not pre-share key and you say cisco said the iOS you was or are using regards IPSec/gre was buggy what did they suggest to do? As in your last post you say you sorted it out.

Res
Paul




Sent from Cisco Technical Support iPad App

[Spyridon Katrivesis]

Hi Paul,

you are absolutely right reagrding the IP split-orizon and next-hop commands. Those should go on both Hubs and not to the spokes.  That most probably was a typo when I was copying the config from the Hubs and adjust that for the spokes.

Clearly cisco says Do NOT use that on the spokes as it will provide instability for EIGRP.

I just removed that from the spokes and lets see how stable EIGRP neigbborship will be.

I dont believe there was any SIA situation. Every spoke is configures as a stub network. There is no relationship or peering between the HUBs. Thre is always one successor and there is no FS

Its only HUb and spoke EIGRP peering. Eigrp routes are coming from the spokes on both Hubs. Both Hubs have always eigrp routes from all the spokes. I run HSRP between the HUbs . When HSRP detects a link failure on the main Hub then the breakout firewall routes traffic to the backup Hub. EIGRP stays at the HUBs. There is no EIGRP rouitng exchange between the HUBs. Its a very simple EIGRP setup. It can be with any routing protocol easily.

I hope you get an idea regarding that design. The failover is not done from EIGRP the failover kicks from HSRP and IP sla tracking.

What cisco couldn't find is the below;

I used to  shut down tunnel 0 on the spoke (tunnel to the primary Hub) and see routes coming from tunnel 1(the secondary Hub). That was expected.

When I unshutdown tunnel 0 although the tunnel was coming Up routes didnt come through. Routes still was coming from the secondary Hub and obviously to resolve that I had to reboot the router.

They said that there are 3-4 bugs on the firmware and I was given the latest IOS version but still I had the same issue.

When I configured a different VPN phase 2 for each tunnel while I was using the same phase 1 resolved the porblem.

So let see the following days how stable EIGRP could be .

If you need to ask anything esle just feel free. Thanks for your help so far.

Regards,

Spyros

[Vasilii Mikhailovskii]

Hello.

I will comment one your point:

When I unshutdown tunnel 0 although the tunnel was coming Up routes didnt come through. Routes still was coming from the secondary Hub and obviously to resolve that I had to reboot the router.

When you unshut the tunnel, the router has neither ISAKMP SA , nor nhrp data. So it's impossible to establish EIGRP unless all the spokes renegotiate ISAKMP/IPSec and register on NHS.

To recover ISAKMP/IPSec quickly you could use crypto isakmp invalid-spi and isakmp keepalive periodic

I guess if ISAKMP renegotiates, NHRP client should trigger the registration on NHS (if not - you could speed it up with lower holdtime value).

[Spyridon Katrivesis]

Hi,

the setup was taken from cisco support documents at least the basics.

I spend so much time troubleshooting. Perhaps the way I was not able to ping th etunnel IPs when I was unshutting down the tunnel 0  game me the idea to focus on the vpn.

I am sure I tried your suggestion . I went to various combination to resolve it.

I managed to make it work when I used different phase II for each Hub  as that was what it was failing on the debbuging.

Although DMVPN is a very smart solution and can replace even MPLS networks its very difficult to troubleshoot as it involved many protocols and many layers...

Guys keep doing what you do . Its a nice way to share knowledge and learn as well from other people's experience.

I will let you know how it goes.

Regards,

Spyros