NHRP route used although spoke to spoke peering is down.

[Larry Sullivan]

Hi,

I have two NHRP spokes whose peering to each other is down.  BGP down as well of course.  Yet when I do a "show ip route (spoke neighbor's LAN address) the next hop is the neighbors tunnel IP.  The very tunnel IP that is down.  So it appears NHRP is essentially black holing any traffic between these two spokes.  Is this normal behavior?  A bug?  Or do I need to add some specialized NHRP command to prevent this.  Already tried "IP NHRP route" and "NHRP route-watch" to no avail.  Also tried clearing NHRP shortcuts.  Another interesting part is the routing tables say it's learning the routes to each other's LAN addresses via BGP but as stated before BGP peering is down due to NHRP being down which is the result of a service provider issues that is being pushed.  Any assistance or insight appreciated.

[paul driver]

Hello

---

@Larry Sullivan wrote:

Hi,

I have two NHRP spokes whose peering to each other is down.  BGP down as well of course.  Yet when I do a "show ip route (spoke neighbor's LAN address) the next hop is the neighbors tunnel IP.  The very tunnel IP that is down.  

---

Can you post your tunnel/bgp configuration for both NHC's and the NHS please.
sh run int tunnel xx
sh run | sec router bgp
sh ip route | be N

[Larry Sullivan]

Spoke 1
interface Tunnel199
description mGRE MULTICAST
bandwidth 100000
ip address 10.99.198.12 255.255.255.0
no ip redirects
no ip unreachables
no ip proxy-arp
ip mtu 1400
ip pim nbma-mode
ip pim sparse-dense-mode
ip nbar protocol-discovery
ip flow monitor LIVEACTION-FLOWMONITOR input
ip flow monitor LIVEACTION-FLOWMONITOR output
ip nhrp authentication xxxx
ip nhrp map multicast dynamic
ip nhrp map 10.99.198.1 10.75.99.1
ip nhrp map multicast 10.75.99.1
ip nhrp map 10.99.198.3 10.75.99.3
ip nhrp map multicast 10.75.99.3
ip nhrp map 10.99.198.5 10.75.99.5
ip nhrp map multicast 10.75.99.5
ip nhrp map 10.99.198.6 10.75.99.6
ip nhrp map multicast 10.75.99.6
ip nhrp map 10.99.198.7 10.75.99.7
ip nhrp map multicast 10.75.99.7
ip nhrp map 10.99.198.8 10.75.99.8
ip nhrp map multicast 10.75.99.8
ip nhrp map 10.99.198.9 10.75.99.9
ip nhrp map multicast 10.75.99.9
ip nhrp map 10.99.198.10 10.75.99.10
ip nhrp map multicast 10.75.99.10
ip nhrp map 10.99.198.11 10.75.99.11
ip nhrp map multicast 10.75.99.11
ip nhrp map 10.99.198.14 10.75.99.14
ip nhrp map multicast 10.75.99.14
ip nhrp map 10.99.198.2 10.75.99.2
ip nhrp map multicast 10.75.99.2
ip nhrp map 10.99.198.17 10.75.99.17
ip nhrp map multicast 10.75.99.17
ip nhrp map 10.99.198.4 10.75.99.4
ip nhrp map multicast 10.75.99.4
ip nhrp network-id 199
ip nhrp holdtime 120
ip nhrp nhs 10.99.198.1
ip nhrp nhs 10.99.198.3
ip nhrp nhs 10.99.198.5
ip nhrp nhs 10.99.198.6
ip nhrp nhs 10.99.198.7
ip nhrp nhs 10.99.198.8
ip nhrp nhs 10.99.198.9
ip nhrp nhs 10.99.198.10
ip nhrp nhs 10.99.198.11
ip nhrp nhs 10.99.198.14
ip nhrp nhs 10.99.198.2
ip nhrp nhs 10.99.198.17
ip nhrp nhs 10.99.198.4
ip nhrp shortcut
ip igmp version 3
ip tcp adjust-mss 1360
tunnel source 10.75.99.12
tunnel mode gre multipoint

Spoke 1
s run | s bgp
router bgp xxxx
bgp always-compare-med
bgp log-neighbor-changes
bgp deterministic-med
network 10.40.229.0 mask 255.255.255.224
network 10.50.229.0 mask 255.255.255.224
network 10.50.229.16 mask 255.255.255.255
network 10.50.229.17 mask 255.255.255.255
network 10.50.229.18 mask 255.255.255.255
network 10.50.229.19 mask 255.255.255.255
network 10.50.229.22 mask 255.255.255.255
network 10.50.229.23 mask 255.255.255.255
network 10.50.229.27 mask 255.255.255.255
network 10.50.229.28 mask 255.255.255.255
network 10.50.229.29 mask 255.255.255.255
network 10.50.229.30 mask 255.255.255.255
network 10.250.21.120 mask 255.255.255.255
timers bgp 15 45
neighbor EBGP peer-group
neighbor EBGP soft-reconfiguration inbound
neighbor EBGP prefix-list BLOCKALL in
neighbor EBGP prefix-list ANNOUNCE out
neighbor IBGP peer-group
neighbor IBGP remote-as xxxx
neighbor IBGP next-hop-self
neighbor IBGP soft-reconfiguration inbound
neighbor 10.40.229.3 peer-group IBGP
neighbor 10.40.229.3 description
neighbor 10.99.198.1 remote-as xxxx
neighbor 10.99.198.1 peer-group EBGP
neighbor 10.99.198.1 description
neighbor 10.99.198.4 remote-as xxxx
neighbor 10.99.198.4 peer-group EBGP
neighbor 10.99.198.4 description Spoke_2

Spoke 2
interface Tunnel199
description mGRE MULTICAST
bandwidth 100000
ip address 10.99.198.4 255.255.255.0
no ip redirects
no ip unreachables
no ip proxy-arp
ip mtu 1400
ip pim nbma-mode
ip pim sparse-dense-mode
ip nbar protocol-discovery
ip flow monitor LIVEACTION-FLOWMONITOR input
ip flow monitor LIVEACTION-FLOWMONITOR output
ip nhrp authentication xxxx
ip nhrp map multicast dynamic
ip nhrp map 10.99.198.1 10.75.99.1
ip nhrp map multicast 10.75.99.1
ip nhrp map 10.99.198.2 10.75.99.2
ip nhrp map multicast 10.75.99.2
ip nhrp map 10.99.198.3 10.75.99.3
ip nhrp map multicast 10.75.99.3
ip nhrp map 10.99.198.5 10.75.99.5
ip nhrp map multicast 10.75.99.5
ip nhrp map 10.99.198.6 10.75.99.6
ip nhrp map multicast 10.75.99.6
ip nhrp map 10.99.198.7 10.75.99.7
ip nhrp map multicast 10.75.99.7
ip nhrp map 10.99.198.8 10.75.99.8
ip nhrp map multicast 10.75.99.8
ip nhrp map 10.99.198.9 10.75.99.9
ip nhrp map multicast 10.75.99.9
ip nhrp map 10.99.198.10 10.75.99.10
ip nhrp map multicast 10.75.99.10
ip nhrp map 10.99.198.11 10.75.99.11
ip nhrp map multicast 10.75.99.11
ip nhrp map 10.99.198.14 10.75.99.14
ip nhrp map multicast 10.75.99.14
ip nhrp map 10.99.198.17 10.75.99.17
ip nhrp map multicast 10.75.99.17
ip nhrp map 10.99.198.12 10.75.99.12
ip nhrp map multicast 10.75.99.12
ip nhrp network-id 199
ip nhrp holdtime 120
ip nhrp nhs 10.99.198.1
ip nhrp nhs 10.99.198.2
ip nhrp nhs 10.99.198.3
ip nhrp nhs 10.99.198.5
ip nhrp nhs 10.99.198.6
ip nhrp nhs 10.99.198.7
ip nhrp nhs 10.99.198.8
ip nhrp nhs 10.99.198.9
ip nhrp nhs 10.99.198.10
ip nhrp nhs 10.99.198.11
ip nhrp nhs 10.99.198.14
ip nhrp nhs 10.99.198.17
ip nhrp nhs 10.99.198.12
ip nhrp shortcut
ip igmp version 3
ip tcp adjust-mss 1360
tunnel source 10.75.99.4
tunnel mode gre multipoint

Spoke 2
#s run | s bgp
router bgp xxxx
bgp always-compare-med
bgp log-neighbor-changes
bgp deterministic-med
network 10.40.195.0 mask 255.255.255.192
network 10.50.195.0 mask 255.255.255.192
network 10.250.21.136 mask 255.255.255.255
timers bgp 15 45
neighbor EBGP peer-group
neighbor EBGP soft-reconfiguration inbound
neighbor EBGP prefix-list BLOCKALL in
neighbor EBGP prefix-list announce out
neighbor IBGP peer-group
neighbor IBGP remote-as xxxx
neighbor IBGP next-hop-self
neighbor IBGP soft-reconfiguration inbound
neighbor 10.40.195.3 peer-group IBGP
neighbor 10.40.195.3 description SECONDARY_ROUTER
neighbor 10.99.198.1 remote-as xxxx
neighbor 10.99.198.1 peer-group EBGP
neighbor 10.99.198.1 description
neighbor 10.99.198.12 remote-as xxxx
neighbor 10.99.198.12 peer-group EBGP
neighbor 10.99.198.12 description Spoke_1

Hub
interface Tunnel199
description mGRE MULTICAST
bandwidth 2000000
ip flow monitor LIVEACTION-FLOWMONITOR input
ip flow monitor LIVEACTION-FLOWMONITOR output
ip address 10.99.198.1 255.255.255.0
no ip redirects
no ip unreachables
no ip proxy-arp
ip mtu 1400
ip nbar protocol-discovery
ip pim nbma-mode
ip pim sparse-dense-mode
ip nhrp authentication xxxx
ip nhrp network-id 199
ip nhrp holdtime 120
ip nhrp redirect
ip tcp adjust-mss 1360
ip igmp version 3
tunnel source 10.75.99.1
tunnel mode gre multipoint

sh ip route | be N output just returns BGP peering which between these two is down. Actual Hub is 10.99.198.1 which both are BGP peered and up to that.

Edit: Provider just fixed their end so can't get "show IP route (neighbor LAN address)" anymore, but I did save the output from when it was down.

s ip route 10.50.229.0
Routing entry for 10.50.229.0/27
Known via "bgp xxxx", distance 20, metric 0
Tag xxxx, type external
Last update from 10.99.198.12 13:11:00 ago
Routing Descriptor Blocks:
* 10.99.198.12, from 10.99.198.1, 13:11:00 ago
Route metric is 0, traffic share count is 1
AS Hops 2
Route tag xxxx

[Georg Pauwen]

Hello,

for how long is the BGP down ? Default NHRP registration is one third of the holdtime (which is 40 minutes, so a registration reuqest would be sent every 13 minutes approximately; it is declared down when 3 requests are missed, so that would be after 40 minutes). You could try and change the registration timeout to a lower value and check the status after that time value has expired:

ip nhrp registration timeout 100

[Larry Sullivan]

Hi Georg,

NHRP and BGP were down for about 14 hours so I don't think timers were the issue.  Thanks.

[Georg Pauwen]

Hello,

I saw that your holdtime was set to 120 seconds already, so indeed the timers couldn't be the problem...:(

What routers and IOS versions is this on ? Always worth checking for bugs...

[paul driver]

Hello
Not sure why you have so many static mapping when you are running phase 2 DMVPN to what looks like a single hub

So assuming you underlying L2 connectivity is working just amend your NHRP config to be this and then test again, remove the all the other static mapping from your spokes, Then once you have the DMVPN working correctly you can focus on your bgp connectivity over the DMVPN.

Hub
interface Tunnel199
ip address 10.99.198.1 255.255.255.0
tunnel source 10.75.99.1
tunnel mode gre multipoint
ip nhrp map multicast dynamic <---------add this to your hub
ip nhrp authentication xxxx
ip nhrp network-id 199
ip nhrp holdtime 120
ip nhrp redirect <------------------not required

Spoke1/2
interface Tunnel xx
ip address 10.99.198.X 255.255.255.0
tunnel source 10.75.99.x
tunnel mode gre multipoint
ip nhrp authentication xxxx
ip nhrp map 10.99.198.1 10.75.99.1
ip nhrp network-id 199
ip nhrp holdtime 120
ip nhrp nhs 10.99.198.1
ip nhrp map multicast 10.75.99.1
ip nhrp map multicast dynamic <----not required
ip nhrp shortcut <----------------not required
ip nhrp map multicast x.x.x..x <-----not required
ip nhrp map x.x.x.x x.x.x.x <-------not required
etc....

[Larry Sullivan]

Hi Paul,

The config implementation for this was not of my own doing and I did mention before that we didn't need all the static for the spokes pointing to each other.  Also, if we want spoke to spoke communications isn't redirect needed at hub and shortcut on spokes?

[paul driver]

Hello Larry
As DMVPN is primarily geared to work best with eigrp and as you may well be aware with the eigrp protocol when a network is received by a eigrp router it will re-advertise that same network to its neighbors as if it originated from itself unless that is you disable " ip next-hop-self eigrp xx" feature, however with nhrp redirect (nhs) and shortcut (nhc) applied then you don’t need to disable this feature and spoke-to-spoke communication will work but now named as phase 3 dmvpn  and as your not even running eigrp I wasnt infering you dont need phase 3 I was just basically trying to assist in getting your dmvpn into an active state prior to appending these two features if they were indeed requried at all

DMVPN spoke-spoke communication does occurs at phase 2 for routing protocols (rip/eigrp/ospf/bgp) but with the introduction of nhrp redirect/shortcut features its then dmvpn becomes phase 3 which can account for for things like smaller routing tables as it allows summarization which phase 2 doesnt , And it removes protocol limitations like the one i have stated previously with eigrp.