PROBLEM WHIT DMVPN PHASE 3

Acruzgreg · ‎11-29-2010

hi all

II have a network whit DMVPN phase 3 configured. My Hub router is a cisco 2800 series and spoke routers are 881 series

I have 12 spokes some only no funcion properly. the problem is that the users in the network lan cant comunicate con users in the other lan.

I can ping the ip of the interface lan in ther other spoke router but I can´t ping the user in the lan

In the picture below show the comunication:

thanks

Acruzgreg · ‎11-29-2010

when I type clear ip nhrp and then I do ping to ip of the host in the other lan I can obtain one or two reply after I cant receive replys

this is the debug in the spoke

RIAS_01#clear ip nhrp

RIAS_01#ping 10.156.90.2

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.156.90.2, timeout is 2 seconds:

!

*Nov 29 21:51:25.627: NHRP: Cache Delete: Converting prefix: '10.156.90.0' in cache: 0x87F8CC94

*Nov 29 21:51:25.627: NHRP: Cache Delete: Converted entry to host: '10.156.90.1'

*Nov 29 21:51:26.515: NHRP: Attempting to send packet via DEST 172.17.17.250

*Nov 29 21:51:26.515: NHRP: Send Purge Request via Tunnel0 vrf 0, packet size: 80

*Nov 29 21:51:26.515: NHRP: Encapsulation failed for destination 172.17.17.250 out Tunnel0

*Nov 29 21:51:26.515: %DMVPN-3-DMVPN_NHRP_ERROR: Tunnel0: NHRP Encap Error for Purge Request , Reason: protocol generic error (7) on (Tunnel: 172.17.17.1 NBMA: 189.135.220.22)

*Nov 29 21:51:26.515: NHRP: Attempting to send packet via NHS 172.17.17.254

*Nov 29 21:51:26.515: NHRP: NHRP successfully resolved 172.17.17.254 to NBMA 201.117.80.12

*Nov 29 21:51:26.515: NHRP: Encapsulation succeeded. Tunnel IP addr 201.117.80.12

*Nov 29 21:51:26.515: NHRP: Send Purge Request via Tunnel0 vrf 0, packet size: 80

*Nov 29 21:51:26.515: NHRP: 108 bytes out Tunnel0

*Nov 29 21:51:26.551: NHRP: Receive Purge Reply via Tunnel0 vrf 0, packet size: 80

*Nov 29 21:51:26.551: NHRP: netid_in = 0, to_us = 1

*Nov 29 21:51:26.555: %DMVPN-5-CRYPTO_SS: Tunnel0: local address : 189.135.220.22 remote address : 207.248.200.21 socket is DOWN

*Nov 29 21:51:26.555: NHRP: Setting cache expiry for 136.66.183.176 to 5000 milliseconds in cache

*Nov 29 21:51:26.555: NHRP: Serious error. Found an overlay endpoint with no

NHRP subblock attached.

*Nov 29 21:51:26.571: NHRP: MACADDR: if_in null netid-in 0 if_out Tunnel0 netid-out 123

*Nov 29 21:51:26.571: NHRP: Sending packet to NHS 172.17.17.254 on Tunnel0

*Nov 29 21:51:26.571: NHRP: NHRP successfully resolved 172.17.17.254 to NBMA 201.117.80.12

*Nov 29 21:51:26.571: NHRP: Checking for delayed event /172.17.17.250 on list (Tunnel0).

*Nov 29 21:51:26.571: NHRP: No node found.

*Nov 29 21:51:26.571: NHRP: Enqueued NHRP Resolution Request for destination: 172.17.17.250

*Nov 29 21:51:26.583: NHRP: Checking for delayed event /172.17.17.250 on list (Tunnel0).

*Nov 29 21:51:26.583: NHRP: No node found.

*Nov 29 21:51:26.583: NHRP: Sending NHRP Resolution Request for dest: 172.17.17.250 to NHS: 172.17.17.254 using our src: 172.17.17.1

*Nov 29 21:51:26.583: NHRP: Attempting to send packet via DEST 172.17.17.254

*Nov 29 21:51:26.583: NHRP: NHRP successfully resolved 172.17.17.254 to NBMA 201.117.80.12

*Nov 29 21:51:26.583: NHRP: Encapsulation succeeded. Tunnel IP addr 201.117.80.12

*Nov 29 21:51:26.583: NHRP: Send Resolution Request via Tunnel0 vrf 0, packet size: 88

*Nov 29 21:51:26.583: NHRP: 116 bytes out Tunnel0

*Nov 29 21:51:26.611: NHRP: Receive Resolution Request via Tunnel0 vrf 0, packet size: 108

*Nov 29 21:51:26.611: NHRP: netid_in = 123, to_us = 1

*Nov 29 21:51:26.611: NHRP: nhrp_rtlookup yielded Tunnel0

*Nov 29 21:51:26.611: NHRP: request was to us, responding with ouraddress

*Nov 29 21:51:26.611: NHRP: Checking for delayed event 172.17.17.250/172.17.17.1 on list (Tunnel0).

*Nov 29 21:51:26.611: NHRP: No node found.

*Nov 29 21:51:26.611: NHRP: Delaying resolution request nbma src:189.135.220.22 nbma dst:207.248.200.21 reason:IPSEC-IFC: need to wait for IPsec SAs.

*Nov 29 21:51:26.663: NHRP: Receive Traffic Indication via Tunnel0 vrf 0, packet size: 100

*Nov 29 21:51:26.663: NHRP: netid_in = 123, to_us = 1

*Nov 29 21:51:26.663: NHRP: netid_out 0, netid_in 123

*Nov 29 21:51:26.767: NHRP: Cache entry is internal only.

*Nov 29 21:51:26.767: NHRP: MACADDR: if_in null netid-in 0 if_out Tunnel0 netid-out 123

*Nov 29 21:51:26.767: NHRP: Sending packet to NHS 172.17.17.254 on Tunnel0

*Nov 29 21:51:26.767: NHRP: NHRP successfully resolved 172.17.17.254 to NBMA 201.117.80.12

*Nov 29 21:51:26.767: NHRP: Checking for delayed event /172.17.17.250 on list (Tunnel0).

*Nov 29 21:51:26.767: NHRP: No node found.

*Nov 29 21:51:26.767: NHRP: Checking for delayed event /172.17.17.250 on list (Tunnel0).

*Nov 29 21:51:26.767: NHRP: No node found.

Marcin Latosiewicz · ‎11-29-2010

Hi,

Are the spokes pointing their default gateway towards the router? Is the LAN interface on other router always reachable?

What is in common between the affected spokes? :-)

Since first one-two pings work OK I guess this is before the NHRP shortcut is installed. Once shortcut is installed all hell breaks loose

Show me:

- NHRP table on spoke and hubs (before and after several pings)

- running config from failing spoke, working spoke and hub

- versions of spoke and hubs.

Marcin

Acruzgreg · ‎11-30-2010

hi Marcin,

when I do ping to the gateway of the LAN on the spoke I have response, but when I do ping to the host in the LAN don´t have response.

I compared the versions of the spokes and I checked de configuration but I dont find the problem

I attached the settings, tables NHRP and versions of equipment

thanks

Marcin Latosiewicz · ‎11-30-2010

Hi!

I didn't go over the configs yet, it's midnight here and I want to get some sleep.

I'm not sure why this hasn't occured to me before.

Traffic to the box (to LAN interface on remote end) as opposed to traffic through DMVPN to remote subnet, is exempted from CEF.

Would it be possible for you to test with CEF disable on all devices you're pushing traffic through?

Since intially packets will go via hub I'd advise to disable CEF there too.

You can try to do it per interface "no ip route-cache cef" on tunnel interfaces, this should have limited impact on performance/services, but as usual I'd wait until low water mark of traffic.

I'll review the logs tomorrow.

Marcin

Acruzgreg · ‎12-01-2010

Hi Marcin,

I have configured "no ip route-cache cef" in the spokes and hub, any other idea with this problem?

thanks

Marcin Latosiewicz · ‎12-01-2010

One note in general, regarding phase 3 DMVPN.

Spokes should have shortcut and redirect in NHRP.

Hubs should have only redirect NHRP.

Check:

http://www.cisco.com/en/US/prod/collateral/iosswrel/ps6537/ps6586/ps6660/ps6808/prod_white_paper0900aecd8055c34e_ps6658_Products_White_Paper.html

All spokes need to have the commands ip nhrp shortcut and the ip nhrp redirect added to their tunnel interfaces. For the hubs use only ip nhrp redirect.

Apart from from configuration looks ok.

NHRP after ping we can see proper entries, or they seem proper at glance.

First few packets will go over link to hub, only later the packets should be using shortcut route.

Maybe the problem lies at routing protocol level?Is routing stable?

Or maybe IPsec between the two sites is not establishing? After you ping can you check "show crypto ipsec sa" if you see the tunnel to 207.248.X.X ?

Marcin