Disclaimer: This content has been initially created by Alex Honore, Graham Barlet, Raffaele Brancaleoni from Cisco.
Introduction.
This document is indetnded to show troubleshooting order of operation when troubleshooting DMVPN setup.
It will also discuss responsibilites of different components, but not in detail.
It is intended as a troubleshooting framework and a referance on which commands to be used.
Before you start.
Please remember, that you should always start troubleshooting of DMVPN on spokes and not hubs.
Impact of changes on spokes is limited to particular location, while changes/troubleshooting on hub side are potetially impacting entire DMVPN cloud.
Using a recent revision of IOS software helps overcome a lot of problems with monitoring and troubleshooting.
Add this command to running configuration on both sides of problematic tunnel:
crypto logging session
allows to see indepth information about failures and potential reasons.
Order of troubleshooting
This section describes step by step approach to troubleshooting DMVPN.
Using this framework allows to easily verify offending component and apply remediation.
NHRP
NHRP is the trigger for spoke to hub initial tunnel establishment.
Useful commands.
Key components at this stage can be verfied in the configuration:
show run interface tunnel X
Things to verify:
- Spoke: contains a static NHRP mapping indicating that hubs' tunnel interface IP is behind a particular NBMA address
- Spoke:NHS should point to hubs' tunnel interface IP address.
- Spoke: If multicast is needed - multicast mapping for NBMA address of hub(s).
- Spoke and hub: NHRP network id is configured.
- Hub: If multicast is needed - multicast dynamic map.
IKE
IKE provides keying material to IPsec and is therefor a preamble to it.
It is during IKE negotiatin that calls are made to PKI infrastructure.
Useful commands.
Understand what state IKE is at the moment.
show crypto isakmp sa
MM_NO_STATE will mean a failure or expired SA.
MM_KEY_EXCH will signifiy that we are in process or stuck during authentication (PKI or pre-shared).
PKI
PKI is component responsible for the authentication of peers in IKE negotiation.
To accomplish its task, it requites the retrieval and processing of revocation information (CRL).
Useful commands.
Check the time:
sh clock
Check the validity of certificates on both sides:
show crypto pki cert
Check if you have a recent version of CRL downloaded:
show cry pki crls
Check availability of CDP:
telnet IP_OR_HOSTNAME 80
GET /<name_of_crl_file> HTTP/1.0
IPsec
IPsec by itself is stateless, but encrypted traffic emission and reception capabilities must be confirmed (think about IP protocol type 50/51 filtering in transport network for ex.)
Useful commands.
Verfies if both IKE and IPsec are up.
show crypto session [detail]
For working session you will see TWO active SAs per peer. One SA for inbound traffic and one SA for outbound traffic.
Verify if tunnel is passing traffic.
show crypto ipsec sa peer IP
Working scenario - encaps and decaps section counters are non-zero AND increasing.
GRE
Provide multiprotocol support capability & multicast support to IPsec along with simplifying configs. Only fails if tunnel keys between GRE endpoints are different
Useful commands.
On both hub and spoke
show tunnel endpoint tunnel X
On spoke this information should be populated by static NHRP mapping.
NHRP (again!)
NHRP performs the initial registration to Hub which forms the basis of DMVPN and is at the root of the spoke to spoke dynamic IPsec tunnels construction
Useful commands.
On spoke, check if you see responses from NHS.
show ip nhrp nhs
On hub , verify if dynamic NHRP entiries were populated.
show ip nhrp tunnel X
show ip nhrp multicast
Routing protocol
Propagate the necessary routing information between hubs and spokes
Useful commands.
It's hard to point to one set of commands to troubleshoot this - DMVPN supports multiple routing protocols.
In any case you need to verify if a particular adjacancy/neighborship came up correctly and whether you are sending and receiving correct prefixes.
BGP
BGP is by far the best protocol to scale in DMVPN networks. With recent additions to it (like dynamic listener) we are able to create working configuration hassle-free.
- Verify if neighborship is up
show ip bgp [vpnv4 vrf VRF_NAME] summary
- Verify which prefixes you have in BGP table and from whom
show ip bgp [vpnv4 vrf VRF_NAME]
- Verify what prefixes you are receiving from particular neighbor
show ip bgp [vpnv4 vrf VRF_NAME] neighbor IP_ADDRESS routes
- Verify what prefixes you are sending to particular neighbor.
show ip bgp [vpnv4 vrf VRF_NAME] neighbor IP_ADDRESS advertised
- Verify what prefixes made it into routing table.
show ip router bgp
EIGRP
- Verify that neighborship is up
show ip eigrp neighbor
- Verify the topology table to see what prefixes are inserted.
show ip eigrp topology
- Verify what prefixes made it into routing table.
show ip router eigrp
