Re: DMVPN hub does not pass traffic for 1 spoke only

TRENT WAITE · ‎08-26-2022

There is an existing DMVPN hub on a C881-SEC router that currently has 12 spokes/clients. Along with the 12 spokes there is an additional 4 active BGP sessions to other routers and firewall. Servers behind the primary firewall use BGP for the path to a specific spoke via the hub. The other day a pre-configured C1111-4P (which had previously been tested at my office) was able to connect, establish BGP session, it's routes were passed to the hub which passed to the firewall and another router.

Except I have no traffic, it is as if the hub has an ACL blocking (it does not). "show crypto IPSec sa" on the spoke shows no errors, neither does "sh ip nhrp nhs detail": 10.10.10.1 RE priority = 0 cluster = 0 req-sent 26 req-failed 1 repl-recv 26 (00:02:54 ago)

Showing ip bgp summary, or ip bgp neighbors advertised-routes on both sides is good, ditto show dmvpn. Of the other 12 spokes, they reside in a variety of different locations from being in NAT environment to directly connected to the internet. Most are on fiber, few on cable. The problem one is on Centurylink DSL that previously had a L2L VPN with a Cisco ASA5506 (the C1111-4P replaced that. By default all spokes are configured with ip mtu 1400 and ip tcp adjust-mss 1360.

Tunnel interface configuration:

interface Tunnel1
ip address 10.10.10.15 255.255.255.192
no ip redirects
ip mtu 1400
ip nhrp authentication DMVPN1
ip nhrp map 10.10.10.1 38.xx.xx.6
ip nhrp map 10.10.10.2 38.xx.xx.7
ip nhrp map multicast 38.xx.xx.6
ip nhrp map multicast 38.xx.xx.7
ip nhrp network-id 1
ip nhrp nhs 10.10.10.1
ip nhrp nhs 10.10.10.2
ip tcp adjust-mss 1360
if-state nhrp
tunnel source GigabitEthernet0/0/0
tunnel mode gre multipoint
tunnel protection ipsec profile protect-gre shared

As of now, for example, Spoke #3 has active DMVPN connection, active BGP session, at the hub side both routers and firewall have the route to Spoke #3 via BGP. All have verified the subnet 192.168.3.0/27 is in the routing table. Yet if I ping from the firewall it does not pass. If I ping from Spoke #3 (either from another device directly connecting or from choosing a source interface of either tunnel, loopback, or VLAN1, none passes.

My biggest obstacle at the moment is I simply can not even seem to generate any actual error for which to use as a starting point. The hub can ping the spoke #3s VLAN1, Tunnel, and loopback interfaces (via BGP I pass subnets for the vlan and loopback). I have reached a limit where I do not know where to look next to resolve this problem.

MHM Cisco World · ‎08-26-2022

show me the Hub & spoke Tunnel config
show me the Hub & Spoke BGP config
show ip bgp in both Hub and Spoke

TRENT WAITE · ‎08-26-2022

Below is BGP summary

Spoke1#sh ip bgp summary
*Aug 26 14:17:52.453: ISAKMP: (1013):purging node 245083734
BGP router identifier 172.30.61.1, local AS number 64589
BGP table version is 27, main routing table version 27
5 network entries using 1240 bytes of memory
8 path entries using 1088 bytes of memory
4/4 BGP path/bestpath attribute entries using 1152 bytes of memory
3 BGP AS-PATH entries using 144 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 3624 total bytes of memory
BGP activity 14/9 prefixes, 26/18 paths, scan interval 60 secs
5 networks peaked at 21:20:40 Aug 25 2022 UTC (16:57:14.053 ago)

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.10.10.1      4        64571     198     193       27    0    0 02:50:33        3
10.10.10.2      4        64571     194     195       27    0    0 02:50:27        3



Hub1#sh ip bgp summary
BGP router identifier 172.30.50.1, local AS number 64571
BGP table version is 3989, main routing table version 3989
41 network entries using 5904 bytes of memory
59 path entries using 4720 bytes of memory
25/19 BGP path/bestpath attribute entries using 3600 bytes of memory
23 BGP AS-PATH entries using 752 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 14976 total bytes of memory
BGP activity 690/649 prefixes, 2369/2310 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.1.1.2        4        64571  369572  369800     3989    0    0 33w1d           2
10.1.1.3        4        65505  369533  369191     3989    0    0 33w1d          16
10.1.1.4        4        65505  370071  369222     3989    0    0 33w1d          16
10.1.1.5        4        64502  303449  369555     3989    0    0 33w1d           0
10.10.10.3      4        64572    5242    5238     3989    0    0 3d07h           2
10.10.10.4      4        64573    5243    5232     3989    0    0 3d07h           2
10.10.10.5      4        64574       0       0        1    0    0 12w2d    Idle
10.10.10.6      4        64575    5239    5231     3989    0    0 3d07h           2
10.10.10.7      4        64577    5234    5241     3989    0    0 3d07h           2
10.10.10.8      4        64578    5249    5235     3989    0    0 3d07h           2
10.10.10.9      4        64579    5238    5243     3989    0    0 3d07h           1
10.10.10.10     4        64580    5239    5228     3989    0    0 3d07h           2
10.10.10.11     4        64581    3808    3805     3989    0    0 2d09h           2
10.10.10.12     4        64582    5242    5234     3989    0    0 3d07h           2
10.10.10.13     4        64583       0       0        1    0    0 4w3d     Active
10.10.10.15     4        64589     194     198     3989    0    0 02:50:39        2

10.10.10.5 is a test unit powered off, 10.10.10.13 is at site yet to be installed so those are normal.

Below are the tunnel configs for the hub & spoke. Hub does get the prefix 192.168.3.0/27, it also forwards that route within the BGP sessions with 10.1.1.3 & 10.1.1.5. In routing table for both the firewall and other router I do see:

Routing entry for 192.168.3.0 255.255.255.224, 1 known subnets
B 192.168.3.0 255.255.255.224 [20/0] via 10.10.10.1, 03:03:43

Hub
interface Tunnel1
 ip address 10.10.10.1 255.255.255.192
 no ip redirects
 ip nhrp authentication DMVPN1
 ip nhrp map multicast dynamic
 ip nhrp network-id 1
 tunnel source 38.xx.xx.6
 tunnel mode gre multipoint
 tunnel protection ipsec profile protect-gre

Spoke
interface Tunnel1
 ip address 10.10.10.15 255.255.255.192
 no ip redirects
 ip mtu 1400
 ip nhrp authentication DMVPN1
 ip nhrp map 10.10.10.1 38.xx.xx.6
 ip nhrp map 10.10.10.2 38.xx.xx.7
 ip nhrp map multicast 38.xx.xx.6
 ip nhrp map multicast 38.xx.xx.7
 ip nhrp network-id 1
 ip nhrp nhs 10.10.10.1
 ip nhrp nhs 10.10.10.2
 ip tcp adjust-mss 1360
 if-state nhrp
 tunnel source GigabitEthernet0/0/0
 tunnel mode gre multipoint
 tunnel protection ipsec profile protect-gre shared

Packet captures on the hub when trying to ping from a server behind the firewall to the spoke gets me an ICMP "No response seen". As if the hub just does not forward (packet capture simultaneously on the spoke shows nothing seen)

MHM Cisco World · ‎08-26-2022

Routing entry for 192.168.3.0 255.255.255.224, 1 known subnets
B 192.168.3.0 255.255.255.224 [20/0] via 10.10.10.1, 03:03:43

this is prefix 192.168.3.0 hub learn from Spoke 3
and the AS is different so the BGP is eBGP and next-hop is change and it must be 10.1.1.1 not 10.10.10.1, can you double check this point? FW and other router know 10.1.1.1 not 10.10.10.1

TRENT WAITE · ‎08-26-2022

That was a typo that got missed as I was changing the subnets to post. Corrected:

B 192.168.3.0 255.255.255.224 [20/0] via 10.1.1.1, 03:03:43

The hub's FE4 interface is 10.1.1.1 (AS 64571) peered to 10.1.1.5 (AS 64502). Each DMVPN spoke sends its loopback and at least 1 interface subnet to the hub, that hub forwards to the FW at 10.1.1.5. All subnets are reachable except the one. Tunnel configs match all spokes except for tunnel IP address. BGP config essentially match except for AS #, etc. If this was some sort of licensing issue whether I have reached # of active VPN or # of active BGP sessions then I would assume and expect different behavior and to have something, anything logged indicating this was the problem.

Sample of the IPSEC SA:

Spoke3#sh crypto ipsec sa

interface: Tunnel1
    Crypto map tag: protect-gre-head-1-IPv4, local addr 172.31.72.2

   protected vrf: (none)
   local  ident (addr/mask/prot/port): (172.31.72.2/255.255.255.255/47/0)
   remote ident (addr/mask/prot/port): (38.xx.xx.6/255.255.255.255/47/0)
   current_peer 38.xx.xx.6 port 4500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 712, #pkts encrypt: 712, #pkts digest: 712
    #pkts decaps: 703, #pkts decrypt: 703, #pkts verify: 703
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 0, #recv errors 0

     local crypto endpt.: 172.31.72.2, remote crypto endpt.: 38.xx.xx.6
     plaintext mtu 1326, path mtu 1400, ip mtu 1400, ip mtu idb GigabitEthernet0/0/0
     current outbound spi: 0x4D0BCD9A(1292619162)
     PFS (Y/N): Y, DH group: group2

     inbound esp sas:
      spi: 0xBC096B9A(3154733978)
        transform: esp-aes esp-sha-hmac ,
        in use settings ={Tunnel UDP-Encaps, }
        conn id: 2033, flow_id: ESG:33, sibling_flags FFFFFFFF80004048, crypto map: protect-gre-head-1-IPv4
        sa timing: remaining key lifetime (k/sec): (4607906/71590)
        IV size: 16 bytes
        replay detection support: Y
        Status: ACTIVE(ACTIVE)

Most of the other spokes are behind front end routers & firewalls, this is the one and only device that has a DSL modem that I can actually log into. The modem is configured to port forward GRE, UDP 4500 & 500 for the previous ASA5506, I have to date left those port forwards in as the new C1111 uses the same host address. I did not think they would have to be removed (in anticipation of putting the 5506 back in line of the C1111 just does not work.

MHM Cisco World · ‎08-26-2022

You mention that You can ping from the Hub to Spoke and VLAN behind Spoke ?
If yes
I see dual Hub config meaning in Spoke there is
show ip nhrp
two static entry
this meaning the Spoke also can learn BGP prefix from two path
may be you face asymmetric
meaning the Hub1 forward traffic but Spoke reply to Hub2.