ā08-26-2022 06:21 AM
There is an existing DMVPN hub on a C881-SEC router that currently has 12 spokes/clients. Along with the 12 spokes there is an additional 4 active BGP sessions to other routers and firewall. Servers behind the primary firewall use BGP for the path to a specific spoke via the hub. The other day a pre-configured C1111-4P (which had previously been tested at my office) was able to connect, establish BGP session, it's routes were passed to the hub which passed to the firewall and another router.
Except I have no traffic, it is as if the hub has an ACL blocking (it does not). "show crypto IPSec sa" on the spoke shows no errors, neither does "sh ip nhrp nhs detail": 10.10.10.1 RE priority = 0 cluster = 0 req-sent 26 req-failed 1 repl-recv 26 (00:02:54 ago)
Showing ip bgp summary, or ip bgp neighbors advertised-routes on both sides is good, ditto show dmvpn. Of the other 12 spokes, they reside in a variety of different locations from being in NAT environment to directly connected to the internet. Most are on fiber, few on cable. The problem one is on Centurylink DSL that previously had a L2L VPN with a Cisco ASA5506 (the C1111-4P replaced that. By default all spokes are configured with ip mtu 1400 and ip tcp adjust-mss 1360.
Tunnel interface configuration:
interface Tunnel1
ip address 10.10.10.15 255.255.255.192
no ip redirects
ip mtu 1400
ip nhrp authentication DMVPN1
ip nhrp map 10.10.10.1 38.xx.xx.6
ip nhrp map 10.10.10.2 38.xx.xx.7
ip nhrp map multicast 38.xx.xx.6
ip nhrp map multicast 38.xx.xx.7
ip nhrp network-id 1
ip nhrp nhs 10.10.10.1
ip nhrp nhs 10.10.10.2
ip tcp adjust-mss 1360
if-state nhrp
tunnel source GigabitEthernet0/0/0
tunnel mode gre multipoint
tunnel protection ipsec profile protect-gre shared
As of now, for example, Spoke #3 has active DMVPN connection, active BGP session, at the hub side both routers and firewall have the route to Spoke #3 via BGP. All have verified the subnet 192.168.3.0/27 is in the routing table. Yet if I ping from the firewall it does not pass. If I ping from Spoke #3 (either from another device directly connecting or from choosing a source interface of either tunnel, loopback, or VLAN1, none passes.
My biggest obstacle at the moment is I simply can not even seem to generate any actual error for which to use as a starting point. The hub can ping the spoke #3s VLAN1, Tunnel, and loopback interfaces (via BGP I pass subnets for the vlan and loopback). I have reached a limit where I do not know where to look next to resolve this problem.
ā08-26-2022 06:51 AM
show me the Hub & spoke Tunnel config
show me the Hub & Spoke BGP config
show ip bgp in both Hub and Spoke
ā08-26-2022 07:38 AM
Below is BGP summary
Spoke1#sh ip bgp summary *Aug 26 14:17:52.453: ISAKMP: (1013):purging node 245083734 BGP router identifier 172.30.61.1, local AS number 64589 BGP table version is 27, main routing table version 27 5 network entries using 1240 bytes of memory 8 path entries using 1088 bytes of memory 4/4 BGP path/bestpath attribute entries using 1152 bytes of memory 3 BGP AS-PATH entries using 144 bytes of memory 0 BGP route-map cache entries using 0 bytes of memory 0 BGP filter-list cache entries using 0 bytes of memory BGP using 3624 total bytes of memory BGP activity 14/9 prefixes, 26/18 paths, scan interval 60 secs 5 networks peaked at 21:20:40 Aug 25 2022 UTC (16:57:14.053 ago) Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 10.10.10.1 4 64571 198 193 27 0 0 02:50:33 3 10.10.10.2 4 64571 194 195 27 0 0 02:50:27 3 Hub1#sh ip bgp summary BGP router identifier 172.30.50.1, local AS number 64571 BGP table version is 3989, main routing table version 3989 41 network entries using 5904 bytes of memory 59 path entries using 4720 bytes of memory 25/19 BGP path/bestpath attribute entries using 3600 bytes of memory 23 BGP AS-PATH entries using 752 bytes of memory 0 BGP route-map cache entries using 0 bytes of memory 0 BGP filter-list cache entries using 0 bytes of memory BGP using 14976 total bytes of memory BGP activity 690/649 prefixes, 2369/2310 paths, scan interval 60 secs Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 10.1.1.2 4 64571 369572 369800 3989 0 0 33w1d 2 10.1.1.3 4 65505 369533 369191 3989 0 0 33w1d 16 10.1.1.4 4 65505 370071 369222 3989 0 0 33w1d 16 10.1.1.5 4 64502 303449 369555 3989 0 0 33w1d 0 10.10.10.3 4 64572 5242 5238 3989 0 0 3d07h 2 10.10.10.4 4 64573 5243 5232 3989 0 0 3d07h 2 10.10.10.5 4 64574 0 0 1 0 0 12w2d Idle 10.10.10.6 4 64575 5239 5231 3989 0 0 3d07h 2 10.10.10.7 4 64577 5234 5241 3989 0 0 3d07h 2 10.10.10.8 4 64578 5249 5235 3989 0 0 3d07h 2 10.10.10.9 4 64579 5238 5243 3989 0 0 3d07h 1 10.10.10.10 4 64580 5239 5228 3989 0 0 3d07h 2 10.10.10.11 4 64581 3808 3805 3989 0 0 2d09h 2 10.10.10.12 4 64582 5242 5234 3989 0 0 3d07h 2 10.10.10.13 4 64583 0 0 1 0 0 4w3d Active 10.10.10.15 4 64589 194 198 3989 0 0 02:50:39 2
10.10.10.5 is a test unit powered off, 10.10.10.13 is at site yet to be installed so those are normal.
Below are the tunnel configs for the hub & spoke. Hub does get the prefix 192.168.3.0/27, it also forwards that route within the BGP sessions with 10.1.1.3 & 10.1.1.5. In routing table for both the firewall and other router I do see:
Routing entry for 192.168.3.0 255.255.255.224, 1 known subnets
B 192.168.3.0 255.255.255.224 [20/0] via 10.10.10.1, 03:03:43
Hub interface Tunnel1 ip address 10.10.10.1 255.255.255.192 no ip redirects ip nhrp authentication DMVPN1 ip nhrp map multicast dynamic ip nhrp network-id 1 tunnel source 38.xx.xx.6 tunnel mode gre multipoint tunnel protection ipsec profile protect-gre Spoke interface Tunnel1 ip address 10.10.10.15 255.255.255.192 no ip redirects ip mtu 1400 ip nhrp authentication DMVPN1 ip nhrp map 10.10.10.1 38.xx.xx.6 ip nhrp map 10.10.10.2 38.xx.xx.7 ip nhrp map multicast 38.xx.xx.6 ip nhrp map multicast 38.xx.xx.7 ip nhrp network-id 1 ip nhrp nhs 10.10.10.1 ip nhrp nhs 10.10.10.2 ip tcp adjust-mss 1360 if-state nhrp tunnel source GigabitEthernet0/0/0 tunnel mode gre multipoint tunnel protection ipsec profile protect-gre shared
Packet captures on the hub when trying to ping from a server behind the firewall to the spoke gets me an ICMP "No response seen". As if the hub just does not forward (packet capture simultaneously on the spoke shows nothing seen)
ā08-26-2022 08:07 AM
Routing entry for 192.168.3.0 255.255.255.224, 1 known subnets
B 192.168.3.0 255.255.255.224 [20/0] via 10.10.10.1, 03:03:43
this is prefix 192.168.3.0 hub learn from Spoke 3
and the AS is different so the BGP is eBGP and next-hop is change and it must be 10.1.1.1 not 10.10.10.1, can you double check this point? FW and other router know 10.1.1.1 not 10.10.10.1
ā08-26-2022 08:42 AM
That was a typo that got missed as I was changing the subnets to post. Corrected:
B 192.168.3.0 255.255.255.224 [20/0] via 10.1.1.1, 03:03:43
The hub's FE4 interface is 10.1.1.1 (AS 64571) peered to 10.1.1.5 (AS 64502). Each DMVPN spoke sends its loopback and at least 1 interface subnet to the hub, that hub forwards to the FW at 10.1.1.5. All subnets are reachable except the one. Tunnel configs match all spokes except for tunnel IP address. BGP config essentially match except for AS #, etc. If this was some sort of licensing issue whether I have reached # of active VPN or # of active BGP sessions then I would assume and expect different behavior and to have something, anything logged indicating this was the problem.
Sample of the IPSEC SA:
Spoke3#sh crypto ipsec sa interface: Tunnel1 Crypto map tag: protect-gre-head-1-IPv4, local addr 172.31.72.2 protected vrf: (none) local ident (addr/mask/prot/port): (172.31.72.2/255.255.255.255/47/0) remote ident (addr/mask/prot/port): (38.xx.xx.6/255.255.255.255/47/0) current_peer 38.xx.xx.6 port 4500 PERMIT, flags={origin_is_acl,} #pkts encaps: 712, #pkts encrypt: 712, #pkts digest: 712 #pkts decaps: 703, #pkts decrypt: 703, #pkts verify: 703 #pkts compressed: 0, #pkts decompressed: 0 #pkts not compressed: 0, #pkts compr. failed: 0 #pkts not decompressed: 0, #pkts decompress failed: 0 #send errors 0, #recv errors 0 local crypto endpt.: 172.31.72.2, remote crypto endpt.: 38.xx.xx.6 plaintext mtu 1326, path mtu 1400, ip mtu 1400, ip mtu idb GigabitEthernet0/0/0 current outbound spi: 0x4D0BCD9A(1292619162) PFS (Y/N): Y, DH group: group2 inbound esp sas: spi: 0xBC096B9A(3154733978) transform: esp-aes esp-sha-hmac , in use settings ={Tunnel UDP-Encaps, } conn id: 2033, flow_id: ESG:33, sibling_flags FFFFFFFF80004048, crypto map: protect-gre-head-1-IPv4 sa timing: remaining key lifetime (k/sec): (4607906/71590) IV size: 16 bytes replay detection support: Y Status: ACTIVE(ACTIVE)
Most of the other spokes are behind front end routers & firewalls, this is the one and only device that has a DSL modem that I can actually log into. The modem is configured to port forward GRE, UDP 4500 & 500 for the previous ASA5506, I have to date left those port forwards in as the new C1111 uses the same host address. I did not think they would have to be removed (in anticipation of putting the 5506 back in line of the C1111 just does not work.
ā08-26-2022 08:59 AM
You mention that You can ping from the Hub to Spoke and VLAN behind Spoke ?
If yes
I see dual Hub config meaning in Spoke there is
show ip nhrp
two static entry
this meaning the Spoke also can learn BGP prefix from two path
may be you face asymmetric
meaning the Hub1 forward traffic but Spoke reply to Hub2.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide