We have a FlexVPN setup with certificate authentication working with one hub and two spokes. When we dont run any routing protocol, nhrp works just as expected and creates a dynamic tunnel between spokes if traffic destined between them is detectec by the hub.
We initiate a ping between the two Tunnel interfaces that have an ip assigned by the HUB
Spoke2#ping vrf LAN 172.19.0.14 source 172.19.0.16 repeat 500
Type escape sequence to abort.
Sending 500, 100-byte ICMP Echos to 172.19.0.14, timeout is 2 seconds:
!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!
Success rate is 99 percent (499/500), round-trip min/avg/max = 36/36/80 ms
We usually loose 1 packet between Ping #4 and #6. After
that, the tunnel is up an traffic flows thru it.
Spoke2#show ip nhrp
172.19.0.14/32 (LAN) via 172.19.0.14
Virtual-Access3 created 00:03:57, expire 00:06:02
Type: dynamic, Flags: router nhop rib
NBMA address: 213.200.228.11
172.19.0.16/32 (LAN) via 172.19.0.16
Virtual-Access3 created 00:03:57, expire 01:56:02
Type: dynamic, Flags: router unique local
NBMA address: 83.150.11.22
(no-socket)
In the log, we can see that the tunnel interface comes up
Apr 13 14:35:54.016: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access3, changed state to down
Apr 13 14:35:54.208: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access3, changed state to up
Spoke1#show crypto ikev2 sa
IPv4 Crypto IKEv2 SA
Tunnel-id Local Remote fvrf/ivrf Status
1 213.200.228.11/500 212.25.31.107/500 none/LAN READY
Encr: AES-CBC, keysize: 256, PRF: SHA256, Hash: SHA256, DH Grp:5, Auth sign: RSA, Auth verify: RSA
Life/Active Time: 86400/37080 sec
Tunnel-id Local Remote fvrf/ivrf Status
2 213.200.228.11/500 83.150.11.22/500 none/LAN READY
Encr: AES-CBC, keysize: 256, PRF: SHA256, Hash: SHA256, DH Grp:5, Auth sign: RSA, Auth verify: RSA
Life/Active Time: 86400/13 sec
After 10min, the tunnel is beeing terminated
Apr 13 14:45:54.257: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access3, changed state to down
Apr 13 14:45:54.265: %LINK-3-UPDOWN: Interface Virtual-Access3, changed state to down
So far so good. Now if we enable EIGRP routing on the hub and spokes, it behaves very different:
We initiate a ping between the two LAN Interfaces of the spoke routers
Spoke2#ping vrf LAN 172.16.220.3 source 172.16.142.3 repeat 500
Type escape sequence to abort.
Sending 500, 100-byte ICMP Echos to 172.16.220.3, timeout is 2 seconds:
Packet sent with a source address of 172.16.142.3
!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!
Success rate is 99 percent (497/500), round-trip min/avg/max = 36/42/80 ms
There are multiple packets that are lost as the tunnel is trying to come up.
I matched the entries in the log to match the color where we loose packets.
Apr 13 16:06:25.572: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access3, changed state to down
Apr 13 16:06:25.768: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access3, changed state to up
Apr 13 16:06:25.880: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access3, changed state to down
Apr 13 16:06:33.581: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access3, changed state to down
Apr 13 16:06:33.773: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access3, changed state to up
Apr 13 16:06:33.885: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access3, changed state to down
Apr 13 16:06:33.893: %LINK-3-UPDOWN: Interface Virtual-Access3, changed state to down
Apr 13 16:06:41.601: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access3, changed state to down
Apr 13 16:06:41.793: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access3, changed state to up
Tunnel is finally up after the 3rd try.
Spoke2#show crypto ikev2 sa
IPv4 Crypto IKEv2 SA
Tunnel-id Local Remote fvrf/ivrf Status
2 83.150.11.22/500 213.200.228.11/500 none/LAN READY
Encr: AES-CBC, keysize: 256, PRF: SHA256, Hash: SHA256, DH Grp:5, Auth sign: RSA, Auth verify: RSA
Life/Active Time: 86400/39 sec
Tunnel-id Local Remote fvrf/ivrf Status
1 83.150.11.22/500 212.25.31.107/500 none/LAN READY
Encr: AES-CBC, keysize: 256, PRF: SHA256, Hash: SHA256, DH Grp:5, Auth sign: RSA, Auth verify: RSA
Life/Active Time: 86400/26147 sec
We can see nexthop override flags in the routing table
Spoke2#show ip route vrf LAN eigrp | begin Gate
Gateway of last resort is 172.19.0.1 to network 0.0.0.0
D*EX 0.0.0.0/0 [170/127268571] via 172.19.0.1, 00:34:56
D EX 172.16.0.0/12 [170/76805120] via 172.19.0.1, 00:34:56
172.16.0.0/16 is variably subnetted, 3 subnets, 2 masks
D % 172.16.220.0/24 [90/102405120] via 172.19.0.1, 00:34:47
172.19.0.0/16 is variably subnetted, 4 subnets, 2 masks
D 172.19.0.0/24 [90/76800640] via 172.19.0.1, 00:40:49
D % 172.19.0.14/32 [90/102400000] via 172.19.0.1, 00:01:56
172.27.0.0/28 is subnetted, 1 subnets
D 172.27.1.0 [90/76805120] via 172.19.0.1, 00:40:49
What bugs me most is, that it is not always 3 tries. Most of the time, yes, but i could see up to 5 tries before the tunnel came up. If i deconfigure routing protocol, tunnel comes up at the first try always. One more thing to mention: sometimes we see following error in the log when the flapping is taking place:
Apr 13 16:06:25.848: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=83.150.11.22, prot=50, spi=0x7EBFE3D1(2126504913), srcaddr=213.200.228.11, input interface=Dialer1
Here I share some configuration details in the hopes someone spots the miss configuration:
Hub:
HUB#show running-config interface virtual-template 2
!
interface Virtual-Template2 type tunnel
description VPN Interface generated from VT2
ip unnumbered Loopback2
ip mtu 1400
ip nhrp network-id 2
ip nhrp redirect
ip tcp adjust-mss 1360
tunnel protection ipsec profile MERBAGVPN
end
HUB#show running-config interface loopback 2
!
interface Loopback2
ip address 172.19.0.1 255.255.255.0
end
Note, that there is no VRF on the HUB router yet. I plan on changing that, but it
shouldnt have an impact on the NHRP topic. EIGRP deactivated towards the LAN side.
HUB#show running-config | begin router eigr
router eigrp LAN
!
address-family ipv4 unicast autonomous-system 1
!
af-interface GigabitEthernet0/0/0.300
passive-interface
exit-af-interface
!
topology base
redistribute static
exit-af-topology
network 172.16.0.0 0.15.255.255
exit-address-family
HUB#show running-config | include ip rou
ip route 172.16.0.0 255.240.0.0 gigabitEthernet 0/0/0.30
We had a route map that read the tags from the spokes and manipulated the metrics. We disabled that to see if it would make any difference. it didn't, but we still left it disabled for now.
Spoke:
Spoke2#show running-config interface tunnel 2
!
interface Tunnel2
description VPN connection to HUB
ip vrf forwarding LAN
ip address negotiated
ip mtu 1400
ip nhrp network-id 2
ip nhrp shortcut virtual-template 2
ip nhrp redirect
ip tcp adjust-mss 1360
tunnel source Dialer1
tunnel destination dynamic
tunnel protection ipsec profile VPN2
end
Spoke2#show running-config interface virtual-template 2
!
interface Virtual-Template2 type tunnel
description VPN connection to SPOKES
ip vrf forwarding LAN
ip unnumbered Tunnel2
ip mtu 1400
ip nhrp network-id 2
ip nhrp shortcut virtual-template 2
ip nhrp redirect
ip tcp adjust-mss 1360
tunnel protection ipsec profile VPN2
end
We disable EIGRP on all VTI Interfaces and only add the Tunnel and LAN interface
Spoke2#show running-config | begin router eigr
router eigrp LAN
!
address-family ipv4 unicast vrf LAN autonomous-system 1
!
af-interface default
passive-interface
exit-af-interface
!
af-interface Tunnel2
no passive-interface
exit-af-interface
!
af-interface Vlan10
no passive-interface
exit-af-interface
!
topology base
exit-af-topology
network 172.16.0.0 0.15.255.255
eigrp default-route-tag 20
exit-address-family
If you are wondering why everything has a 2 (like tunnel2) instead of a 1. This is because we are doing the test setup with our 2nd Hub as the first one still serves as static VPN termination for the spokes. It will later on join the flexVPN setup and become the main hub. If you need anything else from the configuration or a debug output, let me know. I didnt post the FlexVPN stuff as i think the problem lies in the routing part.
Any help is much appreciated.
