Solved: Re: Cannot Enable GRE Tunnel Keepalives - Page 2

jpl861 · ‎07-10-2019

Hi there,

I have 2 scenarios.

1. I have a router with 2 tunnels but the they have different tunnel source and destination IPs. Whenever I configure keepalives, it's bringing the protocol down which is expected if it doesn't reach the remote end. This is the AMER side router.

2. Another router in APAC also have 2 GRE tunnels. One going to AMER and one going local. The the source is the same for both tunnels which is loopback 0. The keepalive works if it's configured the tunnel facing AMER but the GRE to another local router does not.

It is sending keepalives based on debugs but I do not understand why it's not working in some tunnels. I did this in lab environment too but same results. The debug shows that the router is sending keepalives to both addresses.

Iaroslav · ‎07-11-2019

OK, here is what I've got from debug in lab (it's detailed one):

R2#u all
*Jul 11 12:08:29.003: IP: s=192.168.12.1 (FastEthernet0/0), d=192.168.12.2, len 56, input feature, proto=47, MCI Check(101), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Jul 11 12:08:29.007: FIBipv4-packet-proc: route packet from FastEthernet0/0 src 192.168.12.1 dst 192.168.12.2
*Jul 11 12:08:29.007: FIBfwd-proc: Default:192.168.12.2/32 receive entry
*Jul 11 12:08:29.007: FIBipv4-packet-proc: packet routing failed
*Jul 11 12:08:29.007: IP: tableid=0, s=192.168.12.1 (FastEthernet0/0), d=192.168.12.2 (FastEthernet0/0), routed via RIB
*Jul 11 12:08:29.011: IP: s=192.168.12.1 (FastEthernet0/0), d=192.168.12.2 (FastEthernet0/0), len 56, rcvd 3, proto=47
*Jul 11 12:08:29.011: IP: s=192.168.12.1 (FastEthernet0/0), d=192.168.12.2, len 56, stop process pak for forus packet, proto=47
*Jul 11 12:08:29.011: IP: s=192.168.12.2 (Tunnel0), d=192.168.12.1, len 28, input feature, proto=47, MCI Check(101), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Jul 11 12:08:2
R2#u all9.011: FIBipv4-packet-proc: route packet from Tunnel0 src 192.168.12.2 dst 192.168.12.1
*Jul 11 12:08:29.011: FIBfwd-proc: A:0.0.0.0/0 process level forwarding
*Jul 11 12:08:29.011: FIBfwd-proc: depth 0 first_idx 0 paths 1 long 0(0)
*Jul 11 12:08:29.011: FIBfwd-proc: try path 0 (of 1) v4-sp first short ext 0(-1)
*Jul 11 12:08:29.011: FIBfwd-proc: v4-sp valid
*Jul 11 12:08:29.011: FIBfwd-proc:  no nh type 8  - deag
*Jul 11 12:08:29.011: FIBfwd-proc: ip_pak_table 1 ip_nh_table 65535 if none nh none deag 1 chg_if 0 via fib 0 path type special prefix
*Jul 11 12:08:29.011: FIBfwd-proc: A:0.0.0.0/0 not enough info to forward via fib (none none)
*Jul 11 12:08:29.011: FIBipv4-packet-proc: packet routing failed
*Jul 11 12:08:29.011: IP: s=192.168.12.2 (Tunnel0), d=192.168.12.1, len 28, unroutable, proto=47
*Jul 11 12:08:29.011: FIBipv4-packet-proc: route packet from Tunnel0 src 192.168.12.2 dst 192.168.12.1
*Jul 11 12:08:29.011: FIBfwd-proc: A:0.0.0.0/0 process level forwarding
*Ju
R2#u alll 11 12:08:29.011: FIBfwd-proc: depth 0 first_idx 0 paths 1 long 0(0)
*Jul 11 12:08:29.011: FIBfwd-proc: try path 0 (of 1) v4-sp first short ext 0(-1)
*Jul 11 12:08:29.011: FIBfwd-proc: v4-sp valid
*Jul 11 12:08:29.011: FIBfwd-proc:  no nh type 8  - deag
*Jul 11 12:08:29.011: FIBfwd-proc: ip_pak_table 1 ip_nh_table 65535 if none nh none deag 1 chg_if 0 via fib 0 path type special prefix
*Jul 11 12:08:29.011: FIBfwd-proc: A:0.0.0.0/0 not enough info to forward via fib (none none)
*Jul 11 12:08:29.011: FIBipv4-packet-proc: packet routing failed
*Jul 11 12:08:29.011: FIBipv4-packet-proc: route packet from (local) src 192.168.21.2 dst 192.168.12.2
*Jul 11 12:08:29.011: FIBfwd-proc: A:0.0.0.0/0 process level forwarding
*Jul 11 12:08:29.011: FIBfwd-proc: depth 0 first_idx 0 paths 1 long 0(0)
*Jul 11 12:08:29.015: FIBfwd-proc: try path 0 (of 1) v4-sp first short ext 0(-1)
*Jul 11 12:08:29.015: FIBfwd-proc: v4-sp valid
*Jul 11 12:08:29.015: FIBfwd-proc:  no nh type 8  - deag
*Jul
R2#u all 11 12:08:29.015: FIBfwd-proc: ip_pak_table 1 ip_nh_table 65535 if none nh none deag 1 chg_if 0 via fib 0 path type special prefix
*Jul 11 12:08:29.015: FIBfwd-proc: A:0.0.0.0/0 not enough info to forward via fib (none none)
*Jul 11 12:08:29.015: FIBipv4-packet-proc: packet routing failed
*Jul 11 12:08:29.015: IP: s=192.168.21.2 (local), d=192.168.12.2, len 56, unroutable
*Jul 11 12:08:29.015:     ICMP type=3, code=1

Notice ICMP type and code - Destination unreachable. As described in the link above, you have to add a static route for tunnel endpoint to VRF RIB, in your case - 'ip route vrf vrf1 10.19.255.118 255.255.255.255 <next_hop> global' or leaking between VRF and global RIB (EVN, for example).
The thing is in how GRE keepalives are implemented. If you get a dump of traffic you should see that there is the following stack of headers:
IP: 10.19.255.118 -> 10.19.255.111
GRE
IP: 10.19.255.111 -> 10.19.255.118
GRE Keepalive

So there is a description step-by-step:
1) Packet is created and forwarded by global RIB - OK
2) Packet is received in global RIB. According to addresses it is related to Tunnel1005 (for example) so it's handed over to vrf1.
3) vrf1 has no idea what to do with 10.19.255.118 - so it drops the keepalive.

However, after you add a route for the loopback into VRF, the router now knows how to forward the packet so keepalives begin to flow.

Discovering the Why
https://braonle.wordpress.com/

jpl861 · ‎07-11-2019

That actually makes sense so thank you for that. However, I also did all VRF setup in the lab. Here's mine.

All tunnels in both R1 and R2 routers are under VRF1 and the tunnel end-points are in global. I actually tried to do wireshark capture on the remote end to see how the packets are structured and yes they are stacked in a weird way. In my lab below where I used 4451 and IOS 16, even if I don't have a route back to loopback 2001 of R1 in R2 VRF1 RIB, it can normally respond to keepalive packets.

R1

interface Tunnel10000

ip vrf forwarding vrf1

ip address 10.200.12.9 255.255.255.252

ip ospf 1 area 0

keepalive 5 2

tunnel source Loopback2001

tunnel destination 2.0.0.2

interface Tunnel10001

ip vrf forwarding vrf1

ip address 10.200.12.13 255.255.255.252

ip ospf 1 area 0

keepalive 5 2

tunnel source Loopback2001

tunnel destination 3.0.0.3

interface Loopback2001

ip address 2.0.0.1 255.255.255.255

R2

interface Tunnel10000

ip vrf forwarding vrf1

ip address 10.200.12.10 255.255.255.252

ip ospf 1 area 0

tunnel source Loopback2002

tunnel destination 2.0.0.1

interface Tunnel10001

ip vrf forwarding vrf1

ip address 10.200.12.14 255.255.255.252

ip ospf 1 area 0

tunnel source Loopback3003

tunnel destination 2.0.0.1

interface Loopback2002

ip address 2.0.0.2 255.255.255.255

interface Loopback3003

ip address 3.0.0.3 255.255.255.255

R2#sh ip route vrf vrf1 | b Gate

Gateway of last resort is not set

10.0.0.0/8 is variably subnetted, 4 subnets, 2 masks

C 10.200.12.8/30 is directly connected, Tunnel10000

L 10.200.12.10/32 is directly connected, Tunnel10000

C 10.200.12.12/30 is directly connected, Tunnel10001

L 10.200.12.14/32 is directly connected, Tunnel10001

*Jul 11 15:39:13.679: FIBipv4-packet-proc: route packet from (local) src 2.0.0.2 dst 2.0.0.1

*Jul 11 15:39:13.679: FIBfwd-proc: packet routed by adj to GigabitEthernet0/0/1.2002 10.200.12.6

*Jul 11 15:39:13.679: FIBipv4-packet-proc: packet routing succeeded

*Jul 11 15:39:23.683: FIBipv4-packet-proc: route packet from (local) src 3.0.0.3 dst 2.0.0.1

*Jul 11 15:39:23.683: FIBfwd-proc: packet routed by adj to GigabitEthernet0/0/1.2002 10.200.12.6

*Jul 11 15:39:23.683: FIBipv4-packet-proc: packet routing succeeded

*Jul 11 15:39:23.683: FIBipv4-packet-proc: route packet from (local) src 2.0.0.2 dst 2.0.0.1

*Jul 11 15:39:23.683: FIBfwd-proc: packet routed by adj to GigabitEthernet0/0/1.2002 10.200.12.6

*Jul 11 15:39:23.683: FIBipv4-packet-proc: packet routing succeeded

*Jul 11 15:39:33.683: FIBipv4-packet-proc: route packet from (local) src 3.0.0.3 dst 2.0.0.1

*Jul 11 15:39:33.683: FIBfwd-proc: packet routed by adj to GigabitEthernet0/0/1.2002 10.200.12.6

*Jul 11 15:39:33.683: FIBipv4-packet-proc: packet routing succeeded

*Jul 11 15:39:33.683: FIBipv4-packet-proc: route packet from (local) src 2.0.0.2 dst 2.0.0.1

*Jul 11 15:39:33.683: FIBfwd-proc: packet routed by adj to GigabitEthernet0/0/1.2002 10.200.12.6

*Jul 11 15:39:33.683: FIBipv4-packet-proc: packet routing succeeded

jpl861 · ‎07-11-2019

Here’s another one. I have created 2 identical labs. My first lab is physical 4400 routers running v16. They will be labeled as R1 and R2.

My second lab is IOU running v15. Will be named as IOU1 and IOU2.

R1:

interface GigabitEthernet0/0/1.2003

encapsulation dot1Q 2003

ip address 192.168.12.1 255.255.255.0

interface Loopback0

ip address 192.168.255.1 255.255.255.255

interface Tunnel12

ip vrf forwarding vrf1

ip address 10.0.12.1 255.255.255.0

keepalive 5 4

tunnel source Loopback0

tunnel destination 192.168.255.2

router eigrp 1212

network 192.168.12.0

network 192.168.255.1 0.0.0.0

R2:

interface GigabitEthernet0/0/1.2003

encapsulation dot1Q 2003

ip address 192.168.12.2 255.255.255.0

interface Loopback0

ip address 192.168.255.2 255.255.255.255

interface Tunnel12

ip vrf forwarding vrf1

ip address 10.0.12.2 255.255.255.0

tunnel source Loopback0

tunnel destination 192.168.255.1

router eigrp 1212

network 192.168.12.0

network 192.168.255.2 0.0.0.0

In the physical lab, keepalives are working just fine without the added static route on R2 under VRF1.

My IOU lab is exactly the same (except I used EIGRP 12 which shouldn’t be a big deal). However, I needed the static route on IOU2 router so it can return the keepalives properly back to IOU1.

Needed configuration on IOU2:

ip route vrf vrf1 192.168.255.1 255.255.255.255 192.168.12.1 global

Here’s the routing table of R2 as evidence. No route back to 192.168.255.1. I am adding the debug output too.

R2#sh ip route vrf vrf1

!

Gateway of last resort is not set

10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks

C 10.0.12.0/24 is directly connected, Tunnel12

L 10.0.12.2/32 is directly connected, Tunnel12

*Jul 11 16:37:54.676: FIBipv4-packet-proc: route packet from (local) src 192.168.255.2 dst 192.168.255.1

*Jul 11 16:37:54.676: FIBfwd-proc: packet routed by adj to GigabitEthernet0/0/1.2003 192.168.12.1

*Jul 11 16:37:54.676: FIBipv4-packet-proc: packet routing succeeded

Here’s for the IOU. Without the static, interface tunnel 12 on IOU1 will die.

IOU2#sh ip route vrf vrf1

!

Gateway of last resort is not set

10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks

C 10.0.12.0/24 is directly connected, Tunnel12

L 10.0.12.2/32 is directly connected, Tunnel12

192.168.255.0/32 is subnetted, 1 subnets

S 192.168.255.1 [1/0] via 192.168.12.1

So probably it’s either the hardware or the software version fixed this kind of problem.

Iaroslav · ‎07-11-2019

Hi, John,

Agreed. So it seems older versions of IOS require VRF RIB modification to make other tunnel endpoint reachable in it while newer ones support taking the necessary route from global RIB. Though the first path might be more general as it allows using tunnel endpoint in another VRF, e.g. frontdoor VRF

Edit 1
Disregard the last statement. The tunnel keepalives work with endpoints in another VRF via 'tunnel vrf' command. This way keepalive routing is kept in the RIB where endpoints reside. So I believe newer versions exhibit a clearer approach from user perspective although it makes the actual forwarding process a bit more complicated to understand. Looks like we just have to keep in mind such a behaviour difference exists.

Edit 2

The official doc on GRE keepalives describes the first type of behaviour

https://www.cisco.com/c/en/us/support/docs/ip/generic-routing-encapsulation-gre/118370-technote-gre-00.html

I see no reference to the version in the document. So maybe the second type of behaviour is actually a bug or undocumented feature.

Discovering the Why
https://braonle.wordpress.com/