Here’s another one. I have created 2 identical labs. My first lab is physical 4400 routers running v16. They will be labeled as R1 and R2. My second lab is IOU running v15. Will be named as IOU1 and IOU2. R1: interface GigabitEthernet0/0/1.2003 encapsulation dot1Q 2003 ip address 192.168.12.1 255.255.255.0   interface Loopback0 ip address 192.168.255.1 255.255.255.255   interface Tunnel12 ip vrf forwarding vrf1 ip address 10.0.12.1 255.255.255.0 keepalive 5 4 tunnel source Loopback0 tunnel destination 192.168.255.2   router eigrp 1212 network 192.168.12.0 network 192.168.255.1 0.0.0.0   R2: interface GigabitEthernet0/0/1.2003 encapsulation dot1Q 2003 ip address 192.168.12.2 255.255.255.0   interface Loopback0 ip address 192.168.255.2 255.255.255.255   interface Tunnel12 ip vrf forwarding vrf1 ip address 10.0.12.2 255.255.255.0 tunnel source Loopback0 tunnel destination 192.168.255.1   router eigrp 1212 network 192.168.12.0 network 192.168.255.2 0.0.0.0   In the physical lab, keepalives are working just fine without the added static route on R2 under VRF1.     My IOU lab is exactly the same (except I used EIGRP 12 which shouldn’t be a big deal). However, I needed the static route on IOU2 router so it can return the keepalives properly back to IOU1.   Needed configuration on IOU2: ip route vrf vrf1 192.168.255.1 255.255.255.255 192.168.12.1 global       Here’s the routing table of R2 as evidence. No route back to 192.168.255.1. I am adding the debug output too.   R2#sh ip route vrf vrf1 ! Gateway of last resort is not set         10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks C        10.0.12.0/24 is directly connected, Tunnel12 L        10.0.12.2/32 is directly connected, Tunnel12   *Jul 11 16:37:54.676: FIBipv4-packet-proc: route packet from (local) src 192.168.255.2 dst 192.168.255.1 *Jul 11 16:37:54.676: FIBfwd-proc: packet routed by adj to GigabitEthernet0/0/1.2003 192.168.12.1 *Jul 11 16:37:54.676: FIBipv4-packet-proc: packet routing succeeded   Here’s for the IOU. Without the static, interface tunnel 12 on IOU1 will die.   IOU2#sh ip route vrf vrf1 ! Gateway of last resort is not set         10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks C        10.0.12.0/24 is directly connected, Tunnel12 L        10.0.12.2/32 is directly connected, Tunnel12       192.168.255.0/32 is subnetted, 1 subnets S        192.168.255.1 [1/0] via 192.168.12.1   So probably it’s either the hardware or the software version fixed this kind of problem. ... View more

That actually makes sense so thank you for that. However, I also did all VRF setup in the lab. Here's mine. All tunnels in both R1 and R2 routers are under VRF1 and the tunnel end-points are in global. I actually tried to do wireshark capture on the remote end to see how the packets are structured and yes they are stacked in a weird way. In my lab below where I used 4451 and IOS 16, even if I don't have a route back to loopback 2001 of R1 in R2 VRF1 RIB, it can normally respond to keepalive packets. R1 interface Tunnel10000 ip vrf forwarding vrf1 ip address 10.200.12.9 255.255.255.252 ip ospf 1 area 0 keepalive 5 2 tunnel source Loopback2001 tunnel destination 2.0.0.2 interface Tunnel10001 ip vrf forwarding vrf1 ip address 10.200.12.13 255.255.255.252 ip ospf 1 area 0 keepalive 5 2 tunnel source Loopback2001 tunnel destination 3.0.0.3 interface Loopback2001 ip address 2.0.0.1 255.255.255.255 R2 interface Tunnel10000 ip vrf forwarding vrf1 ip address 10.200.12.10 255.255.255.252 ip ospf 1 area 0 tunnel source Loopback2002 tunnel destination 2.0.0.1 interface Tunnel10001 ip vrf forwarding vrf1 ip address 10.200.12.14 255.255.255.252 ip ospf 1 area 0 tunnel source Loopback3003 tunnel destination 2.0.0.1 interface Loopback2002 ip address 2.0.0.2 255.255.255.255 interface Loopback3003 ip address 3.0.0.3 255.255.255.255 R2#sh ip route vrf vrf1 | b Gate Gateway of last resort is not set 10.0.0.0/8 is variably subnetted, 4 subnets, 2 masks C 10.200.12.8/30 is directly connected, Tunnel10000 L 10.200.12.10/32 is directly connected, Tunnel10000 C 10.200.12.12/30 is directly connected, Tunnel10001 L 10.200.12.14/32 is directly connected, Tunnel10001 *Jul 11 15:39:13.679: FIBipv4-packet-proc: route packet from (local) src 2.0.0.2 dst 2.0.0.1 *Jul 11 15:39:13.679: FIBfwd-proc: packet routed by adj to GigabitEthernet0/0/1.2002 10.200.12.6 *Jul 11 15:39:13.679: FIBipv4-packet-proc: packet routing succeeded *Jul 11 15:39:23.683: FIBipv4-packet-proc: route packet from (local) src 3.0.0.3 dst 2.0.0.1 *Jul 11 15:39:23.683: FIBfwd-proc: packet routed by adj to GigabitEthernet0/0/1.2002 10.200.12.6 *Jul 11 15:39:23.683: FIBipv4-packet-proc: packet routing succeeded *Jul 11 15:39:23.683: FIBipv4-packet-proc: route packet from (local) src 2.0.0.2 dst 2.0.0.1 *Jul 11 15:39:23.683: FIBfwd-proc: packet routed by adj to GigabitEthernet0/0/1.2002 10.200.12.6 *Jul 11 15:39:23.683: FIBipv4-packet-proc: packet routing succeeded *Jul 11 15:39:33.683: FIBipv4-packet-proc: route packet from (local) src 3.0.0.3 dst 2.0.0.1 *Jul 11 15:39:33.683: FIBfwd-proc: packet routed by adj to GigabitEthernet0/0/1.2002 10.200.12.6 *Jul 11 15:39:33.683: FIBipv4-packet-proc: packet routing succeeded *Jul 11 15:39:33.683: FIBipv4-packet-proc: route packet from (local) src 2.0.0.2 dst 2.0.0.1 *Jul 11 15:39:33.683: FIBfwd-proc: packet routed by adj to GigabitEthernet0/0/1.2002 10.200.12.6 *Jul 11 15:39:33.683: FIBipv4-packet-proc: packet routing succeeded ... View more

Only the tunnels are under VRF. The loopbacks are all in default RIB. ... View more

Using the same lab setup, I changed everything to VRF and everything worked just fine. Our US routers are not running VRF too. Just plain global routing. For some reason, not one tunnel came up when keepalives were configured. In APAC where we used VRF, one out of the two came up with keepalives. Not sure if this is a hardware limitation or an IOS bug. ... View more

The timers are different story. That one was configured for routing purposes. However, it's not sending alarms to our NMS server that's why we need to see the tunnel go down, that's why we are resorting to keepalives. It has happened couple of times where in there was a problem with the tunnel path but we didn't know it because the interface stayed up. It's quite strange that it worked on some interfaces and some not although end to end reachability between the source and destination interfaces are good. ... View more

The production network is actually not working yet. The lab environment is running 4400 on version 16s while the production is 3900 on 15s. ... View more

The first config I pasted are from my lab so just ignore that. The second configs are all from production. ... View more

Hi there. I was able to make the configurations work. I didn't realize that it was already working. However, that's just my lab running on 4400 routers. The production routers are 3900 and it's running version 15. Here's the configuration. If I put a keeplive configuration on Tunnel 5 of APAC_R1, it will go down. However, Tunnel1 is ok. Both APAC_R1 and APAC_R2 also have GRE tunnels going to US_R1 (not shown). APAC router: APAC_R1: interface Tunnel1 desc Tunnel to US bandwidth 20000 ip vrf forwarding vrf1 ip address 10.20.20.53 255.255.255.252 ip tcp adjust-mss 1436 ip ospf dead-interval 6 ip ospf hello-interval 2 load-interval 30 keepalive 5 3 tunnel source Loopback0 tunnel destination 10.193.142.112 service-policy output EF-GRE-MARKING interface Tunnel5 description Tunnel to APAC_R2 bandwidth 50000 ip vrf forwarding vrf1 ip address 10.20.20.57 255.255.255.252 ip tcp adjust-mss 1436 ip ospf dead-interval 6 ip ospf hello-interval 2 load-interval 30 tunnel source Loopback0 tunnel destination 10.19.255.111 service-policy output EF-GRE-MARKING interface Loopback0 ip address 10.19.255.118 255.255.255.255 APAC_R2: interface Tunnel5 bandwidth 50000 ip vrf forwarding vrf1 ip address 10.20.20.58 255.255.255.252 ip tcp adjust-mss 1436 ip ospf dead-interval 6 ip ospf hello-interval 2 load-interval 30 tunnel source Loopback0 tunnel destination 10.19.255.118 service-policy output EF-GRE-MARKING interface Loopback0 ip address 10.19.255.111 255.255.255.255 ... View more

Here are the configs. There's full routing as the loopbacks are all advertised through EIGRP. R1 interface Tunnel10000 ip address 10.200.12.9 255.255.255.252 keepalive 5 2 tunnel source Loopback2001 tunnel destination 2.0.0.2 interface Tunnel10001 ip address 10.200.12.13 255.255.255.252 keepalive 5 2 tunnel source Loopback2001 tunnel destination 3.0.0.3 interface Loopback2001 ip address 2.0.0.1 255.255.255.255 R2 ip address 10.200.12.10 255.255.255.252 tunnel source Loopback2002 tunnel destination 2.0.0.1 interface Tunnel10001 ip address 10.200.12.14 255.255.255.252 tunnel source Loopback3003 tunnel destination 2.0.0.1 interface Loopback2002 ip address 2.0.0.2 255.255.255.255 interface Loopback3003 ip address 3.0.0.3 255.255.255.255 This is just a regular GRE tunnel without IPSEC. Tunnel 10000 is ok, but tunnel 10001 is not. ... View more