GRE tunnel shows down

[Steven Williams]

Anyone have any ideas why the GRE tunnel is not up when I have connectivity to the other side of the tunnel?

show ip int br
Load for five secs: 9%/5%; one minute: 9%; five minutes: 8%
Time source is NTP, 15:05:43.648 CST Thu Feb 6 2014

Interface                  IP-Address      OK? Method Status                Protocol
GigabitEthernet0/0         10.170.199.6    YES NVRAM  up                    up     
GigabitEthernet0/1         10.170.199.14   YES NVRAM  up                    up     
Serial0/0/0:0              unassigned      YES NVRAM  down                  down   
Serial0/0/1:1              unassigned      YES NVRAM  down                  down   
Serial0/1/0:2              unassigned      YES NVRAM  down                  down   
Serial0/1/1:3              unassigned      YES NVRAM  down                  down   
Serial0/3/0:4              unassigned      YES NVRAM  down                  down   
Serial0/3/1:5              unassigned      YES NVRAM  down                  down   
Serial1/0                  unassigned      YES NVRAM  up                    up     
Serial1/0.100              152.162.63.102  YES NVRAM  up                    up     
NVI0                       unassigned      NO  unset  up                    up     
Loopback0                  172.16.14.13    YES NVRAM  up                    up     
Tunnel0                    10.70.50.1      YES manual up                    up     
Tunnel20                   10.70.50.5      YES manual up                    up     
Tunnel30                   10.70.50.9      YES manual up                    up     
Tunnel60                   10.70.50.17     YES manual up                    up     
Tunnel70                   10.70.50.13     YES manual up                    up     
Tunnel80                   10.70.50.21     YES manual up                    up     
Tunnel100                  10.70.50.29     YES manual up                    down   
Tunnel120                  10.70.50.25     YES manual up                    down   
Tunnel150                  10.70.50.33     YES manual up                    down   
R1169063#ping 10.70.50.30

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.70.50.30, timeout is 2 seconds:
...!!
Success rate is 40 percent (2/5), round-trip min/avg/max = 124/124/124 ms
R1169063#ping 10.70.50.26

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.70.50.26, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 124/124/128 ms
R1169063#ping 10.70.50.34

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.70.50.34, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 116/116/116 ms

[Steven Williams]

what causes one end of a tunnel to be up/up and the other to be up/down?

[Richard Burts]

I have experienced the symptom where I could ping the remote address of a tunnel even when the local end of the tunnel was up/down when the remote end was on a spoke router which had two tunnels (one connected to the router I was on and the other connected to another head end router) and was advertising its tunnel subnet over the other tunnel.

I have seen this symptom several times where one end of the tunnel is up/up but the other end of the tunnel is up/down. In each of the cases it turned out to be some bug in IOS about the process to rekey/renegotiate the SA. What platform and what version of code is this happening on?

HTH

Rick

[Richard Burts]

An additional thought is that one good way to verify what is happening is to use traceroute instead of ping. Do traceroute to the address of the  other end of the tunnel. My guess is that it will not show the one hop result that you expect but will show its path goes over your network to an alternate path to the remote site.

HTH

Rick

[prakhar.gour]

Hi Steven,

As GRE is a stateless protocol, by default it does not know if the tunnel from other side is down. For enabling this feature, do enable GRE keepalives.

interface tunnel

keepalive 10 2

That is why you can see one end of a tunnel to be up/up and the other to be up/down.

Thanks

Prakhar Gour

[Steven Williams]

Keepalive is configured. I am on 12.4(13r)T

[Steven Williams]

Whats even more wierd is that on the router that shows up/down, It cannot even ping its own tunnel IP address?

Ok so heres some more.....Tunnel 1000 is the tunnel in question.

Feb  8 13:35:46: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel1000, changed state to up

Feb  8 13:35:55: %TUN-5-RECURDOWN: Tunnel1000 temporarily disabled due to recursive routing

Feb  8 13:35:56: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel1000, changed state to  

Feb  8 13:35:34.319: Tunnel20: sending keepalive, 10.70.20.32->10.170.199.6 (len=24 ttl=255), counter=1

Feb  8 13:35:34.327: Tunnel30: sending keepalive, 10.70.30.32->10.170.199.6 (len=24 ttl=255), counter=1

Feb  8 13:35:34.331: Tunnel40: sending keepalive, 10.70.40.33->10.170.199.6 (len=24 ttl=255), counter=1

Feb  8 13:35:34.331: Tunnel60: sending keepalive, 10.70.60.32->10.170.199.6 (len=24 ttl=255), counter=1

Feb  8 13:35:34.335: Tunnel70: sending keepalive, 10.70.70.32->10.170.199.6 (len=24 ttl=255), counter=1

Feb  8 13:35:34.339: Tunnel80: sending keepalive, 10.70.80.32->10.170.199.6 (len=24 ttl=255), counter=1

Feb  8 13:35:34.343: Tunnel120: sending keepalive, 10.70.120.21->10.170.199.6 (len=24 ttl=255), counter=1

Feb  8 13:35:34.343: Tunnel20: keepalive received, 10.70.20.32->10.170.199.6 (len=24 ttl=250), resetting counter

Feb  8 13:35:34.347: Tunnel150: sending keepalive, 10.70.150.32->10.170.199.6 (len=24 ttl=255), counter=1

Feb  8 13:35:34.355: Tunnel60: keepalive received, 10.70.60.32->10.170.199.6 (len=24 ttl=250), resetting counter

Feb  8 13:35:34.359: Tunnel80: keepalive received, 10.70.80.32->10.170.199.6 (len=24 ttl=250), resetting counter

Feb  8 13:35:34.359: Tunnel70: keepalive received, 10.70.70.32->10.170.199.6 (len=24 ttl=251), resetting counter

Feb  8 13:35:34.375: Tunnel30: keepalive received, 10.70.30.32->10.170.199.6 (len=24 ttl=247), resetting counter

Feb  8 13:35:34.379: Tunnel40: keepalive received, 10.70.40.33->10.170.199.6 (len=24 ttl=249), resetting counter
Feb  8 13:35:46: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel1000, changed state to up

Feb  8 13:35:55: %TUN-5-RECURDOWN: Tunnel1000 temporarily disabled due to recursive routing

Feb  8 13:35:56: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel1000, changed state to   

What is recursive routing? The only Routing Protocol being ran is PE to CE BGP.

[Richard Burts]

OK. several parts to this answer.

in this context recursive routing means that the router believes that its best route to the tunnel end point is through the tunnel. (see my response to your other post). I have seen this error many times when the router is learning a default route via a routing protocol over the tunnel. The router will also have some route (perhaps a static route or perhaps a route learned via some other protocol) to the tunnel destination) which allows the tunnel to come up. Then there is a problem on the router's outbound interface. The route to the tunnel destination gets removed based on the interface issue. There is (temporarily) still the default route via the tunnel but when the router attempts to use the default route for tunnel traffic it generates the recursive error.  The situation that I just described is temporary - and is more a cosmetic thing than a real problem.

Based on some things in your posts I am guessing that the recursive error is due to something else and not the situation that I just described. I wonder if this issue is, in  fact, related to your other recent post. If you do have a static route to the subnet of the remote and the next hop is through the tunnel then it will cause the recursive error. Is that the issue here??

Other parts of the answer. In my previous response to this post I was assuming that you were doing VTI (IPSec over a tunnel) or perhaps GRE with IPSec. I am not wondering whether this is a regular GRE tunnel or whether IPSec is involved. Can you clarify?

Other part - when a router ping its own IP address on point to point interfaces (including GRE tunnels) it will actually send the ping packet over the tunnel to its peer, and the peer will send the ping packet back. So if the loca tunnel interface is up/down then it is logical that the ping to tis own address will fail since it is not able to send the ping packet out.

HTH

Rick