GRE tunnel line protocol down on one side only

[jbittman]

I have a strange problem with a GRE tunnel. The tunnel is up and line protocol is up at Site #1. But Site #2 is up/down. How is that possible?

I am using IPSEC encrypted GRE tunnels so I can run OSPF across a few remote offices. There is a backup dialer interface. When the primary tunnel goes down, the modem dials out on the aux port. To get the IPSEC tunnel to establish, I had to create a policy route-map on the backup tunnel interface to route gre traffic sourced from the async ip to the async interface otherwise the packets come in over the phone line and go out over the ethernet interface.

I ran a debug ip packet verbose and I see the ipsec tunnel establishing properly then I see routing broacasts from the Site #1 tunnel ip address on the tunnel interface. It's seems like the tunnel is actually up but for some reason the line protocol isn't properly coming up so OSPF is ignoring the multicast traffic coming in.

[Georg Pauwen]

Hello,

I had a similar problem lately where the tunnel would be up/up on one side and up/down on the other. I solved this by configuring keepalives on the tunnel interfaces, this feature is available starting with IOS 12.2(8)T. Check the link below for details.

Can you post the configs of both sides ?

Generic Routing Encapsulation (GRE) Tunnel Keepalive

http://www.cisco.com/en/US/partner/products/sw/iosswrel/ps1839/products_feature_guide09186a0080087cec.html

Regards,

GP

[Kevin Dorrell]

I see that the default for a tunnel interface is 10 seconds. But I also see that it was introduced in 12.2(8)T. Does that mean that before 12.2(8)T there were no keepalives on a GRE? In which case, if you run post-12.2(8)T against a pre-12.2(8)T you will get the symptoms described by default. Is that right?

Kevin Dorrell

Luxembourg

[Richard Burts]

Kevin

Yes before 12.2(8)T there was no GRE tunnel keepalive.

The default mechanism for GRE tunnels is that if the router has a viable route in its routing table to the tunnel destination, the router will consider the tunnel to be up up. This can lead to the situation described in the original post. It is sometimes misleading. Cisco recently added code for GRE tunnels to provide a keepalive function that does work. If the routers on both ends of the tunnel support the function you can implement GRE keepalives. You certainly would not want to implement it if one of the routers supported the feature and one did not. (It is never a good idea to have keepalives on one end but not on the other.)

The original post mentioned analyzing packets on the tunnel interface. But I find it not clear on which router the capture was taking place and what direction the packets were going. My expectation is that if A showed up/up and B showed up/down I would expect A to send OSPF updates and B to receive them. I would expect that B would not send updates and therefore A would not receive them. Due to the one way nature of the exchange they would never form a complete OSPF neighbor relationship and would not exchange any routing information.

HTH

Rick

[jbittman]

The packet capture was at Site #2. You are correct. Routing information is not being exchanged. The ipsec tunnel seems to be passing traffic through tunnel but the GRE tunnel is not totally up at Site #2.

So I need a viable route? Are you saying all I need is to add a route for the subnet on the tunnel interface towards the backup interface or do I just need to implement tunnel keep alives?

[Kevin Dorrell]

If I understand correctly, (correct me if I'm wrong please Rick) each end of the tunnel needs a route to the ip address of the other end of the tunnel, otherwise it cannot properly form the tunnel in the first place. I don't think it would work if only one end knows how to reach the other.

Once you have the routes on each side, the keepalives are the icing on the cake. But if you have them, they must be enabled at both ends. Which is not an option if either end is pre-12.2(8)T. What version do you have on site #2?

Kevin Dorrell

Luxembourg

[Kevin Dorrell]

Thanks for the useful information Rick. I suppose OSPF treats a GRE tunnel as a point-to-point link, i.e. IP 89 multicasts to 224.0.0.5, which is what he is seeing at the remote site. Or is it treated as a broadcast network with election of a DR?

Kevin Dorrell

Luxembourg

[Richard Burts]

trying to answer two questions in one post:

Jobe:

Yes I think you need to add a route (or verify if there should be a route already and there is not, why not).

I find that an important step in troubleshooting GRE tunnels is to do an extended ping. Specify the ping destination as the tunnel destination address and specify the ping source as the tunnel source address. If the extended ping works then the tunnel should work. And if the ping does not work then the tunnel will not work and you need to solve an IP connectivity problem.

Once you have correct connecticity and the tunnel is working then tunnel keepalives will monitor the tunnel and let you know if there is a problem. Also consider that tunnel keepalive does not make anything work, it verifies that something is working or is not.

Kevin:

Yes OSPF treats a GRE tunnel as a point to point link and there is no election of a DR/BDR.

HTH

Rick

[jbittman]

Unforunately I can only test after hours. Last night, I looked at the tunnels and keep alives were already set to "keepalive 2 3" at both ends. I tried setting no keepalives and the tunnels came up. But no traffic would pass. I tried adding keepalives back in at default and the tunnel went down on one side. Then I set them back to original and both sides went down.... So it's not consistent. I attached my config for Site #2.

I think I must have a missing route but not sure what. Here is a quick explaination. Primary tunnel works and passes traffic. I manually shutdown the tunnel0 at Site #1. The dialer watch list sees the route go away and fails over to the Async port and tunnel10(and switches back if I reenable tunnel.) Before I added the policy map I was getting debug messages that the tunnel was sending out the ipsec packets from async5 out the default route rather than out the async interface. So then I tried adding a static route to 1.1.1.1 over the async port. Tunnel10 opened and worked great but obviously when tunnel0 could never come back up but I was able to get tunel10 up and working properly. So Then I decied to add a route map just to tunnel10 to route packets from 3.3.3.3 to 1.1.1.1 to async port. Now it seems the IPSEC part is working when I show crypto peers stats, but tunnel10 will not establish an ospf neighbor relationship. Any ideas what I"m missing here?

[Richard Burts]

Jobe

As I said in my previous post, the best test that you can do for this situation is an extended ping. Do this on both routers: extended ping which specifies the ping target as the tunnel destination and specifies the ping source as the tunnel source.

As a supplement to this test it would be helpful to do a show ip route and make sure that you have a route to the destination and verify that the outbound interface makes sense.

If you find that you do have connectivity problems it may be helpful to do an extended trace which specifies destination as the tunnel destination and specifies source as the tunnel source. This will show the path through the network and may help find the place where things break down.

I do not know why these tests would need to wait till after hours. Obviously trying to fix anything might have to wait. But the tests I am describing are not disruptive.

I have looked at the config that you sent and have not noticed a problem there, but there is quite a bit that we can not see from the config listing, especially things like the static default route to a next hop address, but no way to know if that address (66.126.214.137) is reachable.

Perhaps you can post the output of the extended pings and the show ip routes.

HTH

Rick

[jbittman]

Ok. I see an issue but I'm not sure what it is. I performed and extended ping and trace from Site#1 and turned on icmp and packet debugging. The packet gets through the tunnel to Site#2. Site #2 sends a reply but it seems to disappear at that point. The debugs are mixed together below. So the traffic

23:50:49: IP: s=1.1.1.1 (Async5), d=3.3.3.3 (Async5), len 128, rcvd 3

23:50:49: IP: s=172.16.0.13 (Tunnel10), d=172.16.0.14 (Tunnel10), len 100, rcvd 3

23:50:49: ICMP: echo reply sent, src 172.16.0.14, dst 172.16.0.13

23:50:49: IP: s=172.16.0.14 (local), d=172.16.0.13 (Tunnel10), len 100, sending

23:50:50: IP: s=3.3.3.3 (Tunnel10), d=1.1.1.1 (Async5), g=1.1.1.1, len 28, forward

23:50:50: IP: s=3.3.3.3 (Tunnel10), d=1.1.1.1 (Async5), g=1.1.1.1, len 80, forward

[Kevin Dorrell]

That must mean that Site #2 has no valid route to the IP address of the interface hosting the tunnel at site #1. At site #2, look at the config of Tunnel10, look for the tunnel destination a.b.c.d, then do a show ip route a.b.c.d. Do you see a route?

Also, check that the target ip addresses are matched correctly at each end of the tunnel. Not the tunnel addresses themselves, which we know are 172.16.0.13 and 172.16.0.14, but the tunnel destinations.

Kevin Dorrell

Luxembourg