Solved: GRE issue - Page 2

boban-petrovic · ‎12-28-2011

The probelm is following: there is IP connectivity between public IP addresses, x.y.148.202 and x.y.132.202, but not between 10.0.10.1 and 10.0.10.2. Relevant configurations are following:

Router A:

interface Tunnel0

ip address 10.0.10.1 255.255.255.0

tunnel source Vlan1

tunnel destination x.y.132.202

interface Vlan1

ip address x.y.148.202 255.255.255.252

#sh ip int tu0

Tunnel0 is up, line protocol is up

Internet address is 10.0.10.1/24

Broadcast address is 255.255.255.255

Address determined by setup command

MTU is 1476 bytes

Helper address is not set

Directed broadcast forwarding is disabled

Multicast reserved groups joined: 224.0.0.5

Outgoing access list is not set

Inbound access list is not set

Proxy ARP is enabled

Local Proxy ARP is disabled

Security level is default

Split horizon is enabled

ICMP redirects are always sent

ICMP unreachables are always sent

ICMP mask replies are never sent

IP fast switching is enabled

IP fast switching on the same interface is disabled

IP Flow switching is disabled

IP CEF switching is enabled

IP CEF Feature Fast switching turbo vector

IP multicast fast switching is enabled

IP multicast distributed fast switching is disabled

IP route-cache flags are Fast, CEF

Router Discovery is disabled

IP output packet accounting is disabled

IP access violation accounting is disabled

TCP/IP header compression is disabled

RTP/IP header compression is disabled

Policy routing is disabled

Network address translation is disabled

BGP Policy Mapping is disabled

#sh int tu0

Tunnel0 is up, line protocol is up

Hardware is Tunnel

Internet address is 10.0.10.1/24

MTU 1514 bytes, BW 9 Kbit, DLY 500000 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation TUNNEL, loopback not set

Keepalive not set

Tunnel source x.y.148.202 (Vlan1), destination x.y.132.202

Tunnel protocol/transport GRE/IP

Key disabled, sequencing disabled

Checksumming of packets disabled

Tunnel TTL 255

Fast tunneling enabled

Tunnel transmit bandwidth 8000 (kbps)

Tunnel receive bandwidth 8000 (kbps)

Router B:

interface Tunnel0

ip address 10.0.10.2 255.255.255.0

tunnel source GigabitEthernet0/0

tunnel destination x.y.148.202

interface GigabitEthernet0/0

description outside

ip address x.y.132.202 255.255.255.252

ip nat outside

ip virtual-reassembly in

duplex auto

speed auto

#sh ip int tu0

Tunnel0 is up, line protocol is up

Internet address is 10.0.10.2/24

Broadcast address is 255.255.255.255

Address determined by setup command

MTU is 1476 bytes

Helper address is not set

Directed broadcast forwarding is disabled

Outgoing access list is not set

Inbound access list is not set

Proxy ARP is enabled

Local Proxy ARP is disabled

Security level is default

Split horizon is enabled

ICMP redirects are always sent

ICMP unreachables are always sent

ICMP mask replies are never sent

IP fast switching is enabled

IP fast switching on the same interface is disabled

IP Flow switching is disabled

IP CEF switching is enabled

IP CEF switching turbo vector

IP Null turbo vector

IP multicast fast switching is enabled

IP multicast distributed fast switching is disabled

IP route-cache flags are Fast, CEF

Router Discovery is disabled

IP output packet accounting is disabled

IP access violation accounting is disabled

TCP/IP header compression is disabled

RTP/IP header compression is disabled

Policy routing is disabled

Network address translation is disabled

BGP Policy Mapping is disabled

Input features: MCI Check

WCCP Redirect outbound is disabled

WCCP Redirect inbound is disabled

WCCP Redirect exclude is disabled

#sh int tu0

Tunnel0 is up, line protocol is up

Hardware is Tunnel

Internet address is 10.0.10.2/24

MTU 17916 bytes, BW 100 Kbit/sec, DLY 50000 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation TUNNEL, loopback not set

Keepalive not set

Tunnel source x.y.132.202 (GigabitEthernet0/0), destination x.y.148.202

Tunnel Subblocks:

src-track:

Tunnel0 source tracking subblock associated with GigabitEthernet0/0

Set of tunnels with source GigabitEthernet0/0, 1 member (includes iterators), on interface <OK>

Tunnel protocol/transport GRE/IP

Key disabled, sequencing disabled

Checksumming of packets disabled

Tunnel TTL 255, Fast tunneling enabled

Tunnel transport MTU 1476 bytes

Tunnel transmit bandwidth 8000 (kbps)

Tunnel receive bandwidth 8000 (kbps)

I believe that there is some problem with MTUs, but I'm not so sure. How to correct this configuration?

Peter Paluch · ‎12-29-2011

Dan,

Your observation is very correct as there could be problems with recursive routing. However, Boban does not indicate any recursive routing messages in logs, both his Tunnel interfaces are up/up (which they would not be if recursive routing was the issue), and also, the last experiments show a uni-directional OSPF adjacency in the INIT state. Clearly, a recursive routing would not be at the bottom of this problem until the OSPF reached the FULL state.

Certainly, what you have pointed out is very correct and would indeed cause problems after the OSPF came up, however, it is not, in my opinion, the primary reason of the basic tunnel connectivity problems.

Best regards,

Peter

boban-petrovic · ‎12-29-2011

I've changed configuration on router A regarding OSPF:

Router A:

router ospf 10

log-adjacency-changes

redistribute connected subnets route-map control-ospf

network 10.0.10.0 0.0.0.255 area 0

network 192.168.0.0 0.0.255.255 area 0

!

ip access-list standard ospf-allowed

permit 192.168.0.0 0.0.255.255

permit 172.16.0.0 0.0.255.255

!

route-map control-ospf permit 10

match ip address ospf-allowed

And now it works! Both routers have FULL neighbour state. Now, this requires deep analysis, because I agree with Peter about this issue: router B didn't get routes from router A (INIT state), and still problem was there.

Thanks both of you for help!

Message was edited by: Boban Petrovic

Peter Paluch · ‎12-29-2011

Boban,

In this case, I have provided an incorrect suggestion and do not deserve the rating you have gracefully given me. Dan, +5 points for this, and thank you very much indeed for joining this discussion! Your post helped to solve this issue.

I could explain the issue by a simple reference to recursive routing, as that seems to be the most logical course of action, but I still do not find clear evidence that this was the case, because:

Boban did not indicate any message regarding recursive routing, Tunnel interfaces flapping, CEF midchain issues etc. Boban, did you experience any such messages in your logs?
If a recursive routing with tunnel is experienced, the tunnel would be brought down and would not be in up/up state. However, Boban's outputs suggest that the tunnel was up/up.
Recursive routing needs the OSPF to come up fully, and would cause the OSPF neighborship to fail again because of the tunnel interface going down. Again, Boban did not indicate any OSPF flapping.

I will try to replicate this in Dynamips and come up with my observations.

Best regards,

Peter

boban-petrovic · ‎12-29-2011

Peter,

You deserve all points I gave you for all your effort in his discussion. Maybe you didn't notice the eventual cause of the problem, but your posts help me to have better view at overall issue.

As related with your questions, I didn't notice such messages in my logs because I wasn't looking at that direction. Maybe it was there, but I haven't seen it. I concentrated on tunnel encapsulation problem, and I wasn't expecting that routing caused by OSPF may be the problem.

But, this is becoming very inertesting scenario. It is possible that tunnel initialy works correctly upon bringing interfaces up, which would bring routers in FULL ospf neighbour state. But since router A mistakenly publish WAN netwotk route to router B, router B now can't get to tunnel destination IP address. Should this bring tunnel down? If it is so, OSPF neighbour state falls down, and router B again has access to tunnel destination IP and tunnel is going in up state. So OSPF is again in FULL state, and so on over and over again, basicly tunnel is flapping. On which rate should this happening?

If tunnel is not brought down upon establishing FULL ospf neighbour state, router B can't reach router A, so ospf packets is not reaching router A. Router A notice this as router B is down and removes it from neighbour list, but still sends ospf hello packets to tunnel which router B successfuly receives, and keep router A in INIT state. Since router A was in FULL state in roter B's neigbour list, router B keeps routes learned from router A until router A goes to DOWN state which never happens.

All this don't have to be correct, I don't know ospf in details, but it is my way of thinking, and possible steps flow which can get to initial situation and stuck.

Am I right?

Peter Paluch · ‎12-30-2011

Hi Boban,

You are very kind.

The scenario you described - tunnel initially up, the OSPF coming up, causing recursive routing, hence the tunnel going down together with OSPF, thereby resolving the recursive routing issue and causing the tunnel to come back up and repeating this ad infinitum - is perfectly correct and this is how I have seen it happen multiple times with EIGRP, for example.

The frequency of these tunnel and OSPF adjacency flaps would be roughly a minute and half, perhaps slightly less. The reason is that after a recursive routing through tunnel is detected in the routing table, the tunnel will be brought down, along with tearing down the OSPF adjacency and removing the recursive routing entry from the routing table. After 60 seconds (this is the interval of running the routing table scanner), the tunnel will be reinstated, the OSPF will come up after a couple of seconds, and 5 seconds after the OSPF database sync, the SPF will be run and the recursive routing entry will again be added to the routing table. This will again cause the tunnel to go down and the process will repeat. Hence my estimation of the frequency to be somewhere between 60-90 seconds.

From this, however, it follows that most of the time, the tunnel should be in up/down or down/down state. Your output indicated that the tunnels are up/up. While not impossible, it is not that probable. Also, detection of the recursive routing entry, tearing down the tunnel with the OSPF adjacency, reestablishing it etc. would cause copious amounts of logging messages. Did you see any?

I still have to test your config on real IOSes to see what happens. One thing that may cause troubles replicating your issue is the fact that your IOS on the Router B is fairly new, and may have certain differences in behavior in comparison to the 12.4T IOSes I am using the most.

Best regards,

Peter

Dan Frey · ‎12-30-2011

Peter / Boban,

What you have described is what I see in the lab when advertising the tunnel source through the tunnel. The OSPF adj flap on both routers while the recursive routing messages are present on Router B.

If you were connected to a vty line on the router did you omit the "terminal monitor" command? If so this could be the reason for not seeing the log messages on the terminal session. Above you have the "show logging" output but this only gave ~10 seconds worth of debug messages possibly forcing the %syslog_messages out of the buffer.

Here are the log messages on Router B (aka C901 below) that I see in my lab.

*Dec 30 14:37:26.859: %TUN-5-RECURDOWN: Tunnel3 temporarily disabled due to recursive routing

*Dec 30 14:37:27.859: %OSPF-5-ADJCHG: Process 1, Nbr 192.168.3.1 on Tunnel3 from FULL to DOWN, Neighbor Down: Interface down or detached

*Dec 30 14:37:27.867: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel3, changed state to down

C901(config-if)#

*Dec 30 14:38:27.935: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel3, changed state to up

C901(config-if)#

*Dec 30 14:38:29.435: %OSPF-5-ADJCHG: Process 1, Nbr 192.168.3.1 on Tunnel3 from LOADING to FULL, Loading Done

C901(config-if)#

*Dec 30 14:38:46.935: %TUN-5-RECURDOWN: Tunnel3 temporarily disabled due to recursive routing

*Dec 30 14:38:47.935: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel3, changed state to down

*Dec 30 14:38:47.939: %OSPF-5-ADJCHG: Process 1, Nbr 192.168.3.1 on Tunnel3 from FULL to DOWN, Neighbor Down: Interface down or detached

C901(config-if)#

*Dec 30 14:39:48.015: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel3, changed state to up

C901(config-if)#

*Dec 30 14:39:49.463: %OSPF-5-ADJCHG: Process 1, Nbr 192.168.3.1 on Tunnel3 from LOADING to FULL, Loading Done

C901(config-if)#

*Dec 30 14:40:07.023: %TUN-5-RECURDOWN: Tunnel3 temporarily disabled due to recursive routing

*Dec 30 14:40:08.015: %OSPF-5-ADJCHG: Process 1, Nbr 192.168.3.1 on Tunnel3 from FULL to DOWN, Neighbor Down: Interface down or detached

*Dec 30 14:40:08.027: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel3, changed state to down

C901(config-if)#

*Dec 30 14:41:08.063: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel3, changed state to up

C901(config-if)#

*Dec 30 14:41:09.463: %OSPF-5-ADJCHG: Process 1, Nbr 192.168.3.1 on Tunnel3 from LOADING to FULL, Loading Done

C901(config-if)#

*Dec 30 14:41:27.075: %TUN-5-RECURDOWN: Tunnel3 temporarily disabled due to recursive routing

*Dec 30 14:41:28.075: %OSPF-5-ADJCHG: Process 1, Nbr 192.168.3.1 on Tunnel3 from FULL to DOWN, Neighbor Down: Interface down or detached

*Dec 30 14:41:28.083: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel3, changed state to down

C901(config-if)#

*Dec 30 14:42:28.131: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel3, changed state to up

C901(config-if)#

*Dec 30 14:42:29.479: %OSPF-5-ADJCHG: Process 1, Nbr 192.168.3.1 on Tunnel3 from LOADING to FULL, Loading Done

C901(config-if)#

*Dec 30 14:42:47.151: %TUN-5-RECURDOWN: Tunnel3 temporarily disabled due to recursive routing

*Dec 30 14:42:48.143: %OSPF-5-ADJCHG: Process 1, Nbr 192.168.3.1 on Tunnel3 from FULL to DOWN, Neighbor Down: Interface down or detached

*Dec 30 14:42:48.151: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel3, changed state to down

C901(config-if)#

- Dan

boban-petrovic · ‎12-30-2011

Router A is in production for a long time. A few days ago, I decided to add WAN link. Before that, there was no WAN network connected to it and no static route 0.0.0.0/0. That's why ospf worked well with initial configuration. My mistake is that I completely forgot to adjust ospf config before adding WAN link to it.

Router B is new router on other site, we got it a few days ago. It's brand new and I was only one configuring it. All settings regarding logging was on default values. Only change was turning on debug for tunnel and icmp, and logging buffered debuging. Other values by default. There was no log messages on console, I turned terminal monitor on, today, so I believe terminal moniter is omited by default. Anyway, during testing, I haven't seen log messages regarding recursive routing and bringing tunnel and ospf ajdacency down, with show logging command.

And what is problematic here is what peter noted: as Dan's test showed, tunnel should be in down state 75% of the time. In my scenario, tunnel was up/up 100% of the time! And that is what don't fit...