OSPF Flapping

wasiimcisco · ‎05-10-2011

Hi,

I am having problem in my OSPF network. I have connectivity with IPVPN between My branch and head office using GRE Tunnels.

From branch primary GRE tunnel is 560,

Secondary GRE Tunnel is 570

Disaster Recovery Side third GRE Tunnel is 660.

My users are complaining of disconnection at the branch side.

While checking the logging and OSPF logs I have noticed that OSPF neighborship is breaking three to four times during the day though the MPLS link is stable. I am not able to see any CRC or interface resets on the interface on the head office and branch side as well.

Also there is no topology change in the network and I am not making any changes in the network or modifying any OSPF related configuration. Attached is the configuration of Branch router and Head office Router. I am getting followng below mention error on the branch side.

show logging
Syslog logging: enabled (12 messages dropped, 1 messages rate-limited,
0 flushes, 0 overruns, xml disabled, filtering disabled)

No Active Message Discriminator.

No Inactive Message Discriminator.

    Console logging: level debugging, 1680 messages logged, xml disabled,
                     filtering disabled
    Monitor logging: level debugging, 2 messages logged, xml disabled,
                     filtering disabled
    Buffer logging: level debugging, 1680 messages logged, xml disabled,
                     filtering disabled
    Logging Exception size (4096 bytes)
    Count and timestamp logging messages: disabled
    Persistent logging: disabled

No active filter modules.

ESM: 0 messages dropped

Trap logging: level critical, 1 message lines logged

Log Buffer (4096 bytes):
to down
001660: *May 9 23:38:56.495: %OSPF-5-ADJCHG: Process 1, Nbr 192.168.0.161 on Tunnel504 from FULL to DOWN, Neighbor Down: Interface down or detached
001661: *May 9 23:39:01.491: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel504, changed state to up
001662: *May 9 23:39:05.023: %OSPF-5-ADJCHG: Process 1, Nbr 192.168.0.161 on Tunnel504 from LOADING to FULL, Loading Done
001663: *May 9 23:59:24.211: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel503, changed state to down
001664: *May 9 23:59:24.211: %OSPF-5-ADJCHG: Process 1, Nbr 192.168.0.161 on Tunnel503 from FULL to DOWN, Neighbor Down: Interface down or detached
001665: *May 9 23:59:29.211: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel503, changed state to up
001666: *May 9 23:59:30.219: %OSPF-5-ADJCHG: Process 1, Nbr 192.168.0.161 on Tunnel503 from LOADING to FULL, Loading Done
001667: *May 9 23:59:41.491: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel504, changed state to down
001668: *May 9 23:59:41.491: %OSPF-5-ADJCHG: Process 1, Nbr 192.168.0.161 on Tunnel504 from FULL to DOWN, Neighbor Down: Interface down or detached
001669: *May 9 23:59:46.491: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel504, changed state to up
001670: *May 9 23:59:56.939: %OSPF-5-ADJCHG: Process 1, Nbr 192.168.0.161 on Tunnel504 from LOADING to FULL, Loading Done
001671: *May 10 00:03:29.891: %OSPF-5-ADJCHG: Process 1, Nbr 192.168.0.161 on Tunnel503 from FULL to DOWN, Neighbor Down: Dead timer expired
001672: *May 10 00:03:49.447: %OSPF-5-ADJCHG: Process 1, Nbr 192.168.0.161 on Tunnel503 from LOADING to FULL, Loading Done
001673: *May 10 00:04:39.211: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel503, changed state to down
001674: *May 10 00:04:39.211: %OSPF-5-ADJCHG: Process 1, Nbr 192.168.0.161 on Tunnel503 from FULL to DOWN, Neighbor Down: Interface down or detached
001675: *May 10 00:04:44.211: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel503, changed state to up
001676: *May 10 00:04:45.087: %OSPF-5-ADJCHG: Process 1, Nbr 192.168.0.161 on Tunnel503 from LOADING to FULL, Loading Done
001677: *May 10 00:10:16.527: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel504, changed state to down
001678: *May 10 00:10:16.527: %OSPF-5-ADJCHG: Process 1, Nbr 192.168.0.161 on Tunnel504 from FULL to DOWN, Neighbor Down: Interface down or detached
001679: *May 10 00:10:21.527: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel504, changed state to up
001680: *May 10 00:10:22.403: %OSPF-5-ADJCHG: Process 1, Nbr 192.168.0.161 on Tunnel504 from LOADING to FULL, Loading Done
001681: *May 10 00:11:14.215: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel503, changed state to down
001682: *May 10 00:11:14.219: %OSPF-5-ADJCHG: Process 1, Nbr 192.168.0.161 on Tunnel503 from FULL to DOWN, Neighbor Down: Interface down or detached
001683: *May 10 00:11:19.215: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel503, changed state to up
001684: *May 10 00:11:22.327: %OSPF-5-ADJCHG: Process 1, Nbr 192.168.0.161 on Tunnel503 from LOADING to FULL, Loading Done
001685: *May 10 04:03:24.334: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel503, changed state to down
001686: *May 10 04:03:24.338: %OSPF-5-ADJCHG: Process 1, Nbr 192.168.0.161 on Tunnel503 from FULL to DOWN, Neighbor Down: Interface down or detached
001687: *May 10 04:03:26.662: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel504, changed state to down
001688: *May 10 04:03:26.662: %OSPF-5-ADJCHG: Process 1, Nbr 192.168.0.161 on Tunnel504 from FULL to DOWN, Neighbor Down: Interface down or detached
001689: *May 10 04:03:31.662: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel504, changed state to up
001690: *May 10 04:03:32.310: %OSPF-5-ADJCHG: Process 1, Nbr 192.168.0.161 on Tunnel504 from LOADING to FULL, Loading Done
001691: *May 10 04:03:34.334: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel503, changed state to up
001692: *May 10 04:03:34.998: %OSPF-5-ADJCHG: Process 1, Nbr 192.168.0.161 on Tunnel503 from LOADING to FULL, Loading Done

Please help me out why this OSPF is flapping while the physical link is stable. There is no topology and configuration change in the netwrok.

Calin C. · ‎05-10-2011

The OSPF flaps because your Tunnel interfaces went down.

This occur when the tunnel source / destination IP are not reachable from either side.

The physical interface (e.g. Cellular interface) may be up, but the Virtual one (Tunnel in this case) went down. That says you had a problem over the provider network (more probably) to reach the opposite end. I think that this is a mobile communication line (e.g. GPRS) and it's not uncommon to have some problems like I explained above.

If you have a strong SLA with provider this can be checked.

Cheers,

Calin

wasiimcisco · ‎05-10-2011

Thanks for the reply, but if you see the interface resets physical interface didnt flap even for one time.

and if you see the below mention another branch router there is also you can see that OSPF alog recalculated though the interface has no resets.

TSJL_756_R01#show logging
Syslog logging: enabled (0 messages dropped, 1 messages rate-limited,
0 flushes, 0 overruns, xml disabled, filtering disabled)

No Active Message Discriminator.

No Inactive Message Discriminator.

    Console logging: level debugging, 54 messages logged, xml disabled,
                     filtering disabled
    Monitor logging: level debugging, 0 messages logged, xml disabled,
                     filtering disabled
    Buffer logging: level warnings, 3 messages logged, xml disabled,
                     filtering disabled
    Logging Exception size (4096 bytes)
    Count and timestamp logging messages: disabled
    Persistent logging: disabled
    Trap logging: level informational, 58 message lines logged

Log Buffer (51200 bytes):

*Apr 30 02:59:54.379: %LINK-3-UPDOWN: Interface FastEthernet0/0, changed state to up
*Apr 30 02:59:54.379: %LINK-3-UPDOWN: Interface FastEthernet0/1, changed state to up
*Apr 30 02:59:54.379: %LINK-3-UPDOWN: Interface Serial0/1/0, changed state to down
TSJL_756_R01#
TSJL_756_R01#
TSJL_756_R01#
TSJL_756_R01#
TSJL_756_R01#show osp
TSJL_756_R01#show ip os
TSJL_756_R01#show ip ospf
Routing Process "ospf 1" with ID 172.27.1.111
Start time: 00:00:37.540, Time elapsed: 1w3d
Supports only single TOS(TOS0) routes
Supports opaque LSA
Supports Link-local Signaling (LLS)
Supports area transit capability
Router is not originating router-LSAs with maximum metric
Initial SPF schedule delay 5000 msecs
Minimum hold time between two consecutive SPFs 10000 msecs
Maximum wait time between two consecutive SPFs 10000 msecs
Incremental-SPF disabled
Minimum LSA interval 5 secs
Minimum LSA arrival 1000 msecs
LSA group pacing timer 240 secs
Interface flood pacing timer 33 msecs
Retransmission pacing timer 66 msecs
Number of external LSA 0. Checksum Sum 0x000000
Number of opaque AS LSA 0. Checksum Sum 0x000000
Number of DCbitless external and opaque AS LSA 0
Number of DoNotAge external and opaque AS LSA 0
Number of areas in this router is 1. 0 normal 1 stub 0 nssa
Number of areas transit capable is 0
External flood list length 0
IETF NSF helper support enabled
Cisco NSF helper support enabled
    Area 6
        Number of interfaces in this area is 6 (1 loopback)
        It is a stub area

Area has no authentication
        SPF algorithm last executed 03:51:04.864 ago
        SPF algorithm executed 4793 times
        Area ranges are
        Number of LSA 25. Checksum Sum 0x0BD1BD
        Number of opaque link LSA 0. Checksum Sum 0x000000
        Number of DCbitless LSA 0
        Number of indication LSA 0
        Number of DoNotAge LSA 0
        Flood list length 0

almost every branch is showing me the recalculation of SPF algorithm last executed 03:51:04.864 ago.

Is there any way to control this because link is stable etc.