Routing table/OSPF issue

[amar_5664]

My core routing table is re-converging every 10 seconds, Area 0 SPF calculations are done every few seconds.

    Area BACKBONE(0)
        Number of interfaces in this area is 8 (1 loopback)
        Area has no authentication
        SPF algorithm last executed 00:00:00.200 ago
        SPF algorithm executed 6638 times
        Area ranges are

O IA    10.163.0.0/24 [110/2] via 10.131.10.3, 00:00:07, Vlan510
                      [110/2] via 10.131.10.2, 00:00:07, Vlan510
O IA    10.160.160.0/24 [110/50] via 10.131.10.3, 00:00:07, Vlan510
                        [110/50] via 10.131.10.2, 00:00:08, Vlan510
O IA    10.160.64.0/24 [110/50] via 10.131.10.3, 00:00:08, Vlan510
                       [110/50] via 10.131.10.2, 00:00:08, Vlan510
O IA    10.160.32.0/24 [110/26] via 10.131.10.3, 00:00:08, Vlan510
                       [110/26] via 10.131.10.2, 00:00:08, Vlan510
O IA    10.160.0.0/24 [110/50] via 10.131.10.3, 00:00:08, Vlan510
                      [110/50] via 10.131.10.2, 00:00:08, Vlan510
O IA    10.224.4.0/24 [110/50] via 10.131.8.1, 00:00:08, Vlan508
O IA    10.164.0.0/16 [110/5] via 10.131.4.2, 00:00:08, Vlan504

every ten seconds the routes are converging any idea on what could be the cause.....

[Richard Burts]

It would appear that something in your OSPF network is flapping. Would you post the output of show ip ospf? Perhaps that would help us find the cause of this.

HTH

Rick

[amar_5664]

I checked that multiple times on all of my routers, the routes that are playing up are stable on other Area 0 routers from few weeks. Below is the capture from one of the Core routers

Routing Process "ospf 100" with ID 10.131.251.101
Start time: 00:00:56.736, Time elapsed: 1d00h
Supports only single TOS(TOS0) routes
Supports opaque LSA
Supports Link-local Signaling (LLS)
Supports area transit capability
It is an area border router
Router is not originating router-LSAs with maximum metric
Initial SPF schedule delay 5000 msecs
Minimum hold time between two consecutive SPFs 10000 msecs
Maximum wait time between two consecutive SPFs 10000 msecs
Incremental-SPF disabled
Minimum LSA interval 5 secs
Minimum LSA arrival 1000 msecs
LSA group pacing timer 240 secs
Interface flood pacing timer 33 msecs
Retransmission pacing timer 66 msecs
Number of external LSA 49. Checksum Sum 0x313AAD
Number of opaque AS LSA 0. Checksum Sum 0x000000
Number of DCbitless external and opaque AS LSA 0
Number of DoNotAge external and opaque AS LSA 0
Number of areas in this router is 6. 6 normal 0 stub 0 nssa
Number of areas transit capable is 0
External flood list length 0
IETF NSF helper support enabled
Cisco NSF helper support enabled
Reference bandwidth unit is 100 mbps
    Area BACKBONE(0)
        Number of interfaces in this area is 8 (1 loopback)
        Area has no authentication
        SPF algorithm last executed 00:00:01.888 ago
        SPF algorithm executed 7913 times
        Area ranges are
        Number of LSA 1083. Checksum Sum 0x42791A0
        Number of opaque link LSA 0. Checksum Sum 0x000000
        Number of DCbitless LSA 0
        Number of indication LSA 0
        Number of DoNotAge LSA 0
        Flood list length 0
    Area 42
        Number of interfaces in this area is 1
        Area has no authentication
        SPF algorithm last executed 20:49:12.577 ago
        SPF algorithm executed 7 times
        Area ranges are
        Number of LSA 1229. Checksum Sum 0x3B39CBE
        Number of opaque link LSA 0. Checksum Sum 0x000000
        Number of DCbitless LSA 0
        Number of indication LSA 0
        Number of DoNotAge LSA 0
        Flood list length 1
    Area 134
        Number of interfaces in this area is 1
        Area has no authentication
        SPF algorithm last executed 20:49:22.589 ago
        SPF algorithm executed 6 times
        Area ranges are
           10.134.0.0/16 Active(1) Advertise
        Number of LSA 1281. Checksum Sum 0x3DD7CB6
        Number of opaque link LSA 0. Checksum Sum 0x000000
        Number of DCbitless LSA 0
        Number of indication LSA 0
        Number of DoNotAge LSA 0
        Flood list length 1
    Area 137
        Number of interfaces in this area is 2
        Area has no authentication
        SPF algorithm last executed 20:48:52.221 ago
        SPF algorithm executed 24 times
        Area ranges are
           10.137.0.0/16 Active(1) Advertise
        Number of LSA 1290. Checksum Sum 0x3E3FC26
        Number of opaque link LSA 0. Checksum Sum 0x000000
        Number of DCbitless LSA 0
        Number of indication LSA 0
        Number of DoNotAge LSA 0
        Flood list length 1
    Area 142
        Number of interfaces in this area is 1
        Area has no authentication
        SPF algorithm last executed 20:49:12.469 ago
        SPF algorithm executed 7 times
        Area ranges are
        Number of LSA 1817. Checksum Sum 0x600CE8B
        Number of opaque link LSA 0. Checksum Sum 0x000000
        Number of DCbitless LSA 0
        Number of indication LSA 0
        Number of DoNotAge LSA 0
        Flood list length 1
    Area 153
        Number of interfaces in this area is 1
        Area has no authentication
        SPF algorithm last executed 20:49:22.589 ago
        SPF algorithm executed 7 times
        Area ranges are
           10.153.0.0/16 Active(1) Advertise
        Number of LSA 1277. Checksum Sum 0x3D994A1
        Number of opaque link LSA 0. Checksum Sum 0x000000
        Number of DCbitless LSA 0
        Number of indication LSA 0
        Number of DoNotAge LSA 0
        Flood list length 1

[Richard Burts]

If the SPF runs that frequently something is not stable. If the other areas do not report the same amount of SPF calculation then it would seem that whatever is not stable is in the area 0.

Can you post the output of show ip ospf spf?

HTH

Rick

[amar_5664]

Thanks for that, i just ran a debug

Please let me know if require anything else, i tried almost everything in my bag.

Jul  2 16:31:52.084 AEST: OSPF: Detect change in LSA type 3, LSID 10.131.255.228
, from 10.131.251.103 area 142
Jul  2 16:31:52.084 AEST: OSPF: Schedule partial SPF - type 3 id 10.131.255.228
adv rtr 10.131.251.103
Jul  2 16:31:52.084 AEST: OSPF: Detect change in LSA type 3, LSID 10.131.255.228
, from 10.131.251.103 area 137
Jul  2 16:31:52.084 AEST: OSPF: Schedule partial SPF - type 3 id 10.131.255.228
adv rtr 10.131.251.103
Jul  2 16:31:52.084 AEST: OSPF: Service partial SPF 2/0/0
Jul  2 16:31:52.084 AEST: OSPF process partial spfQ entry
Jul  2 16:31:52.088 AEST: OSPF process partial spfQ LSA id 10.131.255.228: mask
255.255.255.252, type 3 adv_rtr 10.131.251.103, age 2, seq 0x80001E49 (Area 142)
Jul  2 16:31:52.088 AEST: OSPF process summary partial ABR 0x1 txit 0x0 LSA 10.1
31.255.228: mask 255.255.255.252, t3 adv 10.131.251.103, age 2, seq 0x80001E49 (
Area 142)
Jul  2 16:31:52.088 AEST: OSPF: Start partial processing Summary LSA 10.131.255.
228, mask 255.255.255.252, adv 10.131.251.101, age 2, seq 0x800020B0 (Area 142)
type 3
Jul  2 16:31:52.088 AEST: OSPF: Non-backbone/self-originated LSA
Jul  2 16:31:52.088 AEST: OSPF: delete lsa id 10.131.255.228, type 3, adv rtr 10
.131.251.101 from delete list
Jul  2 16:31:52.088 AEST: OSPF: lsa id 10.131.255.228 not on delete list
Jul  2 16:31:52.088 AEST: OSPF: Start partial processing Summary LSA 10.131.255.
228, mask 255.255.255.252, adv 10.131.251.103, age 2, seq 0x80001E49 (Area 142)
type 3
Jul  2 16:31:52.092 AEST: OSPF: Non-backbone/self-originated LSA
Jul  2 16:31:52.092 AEST: OSPF: delete lsa id 10.131.255.228, type 3, adv rtr 10
.131.251.103 from delete list
Jul  2 16:31:52.092 AEST: OSPF: lsa id 10.131.255.228 not on delete list
Jul  2 16:31:52.092 AEST: OSPF: Start partial processing Summary LSA 10.131.255.
228, mask 255.255.255.252, adv 10.143.251.22, age 4, seq 0x8009992E (Area 142) t
ype 3
Jul  2 16:31:52.092 AEST: OSPF: Non-backbone/self-originated LSA
Jul  2 16:31:52.092 AEST: OSPF: delete lsa id 10.131.255.228, type 3, adv rtr 10
.143.251.22 from delete list
Jul  2 16:31:52.092 AEST: OSPF: lsa id 10.131.255.228 not on delete list
Jul  2 16:31:52.092 AEST: OSPF process partial spfQ LSA id 10.131.255.228: mask
255.255.255.252, type 3 adv_rtr 10.131.251.103, age 2, seq 0x80001E49 (Area 137)
Jul  2 16:31:52.092 AEST: OSPF process summary partial ABR 0x1 txit 0x0 LSA 10.1
31.255.228: mask 255.255.255.252, t3 adv 10.131.251.103, age 2, seq 0x80001E49 (
Area 137)
Jul  2 16:31:52.096 AEST: OSPF: Start partial processing Summary LSA 10.131.255.
228, mask 255.255.255.252, adv 10.131.251.101, age 3, seq 0x800020B0 (Area 137)
type 3
Jul  2 16:31:52.096 AEST: OSPF: Non-backbone/self-originated LSA
Jul  2 16:31:52.096 AEST: OSPF: delete lsa id 10.131.255.228, type 3, adv rtr 10
.131.251.101 from delete list
Jul  2 16:31:52.096 AEST: OSPF: lsa id 10.131.255.228 not on delete list
Jul  2 16:31:52.096 AEST: OSPF: Start partial processing Summary LSA 10.131.255.
228, mask 255.255.255.252, adv 10.131.251.103, age 2, seq 0x80001E49 (Area 137)
type 3
Jul  2 16:31:52.096 AEST: OSPF: Non-backbone/self-originated LSA
Jul  2 16:31:52.096 AEST: OSPF: delete lsa id 10.131.255.228, type 3, adv rtr 10
.131.251.103 from delete list
Jul  2 16:31:52.096 AEST: OSPF: lsa id 10.131.255.228 not on delete list
Jul  2 16:31:52.096 AEST: OSPF process partial spfQ entry
Jul  2 16:31:52.096 AEST: OSPF process partial spfQ entry
Jul  2 16:31:52.100 AEST: OSPF: Detect change in LSA type 3, LSID 10.131.255.228
, from 10.131.251.103 area 134
Jul  2 16:31:52.100 AEST: OSPF: Schedule partial SPF - type 3 id 10.131.255.228
adv rtr 10.131.251.103
Jul  2 16:31:52.100 AEST: OSPF: Detect change in LSA type 3, LSID 10.131.255.228
, from 10.131.251.103 area 153
Jul  2 16:31:52.100 AEST: OSPF: Schedule partial SPF - type 3 id 10.131.255.228
adv rtr 10.131.251.103
Jul  2 16:31:52.100 AEST: OSPF: Detect change in LSA type 3, LSID 10.131.255.228
, from 10.131.251.103 area 42
Jul  2 16:31:52.100 AEST: OSPF: Schedule partial SPF - type 3 id 10.131.255.228
adv rtr 10.131.251.103
Jul  2 16:31:52.104 AEST: OSPF: Service partial SPF 3/0/0
Jul  2 16:31:52.104 AEST: OSPF process partial spfQ entry
Jul  2 16:31:52.104 AEST: OSPF process partial spfQ LSA id 10.131.255.228: mask
255.255.255.252, type 3 adv_rtr 10.131.251.103, age 3, seq 0x80001E49 (Area 134)
Jul  2 16:31:52.104 AEST: OSPF process summary partial ABR 0x1 txit 0x0 LSA 10.1
31.255.228: mask 255.255.255.252, t3 adv 10.131.251.103, age 3, seq 0x80001E49 (
Area 134)
Jul  2 16:31:52.104 AEST: OSPF: Start partial processing Summary LSA 10.131.255.
228, mask 255.255.255.252, adv 10.131.251.101, age 3, seq 0x800020B0 (Area 134)
type 3
Jul  2 16:31:52.104 AEST: OSPF: Non-backbone/self-originated LSA
Jul  2 16:31:52.104 AEST: OSPF: delete lsa id 10.131.255.228, type 3, adv rtr 10
.131.251.101 from delete list
Jul  2 16:31:52.104 AEST: OSPF: lsa id 10.131.255.228 not on delete list
Jul  2 16:31:52.104 AEST: OSPF: Start partial processing Summary LSA 10.131.255.
228, mask 255.255.255.252, adv 10.131.251.103, age 3, seq 0x80001E49 (Area 134)
type 3
Jul  2 16:31:52.108 AEST: OSPF: Non-backbone/self-originated LSA
Jul  2 16:31:52.108 AEST: OSPF: delete lsa id 10.131.255.228, type 3, adv rtr 10
.131.251.103 from delete list
Jul  2 16:31:52.108 AEST: OSPF: lsa id 10.131.255.228 not on delete list
Jul  2 16:31:52.108 AEST: OSPF process partial spfQ LSA id 10.131.255.228: mask
255.255.255.252, type 3 adv_rtr 10.131.251.103, age 2, seq 0x80001E49 (Area 153)
Jul  2 16:31:52.108 AEST: OSPF process summary partial ABR 0x1 txit 0x0 LSA 10.1
31.255.228: mask 255.255.255.252, t3 adv 10.131.251.103, age 2, seq 0x80001E49 (
Area 153)
Jul  2 16:31:52.108 AEST: OSPF: Start partial processing Summary LSA 10.131.255.
228, mask 255.255.255.252, adv 10.131.251.101, age 3, seq 0x800020AE (Area 153)
type 3
Jul  2 16:31:52.108 AEST: OSPF: Non-backbone/self-originated LSA
Jul  2 16:31:52.108 AEST: OSPF: delete lsa id 10.131.255.228, type 3, adv rtr 10
.131.251.101 from delete list
Jul  2 16:31:52.108 AEST: OSPF: lsa id 10.131.255.228 not on delete list
Jul  2 16:31:52.112 AEST: OSPF: Start partial processing Summary LSA 10.131.255.
228, mask 255.255.255.252, adv 10.131.251.103, age 2, seq 0x80001E49 (Area 153)
type 3
Jul  2 16:31:52.112 AEST: OSPF: Non-backbone/self-originated LSA
Jul  2 16:31:52.112 AEST: OSPF: delete lsa id 10.131.255.228, type 3, adv rtr 10
.131.251.103 from delete list
Jul  2 16:31:52.112 AEST: OSPF: lsa id 10.131.255.228 not on delete list
Jul  2 16:31:52.112 AEST: OSPF process partial spfQ LSA id 10.131.255.228: mask
255.255.255.252, type 3 adv_rtr 10.131.251.103, age 2, seq 0x80001E49 (Area 42)
Jul  2 16:31:52.112 AEST: OSPF process summary partial ABR 0x1 txit 0x0 LSA 10.1
31.255.228: mask 255.255.255.252, t3 adv 10.131.251.103, age 2, seq 0x80001E49 (
Area 42)
Jul  2 16:31:52.112 AEST: OSPF: Start partial processing Summary LSA 10.131.255.
228, mask 255.255.255.252, adv 10.131.251.101, age 3, seq 0x800020B0 (Area 42) t
ype 3
Jul  2 16:31:52.112 AEST: OSPF: Non-backbone/self-originated LSA
Jul  2 16:31:52.116 AEST: OSPF: delete lsa id 10.131.255.228, type 3, adv rtr 10
.131.251.101 from delete list
Jul  2 16:31:52.116 AEST: OSPF: lsa id 10.131.255.228 not on delete listu
Jul  2 16:31:52.116 AEST: OSPF: Start partial processing Summary LSA 10.131.255.
228, mask 255.255.255.252, adv 10.131.251.103, age 2, seq 0x80001E49 (Area 42) t
ype 3
Jul  2 16:31:52.116 AEST: OSPF: Non-backbone/self-originated LSA
Jul  2 16:31:52.116 AEST: OSPF: delete lsa id 10.131.255.228, type 3, adv rtr 10
.131.251.103 from delete list
Jul  2 16:31:52.116 AEST: OSPF: lsa id 10.131.255.228 not on delete list
Jul  2 16:31:52.116 AEST: OSPF process partial spfQ entry
Jul  2 16:31:52.116 AEST: OSPF process partial spfQ entry
Jul  2 16:31:52.861 AEST: OSPF: Detect change in LSA type 2, LSID 10.131.255.230
, from 10.131.251.102 area 0n all
Jul  2 16:31:52.861 AEST: OSPF: Detect MAXAGE in LSA type 2, LS ID 10.131.255.23
0, from 10.131.251.102
Jul  2 16:31:52.865 AEST: OSPF: Schedule SPF in area 0
      Change in LS ID 10.131.255.230, LSA type N, , spf-type Full
Jul  2 16:31:54.092 AEST: OSPF: Generate sum from intra-area route 10.131.255.22
8, mask 255.255.255.252, type 3, age 0, metric 4, seq 0x800020B1 to area 42
Jul  2 16:31:54.092 AEST: OSPF: Generate sum from intra-area route 10.131.255.22
8, mask 255.255.255.252, type 3, age 0, metric 4, seq 0x800020B1 to area 134
Jul  2 16:31:54.092 AEST: OSPF: Generate sum from intra-area route 10.131.255.22
8, mask 255.255.255.252, type 3, age 0, metric 4, seq 0x800020B1 to area 137
Jul  2 16:31:54.092 AEST: OSPF: Generate sum from intra-area route 10.131.255.22
8, mask 255.255.255.252, type 3, age 0, metric 4, seq 0x800020B1 to area 142

[Richard Burts · Accepted solution]

There are a couple of things in the debug that surprise me. We have

Detect change in LSA type 3, LSID 10.131.255.228, from 10.131.251.103 area 142

and we have

Detect change in LSA type 3, LSID 10.131.255.228, from 10.131.251.103 area 137

so why is the same router reporting changes in the link for 2 different areas?

There is not a Detect change message but it also appears that this link has had an LSA from another router

Start partial processing Summary LSA 10.131.255.228, mask  255.255.255.252, adv 10.131.251.101

so why are 2 different routers reporting changes in this link.

And just as the SPF is processing there is another change reported

Detect change in LSA type 3, LSID 10.131.255.228, from 10.131.251.103 area 134

so we have the same link and same reporting router but a different area. why is this?

And then another change for the same link, same reporting router and another different area

Detect change in LSA type 3, LSID 10.131.255.228, from 10.131.251.103 area 153

and then

Detect change in LSA type 3, LSID 10.131.255.228, from 10.131.251.103 area 42

The above have caused partial SPF calculation. Then a change comes in that causes a full calculation of SPF

Jul  2 16:31:52.861 AEST: OSPF: Detect change in LSA type 2, LSID 10.131.255.230, from 10.131.251.102 area 0n all
Jul  2 16:31:52.861 AEST: OSPF: Detect MAXAGE in LSA type 2, LS ID 10.131.255.230, from 10.131.251.102
Jul  2 16:31:52.865 AEST: OSPF: Schedule SPF in area 0 Change in LS ID 10.131.255.230, LSA type N, , spf-type Full

Can you find an explanation for what is causing this?

HTH

Rick

[amar_5664]

g'day rick...

thanks champ... i was so dug into the issue and annoyed about it but you pointed me in right direction.... it was that particular route that was being sourced for different destinations ...

geez....this was deployed before my time and had to dig in through all weekend... thanks again... your a legend...

cheers

[Richard Burts]

I am glad that my suggestions pointed you in the right direction to solve your problem. Thanks for marking the question as solved (and thanks for the rating and the kind words). It makes the forum more useful when people can read about a problem and can know from the marking that there was a solution to the problem.

I encourage you to continue your participation in the forum.

HTH

Rick