Problem with OSPF

[Niklas.D]

Hi There

So i am running OSPF on my WAN, and have to uplink to my DC and each uplink has a Cisco 4500-X and my remote sites have a Checkpoint Firewall

After a change vs the ISP where we upgraded from a-1000 to a-10G i have a issue that my layer 3 is going from Full to down on my Primary site 

Primay Site

interface Vlan911
description Kristineberg-Ip-Only
ip address 10.240.128.34 255.255.255.240
ip ospf authentication-key 7 03125A181F0B2E5F5E
ip ospf cost 10

Secondary Site

interface Vlan911
description Kristineberg-Ip-Only
ip address 10.240.128.35 255.255.255.240
ip ospf authentication-key 7 03125A181F0B2E5F5E
ip ospf cost 15

Jan 20 09:35:00.798: %OSPF-5-ADJCHG: Process 1, Nbr 10.240.240.5 on Vlan924 from FULL to DOWN, Neighbor Down: Interface down or detached
Jan 20 09:35:31.857: %OSPF-5-ADJCHG: Process 1, Nbr 10.128.2.1 on Vlan911 from LOADING to FULL, Loading Done
Jan 20 09:35:32.952: %OSPF-5-ADJCHG: Process 1, Nbr 10.240.240.5 on Vlan911 from LOADING to FULL, Loading Done
Jan 20 09:35:39.437: %OSPF-5-ADJCHG: Process 1, Nbr 10.128.2.1 on Vlan911 from FULL to DOWN, Neighbor Down: Interface down or detached
Jan 20 09:35:39.437: %OSPF-5-ADJCHG: Process 1, Nbr 10.240.240.5 on Vlan911 from FULL to DOWN, Neighbor Down: Interface down or detached
Jan 20 09:37:11.192: %OSPF-5-ADJCHG: Process 1, Nbr 10.240.240.5 on Vlan911 from LOADING to FULL, Loading Done
Jan 20 09:37:11.859: %OSPF-5-ADJCHG: Process 1, Nbr 10.128.2.1 on Vlan911 from LOADING to FULL, Loading Done

This is effecting all my vlan interfaces. 

The Link is allways working it is just 1 or 2 vlan that will go down and start working. 

Jan 20 10:35:59.557: %OSPF-5-ADJCHG: Process 1, Nbr 10.240.240.5 on Vlan924 from LOADING to FULL, Loading Done
Jan 20 10:36:15.521: %OSPF-5-ADJCHG: Process 1, Nbr 10.128.8.1 on Vlan919 from LOADING to FULL, Loading Done
Jan 20 10:36:18.409: %OSPF-5-ADJCHG: Process 1, Nbr 10.3.2.1 on Vlan924 from LOADING to FULL, Loading Done
Jan 20 10:36:28.717: %OSPF-5-ADJCHG: Process 1, Nbr 10.3.2.1 on Vlan924 from FULL to DOWN, Neighbor Down: Interface down or detached
Jan 20 10:36:28.717: %OSPF-5-ADJCHG: Process 1, Nbr 10.240.240.5 on Vlan924 from FULL to DOWN, Neighbor Down: Interface down or detached
Jan 20 10:50:34.200: %OSPF-5-ADJCHG: Process 1, Nbr 10.240.240.5 on Vlan924 from LOADING to FULL, Loading Done
Jan 20 10:50:58.413: %OSPF-5-ADJCHG: Process 1, Nbr 10.3.2.1 on Vlan924 from LOADING to FULL, Loading Done
Jan 20 10:51:10.770: %OSPF-5-ADJCHG: Process 1, Nbr 10.3.2.1 on Vlan924 from FULL to DOWN, Neighbor Down: Interface down or detached
Jan 20 10:51:10.770: %OSPF-5-ADJCHG: Process 1, Nbr 10.240.240.5 on Vlan924 from FULL to DOWN, Neighbor Down: Interface down or detached
Jan 20 10:53:57.793: %OSPF-5-ADJCHG: Process 1, Nbr 10.240.240.5 on Vlan924 from LOADING to FULL, Loading Done
Jan 20 10:54:21.186: %OSPF-5-ADJCHG: Process 1, Nbr 10.3.2.1 on Vlan924 from LOADING to FULL, Loading Done
Jan 20 10:54:28.794: %OSPF-5-ADJCHG: Process 1, Nbr 10.3.2.1 on Vlan924 from FULL to DOWN, Neighbor Down: Interface down or detached
Jan 20 10:54:28.794: %OSPF-5-ADJCHG: Process 1, Nbr 10.240.240.5 on Vlan924 from FULL to DOWN, Neighbor Down: Interface down or detached
Jan 20 10:57:39.014: %OSPF-5-ADJCHG: Process 1, Nbr 10.240.240.5 on Vlan924 from LOADING to FULL, Loading Done
Jan 20 10:57:48.813: %OSPF-5-ADJCHG: Process 1, Nbr 10.240.240.5 on Vlan924 from FULL to DOWN, Neighbor Down: Interface down or detached
Jan 20 10:59:09.728: %OSPF-5-ADJCHG: Process 1, Nbr 10.240.240.5 on Vlan924 from LOADING to FULL, Loading Done
Jan 20 10:59:28.412: %OSPF-5-ADJCHG: Process 1, Nbr 10.3.2.1 on Vlan924 from LOADING to FULL, Loading Done
Jan 20 12:25:16.993: %OSPF-5-ADJCHG: Process 1, Nbr 10.128.8.1 on Vlan919 from FULL to DOWN, Neighbor Down: Interface down or detached
Jan 20 12:25:16.993: %OSPF-5-ADJCHG: Process 1, Nbr 10.240.240.5 on Vlan919 from FULL to DOWN, Neighbor Down: Interface down or detached
Jan 20 12:29:31.453: %OSPF-5-ADJCHG: Process 1, Nbr 10.240.240.5 on Vlan919 from LOADING to FULL, Loading Done
Jan 20 12:29:55.549: %OSPF-5-ADJCHG: Process 1, Nbr 10.128.8.1 on Vlan919 from LOADING to FULL, Loading Done

has anyone seen anything like this before? 

or has good input to help me narrow down this problem? 

Thank you 

Niklas 

[balaji.bandi]

Since you have mentioned you have uplifted the bandwidth,i would start with that road to investigate.

1. check the interface stable - no up and down,

2. No interface negotiation issue.

3. 10GB - did you change jumbo frame MTU ?

[Niklas.D]

Hello

1. check the interface stable - no up and down,

No link down on the interface  since it went live

2. No interface negotiation issue.

Full-duplex, 10Gb/s, link type is auto, media type is 10GBase-LR, looks correct comfirming with the ISP what they say! 

3. 10GB - did you change jumbo frame MTU ?

Primary

interface TenGigabitEthernet1/1
description IPO-Wan
switchport trunk allowed vlan 902,905,906,911,913,914,917-919,924,926,929,934
switchport trunk allowed vlan add 935
switchport mode trunk
mtu 9198

Secondary

description --IPO-Wan--
switchport trunk allowed vlan 902,905,906,911-914,917-919,924,926,929,934,935
switchport trunk allowed vlan add 937
switchport mode trunk
mtu 9198

[balaji.bandi]

Primary - your site

seconday - Far end right ?

MTU you configured ? who suggested provider ?

can you post interface output and show ip ospf neighbour.

(to confirm before when it was 1GB all stabled ? any other changes other than upgrade link ?)

[Niklas.D]

Both are my sites in this case. 

Primary = Datacenter Uplink One

Seacondary = Datacenter Uplink two

The Seconday site is working all the time. 

Its our config, on the MTU so i can change that. ( so i can change this on the fly) 

TenGigabitEthernet1/1 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet Port, address is 24e9.b342.fa58 (bia 24e9.b342.fa58)
Description: IPO-Wan
MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s, link type is auto, media type is 10GBase-LR
input flow-control is on, output flow-control is on
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output never, output hang never
Last clearing of "show interface" counters 4d02h
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 3040000 bits/sec, 884 packets/sec
5 minute output rate 10249000 bits/sec, 2259 packets/sec
295962878 packets input, 235558379809 bytes, 0 no buffer
Received 19301234 broadcasts (19140253 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
486825967 packets output, 184397433529 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

show ip ospf neighbor

10.128.83.1 10 FULL/DR 00:00:35 10.240.2.222 Vlan902
10.240.240.5 1 FULL/BDR 00:00:39 10.240.2.210 Vlan902
10.19.2.1 10 FULL/DR 00:00:35 10.240.1.30 Vlan531
10.240.240.5 1 FULL/DROTHER 00:00:36 10.240.1.19 Vlan531
10.240.1.206 10 FULL/DR 00:00:35 10.240.1.206 Vlan538
10.240.240.5 1 FULL/DROTHER 00:00:33 10.240.1.195 Vlan538
10.240.240.5 1 FULL/BDR 00:00:36 10.240.240.5 Vlan1000
10.240.3.108 10 FULL/DR 00:00:37 10.240.3.108 Vlan936
10.240.240.5 1 FULL/DROTHER 00:00:35 10.240.3.99 Vlan936
10.240.3.94 10 FULL/DR 00:00:38 10.240.3.94 Vlan935
10.240.240.5 1 FULL/BDR 00:00:36 10.240.3.83 Vlan935
10.240.3.78 10 FULL/DR 00:00:37 10.240.3.78 Vlan934
10.240.240.5 1 FULL/BDR 00:00:36 10.240.3.67 Vlan934
10.240.2.254 10 FULL/DR 00:00:33 10.240.2.254 Vlan929
10.240.240.5 1 FULL/DROTHER 00:00:33 10.240.2.243 Vlan929
10.24.2.1 10 FULL/DR 00:00:33 10.240.2.206 Vlan926
10.240.240.5 1 FULL/DROTHER 00:00:38 10.240.2.195 Vlan926
10.240.2.126 10 FULL/DR 00:00:36 10.240.2.126 Vlan925
10.3.2.1 10 FULL/DR 00:00:33 10.240.2.190 Vlan924
10.240.240.5 1 FULL/BDR 00:00:36 10.240.2.179 Vlan924
10.128.6.1 10 FULL/DR 00:00:38 10.240.3.156 Vlan918
10.240.240.5 1 FULL/BDR 00:00:38 10.240.3.147 Vlan918
10.128.4.1 10 FULL/DR 00:00:37 10.240.3.140 Vlan917
10.240.240.5 1 FULL/BDR 00:00:36 10.240.3.131 Vlan917
10.240.2.145 10 FULL/DR 00:00:33 10.240.2.158 Vlan914
10.240.240.5 1 FULL/BDR 00:00:36 10.240.2.147 Vlan914
10.240.2.129 10 FULL/DR 00:00:35 10.240.2.142 Vlan913
10.240.240.5 1 FULL/BDR 00:00:36 10.240.2.131 Vlan913
10.128.2.1 10 FULL/DR 00:00:38 10.240.128.46 Vlan911
10.240.240.5 1 FULL/BDR 00:00:38 10.240.128.35 Vlan911
10.128.5.1 10 FULL/DR 00:00:34 10.240.3.124 Vlan906
10.240.240.5 1 FULL/BDR 00:00:36 10.240.3.115 Vlan906
10.240.2.94 10 FULL/DR 00:00:38 10.240.2.94 Vlan905
10.240.240.5 1 FULL/BDR 00:00:38 10.240.2.82 Vlan905
10.240.3.94 10 FULL/DR 00:00:38 10.240.2.46 Vlan535
10.240.240.5 1 FULL/DROTHER 00:00:39 10.240.2.35 Vlan535
10.240.3.78 10 FULL/DR 00:00:37 10.240.2.30 Vlan534
10.240.240.5 1 FULL/DROTHER 00:00:38 10.240.2.19 Vlan534
10.23.2.1 10 FULL/DR 00:00:35 10.240.1.62 Vlan533
10.240.240.5 1 FULL/DROTHER 00:00:39 10.240.1.51 Vlan533
10.20.2.1 10 FULL/DR 00:00:36 10.240.1.46 Vlan532
10.240.240.5 1 FULL/DROTHER 00:00:33 10.240.1.35 Vlan532
10.17.2.1 10 FULL/DR 00:00:35 10.240.1.222 Vlan527
10.240.240.5 1 FULL/DROTHER 00:00:39 10.240.1.211 Vlan527
10.240.2.126 10 FULL/DR 00:00:36 10.240.1.126 Vlan525
10.240.2.110 10 FULL/DR 00:00:35 10.240.1.110 Vlan512
10.240.240.5 1 FULL/DROTHER 00:00:39 10.240.1.99 Vlan512
10.240.128.14 10 FULL/DR 00:00:35 10.240.128.14 Vlan510
10.240.240.5 1 FULL/DROTHER 00:00:36 10.240.128.3 Vlan510
10.128.1.1 10 FULL/DR 00:00:37 10.240.128.30 Vlan509
10.240.240.5 1 FULL/DROTHER 00:00:33 10.240.128.19 Vlan509
10.240.0.33 20 FULL/DR 00:00:33 10.240.1.78 Vlan504
10.240.240.5 1 FULL/BDR 00:00:36 10.240.1.66 Vlan504
10.240.240.5 1 FULL/DR 00:00:33 10.240.255.11 Vlan301

Everything worked fine on the old 1GB up-link correct.

[balaji.bandi]

May be i am confused here -- is this different sites where you have 10GB link between these sites ?

Primary = Datacenter Uplink One

Seacondary = Datacenter Uplink two

The Seconday site is working all the time.

MTU can be changed but suggest to do it in maintenance window always. - what is the MTU Set at provider side?

I maintain several 10GB Link with Long reach datacenter set " mtu 9216" (may be not the case with you - this is in general information)

I would ask to test some ping of point to point link atlease 10000 pings and see any packet loss ?

this test required to conduct from primary site router - where 10GB Link termnated to Ohter site IP where 10GB Link termnated, and let know .

example : i did ping 1000 pings below results on 10GB DCI Link.

Success rate is 100 percent (1000/1000), round-trip min/avg/max = 1/1/84 ms

[Niklas.D]

So if we take vlan 911 i have it on my Datacenter it has a Priamry Uplink and a Secondary and its connects to a Site in the WAN 

Datacenter:

Primary Link - IP: 10.240.128.34

ping 10.240.128.46 source vlan 911 repeat 10000

Success rate is 100 percent (10000/10000), round-trip min/avg/max = 1/2/8 ms

ping 10.240.128.35 source vlan 911 repeat 10000

Success rate is 100 percent (10000/10000), round-trip min/avg/max = 1/3/44 ms

Secondary Link - IP: 10.240.128.35

ping 10.240.128.46 source vlan 911 repeat 10000

Success rate is 100 percent (10000/10000), round-trip min/avg/max = 1/2/20 ms

ping 10.240.128.35 source vlan 911 repeat 10000

Success rate is 100 percent (10000/10000), round-trip min/avg/max = 1/3/44 ms

Site:

Link - IP: 10.240.128.46

And then randomly the vlan 911 will go in DOWN mode, then no traffik will work

So when 911 goes in DOWN, i still have 20 other VLANs that are still working. 

so i could 

[balaji.bandi]

So you have only issue with VLAN 911 ? as per below messge ?

So when 911 goes in DOWN, i still have 20 other VLANs that are still working. -- BB  this means VLAN 911 Ping loss , will not have other VLAN Ping loss.

As per the diagram you have only 1 Linlk, Do you have any other path ? between these site ?

can you capture all the logs when the issue occurs from both the side, how many ports associated with VLAN 911 ?

[Jaderson Pessoa]

Hello guy,

Please post here full configuration of this ospf neighborship, a simple draw of your topology will help us.

Check ospf debug to looking for if something is missing, like mtu, bad nic interface.

post here output from:
show running-config
show ip ospf neighbor
show ip ospf database
show ip int bri
show ip ospf int brief
show ip route
show ip route ospf

[Niklas.D]

Topologyskis

[Niklas.D · Accepted solution]

This was a spanning tree issue, the ISP in there new config let spanning tree thrue on there side, adding "spanning-tree bpdufilter enable" on the port connected to the ISP solved the Issue!