10-25-2021 01:31 AM
Hi all,
We are facing an issue where one device (ME3600X) is unable to establish an OSPF adjacency with its neighbours. It connects to the other OSPF peers via a /29 network with OSPF in broadcast mode. The other peers within the /29 are able to peer successfully. On the other peers, OSPF goes from LOADING -> FULL as you would expect, but this repeats every few seconds. On the problem box OSPF repeatedly logs LOADING -> DOWN and "Dead timer expired".
Things I have checked:
- MTU common across all devices
- Layer 3 connectivity also fine between all peers
- OSPF timers the same across all peers
The following messages were logged on the peer on which OSPF doesn't establish:
Oct 25 08:21:56.399: OSPF-1 ADJ Vl901: Exchange Done with 1.1.2.127
Oct 25 08:21:56.399: OSPF-1 ADJ Vl901: Send DBD to 1.1.2.127 seq 0x52B95 opt 0x52 flag 0x0 len 32
Oct 25 08:21:56.399: OSPF-1 ADJ Vl901: Send with youngest Key 1
Oct 25 08:21:57.619: OSPF-1 ADJ Vl901: Send with youngest Key 1
Oct 25 08:21:58.667: OSPF-1 ADJ Vl901: Send with youngest Key 1
Oct 25 08:21:58.935: OSPF-1 ADJ Vl901: Nbr 1.1.2.127 retransmitting LS REQ
Oct 25 08:21:58.935: OSPF-1 ADJ Vl901: Send with youngest Key 1
Oct 25 08:21:58.935: OSPF-1 ADJ Vl901: Send LS REQ to 1.1.2.127 length 1464 LSA count 120
Oct 25 08:21:59.539: OSPF-1 ADJ Vl901: 1.1.3.226 address 1.1.2.49 is dead
Oct 25 08:21:59.539: OSPF-1 ADJ Vl901: 1.1.3.226 address 1.1.2.49 is dead, state DOWN
Oct 25 08:21:59.539: %OSPF-5-ADJCHG: Process 1, Nbr 1.1.3.226 on Vlan901 from LOADING to DOWN, Neighbor Down: Dead timer expired
Oct 25 08:21:59.539: OSPF-1 ADJ Vl901: Nbr 1.1.3.226: Clean-up dbase exchange
Oct 25 08:21:59.539: OSPF-1 ADJ Vl901: Neighbor change event
Oct 25 08:21:59.539: OSPF-1 ADJ Vl901: DR/BDR election
Oct 25 08:21:59.539: OSPF-1 ADJ Vl901: Elect BDR 1.1.2.127
Oct 25 08:21:59.539: OSPF-1 ADJ Vl901: Elect DR 1.1.2.127
Oct 25 08:21:59.539: OSPF-1 ADJ Vl901: DR: 1.1.2.127 (Id) BDR: 1.1.2.127 (Id)
Oct 25 08:21:59.539: OSPF-1 ADJ Vl901: Remember old DR 1.1.3.226 (id)
Oct 25 08:21:59.715: OSPF-1 ADJ Vl901: Send with youngest Key 1
Oct 25 08:21:59.719: OSPF-1 ADJ Vl901: 2 Way Communication to 1.1.3.226, state 2WAY
Oct 25 08:21:59.719: OSPF-1 ADJ Vl901: Neighbor change event
Oct 25 08:21:59.719: OSPF-1 ADJ Vl901: DR/BDR election
Oct 25 08:21:59.719: OSPF-1 ADJ Vl901: Elect BDR 1.1.2.127
Oct 25 08:21:59.719: OSPF-1 ADJ Vl901: Elect DR 1.1.3.226
Oct 25 08:21:59.719: OSPF-1 ADJ Vl901: DR: 1.1.3.226 (Id) BDR: 1.1.2.127 (Id)
Oct 25 08:21:59.719: OSPF-1 ADJ Vl901: Nbr 1.1.3.226: Prepare dbase exchange
Oct 25 08:21:59.719: OSPF-1 ADJ Vl901: Send DBD to 1.1.3.226 seq 0x1A5F opt 0x52 flag 0x7 len 32
Oct 25 08:21:59.719: OSPF-1 ADJ Vl901: Send with youngest Key 1
Oct 25 08:21:59.719: OSPF-1 ADJ Vl901: Remember old DR 1.1.2.127 (id)
Oct 25 08:21:59.719: OSPF-1 ADJ Vl901: Neighbor change event
Oct 25 08:21:59.719: OSPF-1 ADJ Vl901: DR/BDR election
Oct 25 08:21:59.719: OSPF-1 ADJ Vl901: Elect BDR 1.1.2.127
Oct 25 08:21:59.719: OSPF-1 ADJ Vl901: Elect DR 1.1.3.226
Oct 25 08:21:59.719: OSPF-1 ADJ Vl901: DR: 1.1.3.226 (Id) BDR: 1.1.2.127 (Id)
Oct 25 08:21:59.915: OSPF-1 ADJ Vl901: Rcv DBD from 1.1.3.226 seq 0x56C70 opt 0x52 flag 0x7 len 32 mtu 1500 state EXSTART
Oct 25 08:21:59.915: OSPF-1 ADJ Vl901: NBR Negotiation Done. We are the SLAVE
Oct 25 08:21:59.915: OSPF-1 ADJ Vl901: Nbr 1.1.3.226: Summary list built, size 52
Corresponding logs from 1.1.2.127:
Oct 25 08:21:53.494: OSPF-1 ADJ Vl901: Send with youngest Key 1 Oct 25 08:21:53.494: OSPF-1 ADJ Vl901: Rcv DBD from 1.1.2.126 seq 0x2 opt 0x52 flag 0x7 len 32 mtu 1500 state INIT Oct 25 08:21:53.494: OSPF-1 ADJ Vl901: 2 Way Communication to 1.1.2.126, state 2WAY Oct 25 08:21:53.494: OSPF-1 ADJ Vl901: Neighbor change event Oct 25 08:21:53.494: OSPF-1 ADJ Vl901: DR/BDR election Oct 25 08:21:53.494: OSPF-1 ADJ Vl901: Elect BDR 1.1.2.127 Oct 25 08:21:53.494: OSPF-1 ADJ Vl901: Elect DR 1.1.3.226 Oct 25 08:21:53.494: OSPF-1 ADJ Vl901: DR: 1.1.3.226 (Id) Oct 25 08:21:53.494: OSPF-1 ADJ Vl901: BDR: 1.1.2.127 (Id) Oct 25 08:21:53.494: OSPF-1 ADJ Vl901: Nbr 1.1.2.126: Prepare dbase exchange Oct 25 08:21:53.494: OSPF-1 ADJ Vl901: Send DBD to 1.1.2.126 seq 0x52B90 opt 0x52 flag 0x7 len 32 Oct 25 08:21:53.494: OSPF-1 ADJ Vl901: Send with youngest Key 1 Oct 25 08:21:53.494: OSPF-1 ADJ Vl901: First DBD and we are not SLAVE Oct 25 08:21:54.226: OSPF-1 ADJ Vl901: Neighbor change event Oct 25 08:21:54.226: OSPF-1 ADJ Vl901: DR/BDR election Oct 25 08:21:54.226: OSPF-1 ADJ Vl901: Elect BDR 1.1.2.127 Oct 25 08:21:54.226: OSPF-1 ADJ Vl901: Elect DR 1.1.3.226 Oct 25 08:21:54.226: OSPF-1 ADJ Vl901: DR: 1.1.3.226 (Id) Oct 25 08:21:54.226: OSPF-1 ADJ Vl901: BDR: 1.1.2.127 (Id) Oct 25 08:21:54.226: OSPF-1 ADJ Vl901: Neighbor change event Oct 25 08:21:54.226: OSPF-1 ADJ Vl901: DR/BDR election Oct 25 08:21:54.226: OSPF-1 ADJ Vl901: Elect BDR 1.1.2.127 Oct 25 08:21:54.226: OSPF-1 ADJ Vl901: Elect DR 1.1.3.226 Oct 25 08:21:54.226: OSPF-1 ADJ Vl901: DR: 1.1.3.226 (Id) Oct 25 08:21:54.226: OSPF-1 ADJ Vl901: BDR: 1.1.2.127 (Id) Oct 25 08:21:55.162: OSPF-1 ADJ Vl901: Send with youngest Key 1 Oct 25 08:21:55.162: OSPF-1 ADJ Vl901: Rcv DBD from 1.1.2.126 seq 0x52B90 opt 0x52 flag 0x2 len 1072 mtu 1500 state EXSTART Oct 25 08:21:55.162: OSPF-1 ADJ Vl901: NBR Negotiation Done. We are the MASTER Oct 25 08:21:55.162: OSPF-1 ADJ Vl901: Nbr 1.1.2.126: Summary list built, size 343 Oct 25 08:21:55.162: OSPF-1 ADJ Vl901: Send DBD to 1.1.2.126 seq 0x52B91 opt 0x52 flag 0x3 len 1412 Oct 25 08:21:55.162: OSPF-1 ADJ Vl901: Send with youngest Key 1 Oct 25 08:21:55.906: OSPF-1 ADJ Vl901: Neighbor change event Oct 25 08:21:55.906: OSPF-1 ADJ Vl901: DR/BDR election Oct 25 08:21:55.906: OSPF-1 ADJ Vl901: Elect BDR 1.1.2.127 Oct 25 08:21:55.906: OSPF-1 ADJ Vl901: Elect DR 1.1.3.226 Oct 25 08:21:55.906: OSPF-1 ADJ Vl901: DR: 1.1.3.226 (Id) Oct 25 08:21:55.906: OSPF-1 ADJ Vl901: BDR: 1.1.2.127 (Id) Oct 25 08:21:55.914: OSPF-1 ADJ Vl901: Rcv DBD from 1.1.2.126 seq 0x52B91 opt 0x52 flag 0x0 len 32 mtu 1500 state EXCHANGE Oct 25 08:21:55.914: OSPF-1 ADJ Vl901: Send DBD to 1.1.2.126 seq 0x52B92 opt 0x52 flag 0x3 len 1412 Oct 25 08:21:55.914: OSPF-1 ADJ Vl901: Send with youngest Key 1 Oct 25 08:21:56.466: OSPF-1 ADJ Vl901: Send with youngest Key 1 Oct 25 08:21:56.466: OSPF-1 ADJ Vl901: Rcv DBD from 1.1.2.126 seq 0x52B92 opt 0x52 flag 0x0 len 32 mtu 1500 state EXCHANGE Oct 25 08:21:56.470: OSPF-1 ADJ Vl901: Send DBD to 1.1.2.126 seq 0x52B93 opt 0x52 flag 0x3 len 1412 Oct 25 08:21:56.470: OSPF-1 ADJ Vl901: Send with youngest Key 1 Oct 25 08:21:56.874: OSPF-1 ADJ Vl901: Neighbor change event Oct 25 08:21:56.874: OSPF-1 ADJ Vl901: DR/BDR election Oct 25 08:21:56.874: OSPF-1 ADJ Vl901: Elect BDR 1.1.2.127 Oct 25 08:21:56.874: OSPF-1 ADJ Vl901: Elect DR 1.1.3.226 Oct 25 08:21:56.874: OSPF-1 ADJ Vl901: DR: 1.1.3.226 (Id) Oct 25 08:21:56.874: OSPF-1 ADJ Vl901: BDR: 1.1.2.127 (Id) Oct 25 08:21:57.018: OSPF-1 ADJ Vl901: Rcv DBD from 1.1.2.126 seq 0x52B93 opt 0x52 flag 0x0 len 32 mtu 1500 state EXCHANGE Oct 25 08:21:57.018: OSPF-1 ADJ Vl901: Send DBD to 1.1.2.126 seq 0x52B94 opt 0x52 flag 0x3 len 1412 Oct 25 08:21:57.018: OSPF-1 ADJ Vl901: Send with youngest Key 1 Oct 25 08:21:58.150: OSPF-1 ADJ Vl901: Send with youngest Key 1 Oct 25 08:21:58.170: OSPF-1 ADJ Vl901: Rcv DBD from 1.1.2.126 seq 0x52B94 opt 0x52 flag 0x0 len 32 mtu 1500 state EXCHANGE Oct 25 08:21:58.170: OSPF-1 ADJ Vl901: Send DBD to 1.1.2.126 seq 0x52B95 opt 0x52 flag 0x1 len 1372 Oct 25 08:21:58.170: OSPF-1 ADJ Vl901: Send with youngest Key 1 Oct 25 08:21:58.586: OSPF-1 ADJ Vl901: Send with youngest Key 1 Oct 25 08:21:58.590: OSPF-1 ADJ Vl901: Rcv DBD from 1.1.2.126 seq 0x52B95 opt 0x52 flag 0x0 len 32 mtu 1500 state EXCHANGE Oct 25 08:21:58.590: OSPF-1 ADJ Vl901: Exchange Done with 1.1.2.126 Oct 25 08:21:58.590: OSPF-1 ADJ Vl901: Synchronized with 1.1.2.126, state FULL Oct 25 08:21:59.790: OSPF-1 ADJ Vl901: Send with youngest Key 1 Oct 25 08:22:00.886: OSPF-1 ADJ Vl901: Send with youngest Key 1
In these log file, the peers are as follows:
1.1.2.126 <== problem peer
1.1.2.127 & 1.1.3.226 <== other, working OSPF peers, all of which have interfaces in the /29
I can't see any errors here so am struggling to understand why OSPF isn't establishing, I would be very grateful of any input.
Thanks
George
Solved! Go to Solution.
10-27-2021 06:25 AM
To anyone else facing the same issue, I managed to solve this in the end.
The issue appeared to have been the result of a software bug. I checked (far too late in the process!) the image on the box and found it to be a very old version 15.4-1.S1. I updated the image and OSPF came up, and stayed up.
Thanks to all for their support.
10-25-2021 01:50 AM - edited 10-25-2021 01:52 AM
Looks for me split brain issue i guess here, I am thinking devices to elect more than 1 DR in the Segment.
can you post below output :
show run interface vlan 901
show ip ospf interface vlan 901
show ip ospf neighbor
some troubleshoot tips :
https://www.cisco.com/c/en/us/support/docs/ip/open-shortest-path-first-ospf/13699-29.html
10-25-2021 01:57 AM
Thanks, not encountered this before, here's the output mentioned:
WEST-CORE1#sh run int vl901 Building configuration... Current configuration : 274 bytes ! interface Vlan901 ip address 1.1.2.52 255.255.255.248 ip ospf authentication message-digest ip ospf message-digest-key 1 md5 7 XXXXXXXXXXXXXXXX ip ospf network broadcast ip ospf hello-interval 1 ip ospf 1 area 0 end WEST-CORE1#show ip ospf interface vlan 901 Vlan901 is up, line protocol is up Internet Address 1.1.2.52/29, Area 0, Attached via Interface Enable Process ID 1, Router ID 1.1.2.126, Network Type BROADCAST, Cost: 10 Topology-MTID Cost Disabled Shutdown Topology Name 0 10 no no Base Enabled by interface config, including secondary ip addresses Transmit Delay is 1 sec, State DR, Priority 1 Designated Router (ID) 1.1.2.126, Interface address 1.1.2.52 No backup designated router on this network Timer intervals configured, Hello 1, Dead 4, Wait 4, Retransmit 5 oob-resync timeout 40 Hello due in 00:00:00 Supports Link-local Signaling (LLS) Cisco NSF helper support enabled IETF NSF helper support enabled Can be protected by per-prefix Loop-Free FastReroute Can be used for per-prefix Loop-Free FastReroute repair paths Index 2/2, flood queue length 0 Next 0x0(0)/0x0(0) Last flood scan length is 1, maximum is 1 Last flood scan time is 0 msec, maximum is 0 msec Neighbor Count is 0, Adjacent neighbor count is 0 Suppress hello for 0 neighbor(s) Cryptographic authentication enabled Youngest key id is 1 WEST-CORE1#show ip ospf neighbor Neighbor ID Pri State Dead Time Address Interface 1.1.2.127 1 LOADING/BDR 00:00:03 1.1.2.51 Vlan901 1.1.3.226 1 LOADING/DR 00:00:03 1.1.2.49 Vlan901 WEST-CORE1#
10-25-2021 02:26 AM
Are you trying to neighbour with 2 peers ? or only one, do you have any network how these connected ?
because of this this will try to establish neighbourship in the broadcast domain, that is vlan 901 with any IP range (1.1.2.52 255.255.255.248)
ip ospf network broadcast
try :
ip ospf network point-to-point
this is example ? but we need to know more information as i have asked.
10-25-2021 02:47 AM
Hi BB
No, there are 4 peers in a /29 - like this:
The green one doesn't yet have OSPF configured on this VLAN, the red is the one mentioned above on which OSPF logs "dead timer expired"
All peers can ping each other within 1.1.2.48/29. All are in broadcast mode - which I believe is correct for this topology?
10-25-2021 02:59 AM
is this a real or LAB environment ?
Can you post same output (the one posted before from all routers)
and which one you like to be DR ?
10-25-2021 03:17 AM
This is a real environment, not lab, although of course the IPs have been edited for public posting.
I'm happy with 1.1.3.226 being the DR and when all is configured I would have 1.1.3.227 as BDR, but for the moment the other peer (1.1.2.127) is fine.
Output from 1.1.3.226 below:
DX-CORE1-NEW#sh run int vl901 Building configuration... Current configuration : 276 bytes ! interface Vlan901 ip address 1.1.2.49 255.255.255.248 ip ospf authentication message-digest ip ospf message-digest-key 1 md5 7 XXXXXXXXXXXXXXXXXXX ip ospf network broadcast ip ospf hello-interval 1 ip ospf 1 area 0 end DX-CORE1-NEW#show ip ospf interface vlan 901 Vlan901 is up, line protocol is up Internet Address 1.1.2.49/29, Area 0, Attached via Interface Enable Process ID 1, Router ID 1.1.3.226, Network Type BROADCAST, Cost: 10 Topology-MTID Cost Disabled Shutdown Topology Name 0 10 no no Base Enabled by interface config, including secondary ip addresses Transmit Delay is 1 sec, State DR, Priority 1 Designated Router (ID) 1.1.3.226, Interface address 1.1.2.49 Backup Designated router (ID) 1.1.2.127, Interface address 1.1.2.51 Timer intervals configured, Hello 1, Dead 4, Wait 4, Retransmit 5 oob-resync timeout 40 Hello due in 00:00:00 Supports Link-local Signaling (LLS) Cisco NSF helper support enabled IETF NSF helper support enabled Can be protected by per-prefix Loop-Free FastReroute Can be used for per-prefix Loop-Free FastReroute repair paths Index 5/5, flood queue length 0 Next 0x0(0)/0x0(0) Last flood scan length is 1, maximum is 40 Last flood scan time is 0 msec, maximum is 4 msec Neighbor Count is 2, Adjacent neighbor count is 2 Adjacent with neighbor 1.1.2.126 Adjacent with neighbor 1.1.2.127 (Backup Designated Router) Suppress hello for 0 neighbor(s) Cryptographic authentication enabled Youngest key id is 1 DX-CORE1-NEW#show ip ospf neighbor | i 901 1.1.2.126 1 FULL/DROTHER 00:00:03 1.1.2.52 Vlan901 1.1.2.127 1 FULL/BDR 00:00:03 1.1.2.51 Vlan901
Output from 1.1.2.127 below:
EAST-CORE1#sh run int vl901 Building configuration... Current configuration : 275 bytes ! interface Vlan901 ip address 1.1.2.51 255.255.255.248 ip ospf authentication message-digest
ip ospf message-digest-key 1 md5 7 XXXXXXXXXXXXXXXXXXX ip ospf network broadcast ip ospf hello-interval 1 ip ospf 1 area 0 end EAST-CORE1#show ip ospf interface vlan 901 Vlan901 is up, line protocol is up Internet Address 1.1.2.51/29, Area 0, Attached via Interface Enable Process ID 1, Router ID 1.1.2.127, Network Type BROADCAST, Cost: 10 Topology-MTID Cost Disabled Shutdown Topology Name 0 10 no no Base Enabled by interface config, including secondary ip addresses Transmit Delay is 1 sec, State BDR, Priority 1 Designated Router (ID) 1.1.3.226, Interface address 1.1.2.49 Backup Designated router (ID) 1.1.2.127, Interface address 1.1.2.51 Timer intervals configured, Hello 1, Dead 4, Wait 4, Retransmit 5 oob-resync timeout 40 Hello due in 00:00:00 Supports Link-local Signaling (LLS) Cisco NSF helper support enabled IETF NSF helper support enabled Can be protected by per-prefix Loop-Free FastReroute Can be used for per-prefix Loop-Free FastReroute repair paths Index 1/1, flood queue length 0 Next 0x0(0)/0x0(0) Last flood scan length is 0, maximum is 40 Last flood scan time is 0 msec, maximum is 792 msec Neighbor Count is 2, Adjacent neighbor count is 1 Adjacent with neighbor 1.1.3.226 (Designated Router) Suppress hello for 0 neighbor(s) Cryptographic authentication enabled Youngest key id is 1 EAST-CORE1#show ip ospf neighbor | i 901 1.1.2.126 1 FULL/DROTHER 00:00:03 1.1.2.52 Vlan901 1.1.3.226 1 FULL/DR 00:00:03 1.1.2.49 Vlan901
10-25-2021 04:00 AM
thanks for the information, if they all part of VLAN 901 and you are sure (since i did not see otehr device config here)
by default all OSPF take as priorty 1
This for DR
ip ospf priority 100
This for BDR
ip ospf priority 90
DROTHER
This for DR
ip ospf priority 0
and test it.
10-25-2021 03:37 AM
Hello,
just for clarification: you have Vlan901 configured:
interface Vlan901
ip address 1.1.2.52 255.255.255.248
The host range is:
1.1.2.49 - 1.1.2.54
I am not sure how these two IP addresses can be part of Vlan 901, can you clarify the IP addressing ?
Vl901: Elect BDR 1.1.2.127
Vl901: Elect DR 1.1.3.226
10-25-2021 03:48 AM
Hi Georg,
Sure I can see how that is confusing. The IPs 1.1.2.127 and 1.1.3.226 are the IPs of Loopback0 on both of these two peers, which all have a 32 bit mask and are also the OSPF router-ids. In VLAN901 each of these peers has an IP and they can all ping each other on those. If OSPF came up it would share the loopback IPs and everything would work.
The diagram I posted before showed this, let me know if this doesn't make sense for any reason:
10-25-2021 03:55 AM
Hello,
that makes sense of course. I assume the IP addresses of the non-Meraki devices are configured on physical interfaces ?
10-25-2021 04:31 AM
@Georg Pauwen wrote:I assume the IP addresses of the non-Meraki devices are configured on physical interfaces ?
Well, these are ME3600X devices so the IP addresses are on SVIs - but to all intents and purposes these behave like physical interfaces so yes, the IPs in 1.1.2.48/29 are effectively assigned to "physical" interfaces.
I have also in the meantime tried using `ip ospf mtu-ignore` on the interfaces concerned (despite the MTU being the same, just to rule this out) but this has also had no effect on the issue.
I am thinking next I will reset the OSPF processes on all peers just to see if something is cached in memory but I cannot do that until this evening.
10-25-2021 04:48 AM
Hello,
in your original post, it sounded you had one Meraki device mixed with other devices:
--> We are facing an issue where one device (ME3600X) is unable to establish an OSPF adjacency with its neighbours
Either way, try and configure Loopback IP address that are completely different from the IP addressing you use for Vlan 901, e.g.:
Loopback1
ip address 11.11.11.11 255.255.255.255
etc.
10-25-2021 07:08 AM
@Georg Pauwen wrote:in your original post, it sounded you had one Meraki device mixed with other devices:
Sorry for the confusion - ME3600X is a Cisco metro Ethernet layer 3 switch
@Georg Pauwen wrote:Either way, try and configure Loopback IP address that are completely different from the IP addressing you use for Vlan 901, e.g.:
Loopback1
ip address 11.11.11.11 255.255.255.255
Thanks - but why would this help? The Loopback addresses are definitely not in 1.1.2.48/29 so surely it would be the same? I'm all for trying stuff but like there to be some logic to it before jumping in!
10-25-2021 10:07 AM
Hello
@RouteOps_gm wrote:
- MTU common across all devices
- Layer 3 connectivity also fine between all peers
- OSPF timers the same across all peers
Post output
debug ip ospf adj
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide