ACI OSPF stuck in Initializing after hardware refresh

[MichaelMcCoyOU]

Hello all,  We refreshed our hardware last week at our DR site.  We replaced three (3) APICS and two (2) Spines.  Everything went off mostly without a hitch, but we did notice the next day that one of the spine switches was still in initializing OSPF state.  

The other Spine and fully up, and all the APICS are fully fit.  

We have checked the timers on both ends, the MTU for any mismatches.  This change was strictly a hardware refresh and other than decommission/commission etc. small things like that, the GUI was left alone.  Has anyone seen this or perhaps have any insights as to what the problem may be?

[Daniel Slyk]

Hi Michael,

I assume we are talking about OSPF adjacency between your Spine and IPN switch.
"Initializing" means the Spine is receiving OSPF hellos from a neighbor, but its own Router ID is not appearing in that neighbor's hello packet.

Some stuff I would check:

1. Check for "ghost" entries from old hardware, to make sure the old Spine is not there anymore. Fabric → Inventory → Fabric Membership
2. Check OSPF neighbor details on the IPN side, to see how it sees the affected Spine. Compare with the healthy Spine.
3. Check LLDP neighbors on the affected Spine and also on the IPN, to confirm the links are physically seen correctly.
4. Check the Faults tab on the affected Spine in APIC.
5. Try a reload.
6. Last resort, decommission the Spine from APIC first, then wipe clean and reload.
7. Very last resort, Cisco TAC.

Try to get more info please, debugs etc., then let us know and lets see.

Good luck.

[MichaelMcCoyOU]

Thank you @Daniel Slyk I will check on these and report back, and yes OSPF.  Thank you again.

[Daniel Slyk]

Hi Michael,

Sure.

It's clear that its OSPF, I was just double checking and making sure that it is the OSPF adjacency between your Spine and the IPN device.

Thanks.

[MichaelMcCoyOU]

Morning, So the issue turned out to be a L1 issue.  In our HW refresh some cables became mislabeled and got swapped.  I knew we didn't have any configuration changes, so I probably should have thought more about L1 before posting.  Thank you for taking the time to respond though.

[Daniel Slyk]

Hi Michael,

thanks for letting us know.

Stuff happens, to all of us 😉.

I am glad you figured it out.