Infra VLAN mismatch when rebuilding the Fabric

[m1xed0s]

I have a small ACI fabric (3 APIC, 2 SPINE and 2 LEAF). The fabric was built and APIC/Switch firmware have been upgraded to 4.2(6d). Now we decided to rebuilt the fabric to use a different infra VLAN.

Following instruction from link below, the APICs and leaf/spine switches have been reset and then APICs were  reconfigured through initialization with the same information except new Infra VLAN.  

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/troubleshooting/b_APIC_Troubleshooting/b_APIC_Troubleshooting_chapter_01001.html

However the APIC01 just can not discover the leaf01 and rest of the fabric. The CLI LLDP Neighbour shows the proper linkage on both APIC01 and LEAF01 though.

Following instruction below, the leaf01 shows "infra-vlan-mismatch" on links to APIC...Then we reset the switches again but APIC still have issue discover the switches...

https://community.cisco.com/t5/data-center-documents/guide-by-tac-fabric-discovery-troubleshooting-discovering-the/ta-p/3316291 

Suggestions?

[Robert Burns]

Most likely, the Leaf wasn't correctly wiped.  This could be a timing thing, where inbetween the Leaf & APIC wipe, it re-pulled the config back.  You can verify LLDP info on the APIC for what it's sending vs. receiving from the connected Leaf.

apic1# acidiag run lldptool out eth2-1 (will show what APIC is sending)
apic1# acidiag run lldptool in eth2-1 (will show what APIC is receiving from Leaf)

Repeat above for both interfaces, eth2-1 and eth2-2 and you'll find the discrepancy.  Likely just need to re-run the "setup-clean-config.sh" script on the Leaf, but making sure all other switches (especially spines) are also wiped and in discovery-pending state.

Robert

[m1xed0s]

I was thinking about the order of operation...So I disconnected the infrastructure links from all APICs and then ran the "setup-clean-config.sh" and reboot of leaf/spine switches...Once the switches rebooted, I reconnected the infrastructure links on APIC but same result...

Thanks for the LLDP commands. From the "acidiag run lldptool in eth2-1", the leaf01 is indeed sending the old infrastructure VLAN while APIC is sending the new infra vlan...

I will have to execute the reset one more time once my tech is back onsite...Any further suggestion on the sequence?

[Robert Burns · Accepted solution]

Here's my order:
Wipe all APICs (acidiag touch clean, acidiag touch setup), let them reboot into the setup dialogue and leave
Wipe all Spines (setup-clean-config.sh, acidiag reboot) and leave
Wipe all Leafs (setup-clean-config.sh, acidiag reboot) ) and leave
Start with APIC1, then discover all switches followed by APIC2 & 3.

Robert

[m1xed0s]

Thanks! But that was pretty much what I did. Anyway will try it again with more patience then. Would you also recommend to disconnect the infrastructure links between APICs and Leafs before start?

[Robert Burns]

Nah no need.  I never have.  If you want to be extra cautious, issue a "sync" command after you wipe the switches.  this really only applies to dual sup Spines, but I've always done it out of habit.  Also after wiping my APICs,  I bounce all my switches at the same time (console1 > acidiag reboot, console2 > acidiag reboot, etc boom boom boom like that.

Robert

[RedNectar]

Did you look at this old post?

Works every time when resetting the fabric.

[Robert Burns]

"60% of the time, it works every time..."