08-16-2022 09:44 AM - edited 08-16-2022 09:55 AM
Hello all,
We recently upgraded the CIMC version on one of our APIC's (APIC1) to 4.1(2k), and afterwards APIC1 has fallen out of the cluster. I do have local admin access to APIC1, and running avread on APIC1 I can see that APIC1 only sees itself as active and APIC2 and APIC3 as inactive , but running avread on APIC2 and APIC3 shows that APIC2 and 3 are both active and APIC1 is the inactive one, which matches reality.
APIC1 is able to ping APIC2 and APIC3.
After the CIMC upgrade on APIC1, the CIMC had a different MAC address than before - thought that was interesting.
APIC1 didn't get a new ChassisID when it rebooted after the CIMC upgrade, so I know it's not that. I would like to avoid having to rebuild APIC1 via .iso file on USB or IIS over SoL as that takes a while.
Is there a set of commands I can run on APIC1 that will wipe only APIC1's config? I only want to clear/reset APIC1 so I can rejoin it to the otherwise healthy cluster of APIC2 and APIC3.
Thanks!
Solved! Go to Solution.
08-16-2022 11:24 AM - edited 08-16-2022 11:27 AM
@Robert Burns Thankfully I was able to get this one resolved, but to answer your question here is the output you requested. Should LLDP be enabled, or no?
apic1-cimc /chassis/adapter # show detail | grep LLDP
LLDP: Disabled
So this one actually ended up resolving itself as the cluster finally became Fully Fit but it took a while (3 hours). I'm not used to an APIC disconnecting from the cluster after a CIMC upgrade. : )
And in terms of my original question in this post, I was actually able to solve this one on my own. I ended up reading through @RedNectar 's reply to a forum post about how to wipe an entire fabric, and I inferred from Steps 2 and 3 that for my scenario I can just run the acidiag touch clean and acidiag touch setup commands on the single APIC in question (APIC1), which would only wipe the local APIC config.
So I used our LAB instance of ACI to test this, and I logged in to LAB-APIC2 and from there I decommissioned LAB-APIC1 from the cluster, and then I ran those 2 commands locally on LAB-APIC1 to wipe its config and then used acidiag reboot to reload it. After it came up I went through the standard APIC Setup and entered the initial config parameters as taken from LAB-APIC2 using this awesome guide I had saved that was posted by @Tomas de Leon . Then all I had to do was use LAB-APIC2 to Commission LAB-APIC1 back into the cluster and everything was just as happy as ever. : )
But thankfully I didn't end up needing to do this in production as the cluster finally became Fully Fit after 3 hours.
08-16-2022 11:40 AM
Yes, LLDP should show disabled. This is so the APIC's OS can consume the LLDP packets. The reason I had you check this is because you mentioned an IMC upgrade, and I have seen in cases where some of the IMC settings get reset during upgrades. Didn't want you to have to run through a wipe/erase but since it was lab not a big deal. Glad you got it resolved!
Robert
08-16-2022 10:24 AM
Log into the IMC of APIC and issue the following commands:
APIC1-IMC# scope chassis
APIC1-IMC /chassis # scope adapter 1
APIC1-IMC /chassis/adapter # show detail | inc LLDP
Paste the output here.
Robert
08-16-2022 11:24 AM - edited 08-16-2022 11:27 AM
@Robert Burns Thankfully I was able to get this one resolved, but to answer your question here is the output you requested. Should LLDP be enabled, or no?
apic1-cimc /chassis/adapter # show detail | grep LLDP
LLDP: Disabled
So this one actually ended up resolving itself as the cluster finally became Fully Fit but it took a while (3 hours). I'm not used to an APIC disconnecting from the cluster after a CIMC upgrade. : )
And in terms of my original question in this post, I was actually able to solve this one on my own. I ended up reading through @RedNectar 's reply to a forum post about how to wipe an entire fabric, and I inferred from Steps 2 and 3 that for my scenario I can just run the acidiag touch clean and acidiag touch setup commands on the single APIC in question (APIC1), which would only wipe the local APIC config.
So I used our LAB instance of ACI to test this, and I logged in to LAB-APIC2 and from there I decommissioned LAB-APIC1 from the cluster, and then I ran those 2 commands locally on LAB-APIC1 to wipe its config and then used acidiag reboot to reload it. After it came up I went through the standard APIC Setup and entered the initial config parameters as taken from LAB-APIC2 using this awesome guide I had saved that was posted by @Tomas de Leon . Then all I had to do was use LAB-APIC2 to Commission LAB-APIC1 back into the cluster and everything was just as happy as ever. : )
But thankfully I didn't end up needing to do this in production as the cluster finally became Fully Fit after 3 hours.
08-16-2022 11:40 AM
Yes, LLDP should show disabled. This is so the APIC's OS can consume the LLDP packets. The reason I had you check this is because you mentioned an IMC upgrade, and I have seen in cases where some of the IMC settings get reset during upgrades. Didn't want you to have to run through a wipe/erase but since it was lab not a big deal. Glad you got it resolved!
Robert
08-16-2022 11:54 AM
Great! I'll add that info about LLDP to my notes as well. Thanks again!
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: