APIC Cannot decomission or remove a replaced switch from the controller

[Alexander Pickar]

Hi Community!

I have replaced a failed Leaf switch in an ACI Fabric with a new one.

- The old switch still exists in the Inventory without having a Node ID, name or IP assigned.

- The only visible thing in the Fabric Membership is the serial number, Model and Role (Leaf).

- When trying to decomission the switch from the APIC inventory or use the "Remove From Controller" option, I end up with an error "RsDecommissionNode relation type only supports spines and leafs".

Any idea, how to remove the switch from the inventory?

Regards,

Alexander Pickar

[nvermand]

Hi,

you can try to reset the leaf for discovery and try the process again.
login on the leaf and then:
-acidiag touch clean
-acidiag touch setup

Reload the leaf then try the decomission process + Remove from controller again

Nic

[Alexander Pickar]

Hi Nic,

I wish I could connect to the switch, wipe out the config, do the decomissioning again, whatever. But the switch was replaced due to RMA (two fan trays were faulty) and is on it's way back who knows where (Where do all the RMA switches go, to the RMA heaven?).

Since we only have the replaced switch connected to the ACI fabric, your suggested procedure unfortunatelly cannot be done.

Any other idea anyone?

Regards,

Alexander

[Jayesh Singh]

Hi Alex,

What version are you running?

Also, by any chance are you seeing F2547 fault in your fabric?

Regards,

Jayesh

[Alexander Pickar]

Hi Jayesh,

• We are running APIC version 3.2(2o).
• On Thursday, we plan to upgrade to 3.2(3o).
• When I check the System / Faults, I see faults F2547 raised for nodes 201-204, which were removed and then assigned new node IDs 211-214.
• When I try to remove the replaced Leaf switch, the fault F2547 is raised again for the nodes 201-204.

Regards,

Alexander

[davirobb]

There are a couple of ways to delete this through the APIC CLI.  Do you have root access?

[Alexander Pickar]

Hi,

If you're asking about the Admin rights, then the answer is Yes, I do have a full access to the fabric.

If you're asking about Root access, I thought, that this is only possible with Cisco TAC involvement and that one I did not try yet.

I have tried to  google, what CLI command I could use and logged into the APIC CLI as well, but did not find any relevant commands yet. Which means, that I'll be more than glad if you can point me to some correct method of removing the "zombie switch".

Regards,

Alexander

[Jayesh Singh]

Hi Alex,

Can you please try below commands to clear the entries via APIC CLI,

apic1#conf t
apic1#decommission switch <node_id> remove-from-controller pod <pod_id>

E.g. apic1#decommission switch 102 remove-from-controller pod 1

Share the errors if you see any after executing that command.

Regards,

Jayesh

[Alexander Pickar · Accepted solution]

Hi Jayesh,

Even though your commands did not work, you have pointed me to a right direction.

- The replaced switch entry had Node ID = 0

- When trying to do a "decommission switch 0 remove-from-controller pod 1", APIC has complained, that it supports only Node ID between 101 and 4000.

- So that I have assigned an unused Node ID number (300) to the zombie switch, gave it a silly name and suddently, the decomission process did work.

Thanks for your lucky hint and have a great day.

Alexander

[Jayesh Singh]

Oh great!!!

Good to know finally its resolved :-)

I was suspecting some error while decommissioning related to node id  that such node id doesn't exist or something.

Output of command acidiag fnvread from apic cli would have helped notice abnormal node id.

Good piece of knowledge about the fix though.

Have a wonderful day ahead!!!

Thanks,

Jayesh