ACI spine interface disabled

[Harish Chopra]

Hi,

I accidentally disabled my 2 interfaces of ACI spine during some testing and due to which the Spine is showing up unreachable. I have lost access on the spine now. I have access to the APIC controller.

can anyone please guide me how to bring that interface back up so that it starts showing up again in the Pod1.

Regards

Harish

[Tomas de Leon]

Harish,

Hopefully you have access to the CONSOLE port of the SPINE in question.  Either by a Terminal Server connection or physical access.  If you can reset the configuration on the SPINE so that it has to RE-REGISTER with the Fabric and then it will reconfigure itself.

Assuming Console access is in place:

From the APIC GUI ->

1. Decommission the SPINE in question
2. Once the SPINE is decommissioned, DELETE and REMOVE FROM CONTROLLER
   After this is completed the SPINE is no longer a known object and is removed from the database.

From the CONSOLE of the SPINE ->

1. LOGIN to the SPINE as ADMIN user
2. cd /bootflash
3. ls aci*
4. Determine which firmware version that you are running on spine  
   (ie. aci-n9000-dk9.11.0.2m.bin )
5. Reset configuration along with setting the BOOTVARS using the following commands:

setup-bootvars.sh <firmware image>
setup-clean-config.sh <firmware image>

For example:

setup-bootvars.sh aci-n9000-dk9.11.0.2m.bin
setup-clean-config.sh aci-n9000-dk9.11.0.2m.bin

Confirm with "Y or y" for yes and when the commands have completely configured. Reboot\reload the SPINE switch.

At this time the SPINE **SHOULD** reload and come back up into "discovery" mode.

From the APIC GUI ->

1. Return to FABRIC > FABRIC MEMBERSHIP
2. When you see the SPINE assign the NODE ID and NODE NAME, and the SPINE should register with Fabric and can be configured again.
3. Once SPINE re-registers, verify access to all ports.

If this does not work, you should open a Cisco TAC Case so that we can get access to resolve your issues.  Thank you for using the Cisco Support Communities and I hope this information helps!

Cheers!

T.

Reference Material:

TIP OF THE DAY: How to erase a LEAF \ SPINE configuration and restore to factory defaults
https://supportforums.cisco.com/document/12325246/tip-day-how-erase-leaf-spine-configuration-and-restore-factory-defaults

[Harish Chopra]

Thanks Tomas,

The problem is - I don't have console access and physical access. What my thought was - what if we decommission the spine totally and then under the Fabric Membership - commission it again. Will that help.

I am not sure, if the chassis will come back up it keep the interface "Admin UP" or by default it's shutdown.

Otherwise the option of erasing config and commissioning it as new is only remained which you have suggested.

your thought ?

Regards

Harish

[Tomas de Leon]

Harish,

From your original comments, "I accidentally disabled my 2 interfaces of ACI spine during some testing"

From this I am making an assumption that you shutdown the FABRIC interfaces which is used by the APICs.  So the APICs will NOT be able to configure the SPINES.

Please provide the output of the command "acidiag fnvread".  Perform on the CLI of APIC.  SSH to APIC IP address.

If you have OOB management access configured on APIC & SPINE, you SHOULD be able to SSH to SPINE and perform the ABOVE TASKS to reset the SPINE.

Do you have OOB management addresses setup on your switch NODES?

T.

[mtimm]

Do you have out-of-band access to the spine?  If so you can use a local task to bring the interface back up.  The command on the spine to do so is 'setinservice' but there are issues with this command in all shipping versions of code that prevent it from working.  Regardless the actions the command on the spine take can be accomplished with the REST API and the Python SDK.  This gist gives an example of bringing an interface backup via the Python SDK.

https://gist.github.com/mtimm/d6998c755c5235a2dcde

Mike

Edit:  To be clear, you still need to remove the policy from the APIC then bring it up on the fabric node, otherwise the APIC will just shut it back down.