TIP OF THE DAY: How to erase a LEAF \ SPINE configuration and restore to factory defaults

[Tomas de Leon]

Today's TIP OF THE DAY is to give an example of how to erase a LEAF \ SPINE configuration and restore node to factory defaults.  

Here are some of the frequently asked questions regarding this topic on the ACI Cisco Support Community:

• How do a erase the Leaf \ Spine configuration?
• How do I reset the Leaf \ Spine configuration to factory default?
• If I restore the APIC(s) back to factory defaults and re-run setup scripts, Do I need to restore Leaf \ Spine(s) to factory default also?

The process below should work in addressing the questions listed above:

If the APICs are restored to factory defaults and the setup script is run again, a "New" Fabric is established.  All devices in the "OLD" fabric will need to go thru "discovery" again.  In order to do this, all of the nodes (leaf\spine) will also need to be reset back to factory defaults also.  Once the existing configuration is erased and the node reboots, the node will go thru the discovery process.  Once the node(s) is added to the "New" fabric, the APIC(s) will push the policy configurations to the registered node(s).

Procedure for erasing Node configuration and restoring to factory defaults:

1. Use SSH to access the NODE ( leaf\spine ) as user "admin".  If SSH is not applicable, access the NODE via the CONSOLE port and login as user "admin"

2. Verify that bootflash on the node contains an ACI Nexus 9000 firmware image to be run on the leaf or spine.  If bootflash does not contain or have the correct ACI Nexus 9000 firmware image, make sure to add the ACI Nexus 9000 firmware image to bootflash before proceeding.

leaf# dir /bootflash/aci-n9000*

/bootflash/aci-n9000-dk9.11.0.1b.bin
/bootflash/aci-n9000-dk9.11.0.1c.bin
/bootflash/aci-n9000-dk9.11.0.1d.bin

3. Check and verify the current settings for the boot variables for this NODE

cat /mnt/cfg/0/boot/grub/menu.lst.local
cat /mnt/cfg/1/boot/grub/menu.lst.local

leaf# cat /mnt/cfg/0/boot/grub/menu.lst.local
#
# General configuration
#
disable certificate
title bootflash:aci-n9000-dk9.11.0.1c.bin
boot bootflash:aci-n9000-dk9.11.0.1c.bin

leaf# cat /mnt/cfg/1/boot/grub/menu.lst.local
#
# General configuration
#
disable certificate
title bootflash:aci-n9000-dk9.11.0.1c.bin
boot bootflash:aci-n9000-dk9.11.0.1c.bin

Note: the current boot variables are set to "aci-n9000-dk9.11.0.1c.bin" but the intended version or desired version for this node is "aci-n9000-dk9.11.0.1d.bin"

4. Set the Node's boot variables to the intended firmware version or desired firmware version for this node. The firmware must be in /bootflash. Use the "setup-bootvars.sh <firmware image>" command.

For Example: "aci-n9000-dk9.11.0.1d.bin"

leaf# setup-bootvars.sh aci-n9000-dk9.11.0.1d.bin
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
Done

leaf# cat /mnt/cfg/0/boot/grub/menu.lst.local
boot aci-n9000-dk9.11.0.1d.bin

leaf# cat /mnt/cfg/1/boot/grub/menu.lst.local
boot aci-n9000-dk9.11.0.1d.bin

5. Once the Node's boot variables are set and verified, you can reset and restore the node's configuration back to factory defaults.  Use the "setup-clean-config.sh <firmware image>" command.

For Example: "aci-n9000-dk9.11.0.1d.bin"

leaf# setup-clean-config.sh aci-n9000-dk9.11.0.1d.bin
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
Done

leaf# reload
This command will reload the chassis, Proceed (y/n)? [n]: y

From the Console of the Node, you will see a lot of display output but look for things like:

Version 2.16.1240. Copyright (C) 2013 American Megatrends, Inc.
Board type  2
CISCO TOR
IOFPGA @ 0xc8000000
SLOT_ID @ 0xf
 Filesystem type is ext2fs, partition type 0x83
Trying to read config file /boot/grub/menu.lst.local from (hd0,4)
 Filesystem type is ext2fs, partition type 0x83

Booting aci-n9000-dk9.11.0.1d.bin...
Booting aci-n9000-dk9.11.0.1d.bin
Trying diskboot
 Filesystem type is ext2fs, partition type 0x83
Image valid
..
..
..
..
[  122.087592] t2usd_tor (3947) Ran 4899 msecs in last 5012 msecs

User Access Verification
(none) login: admin
********************************************************************************
     Fabric discovery in progress, show commands are not fully functional
     Logout and Login after discovery to continue to use show commands.
********************************************************************************

(none)# show version
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac

Software
  BIOS:      version 07.15
  kickstart: version 11.0(1d) [build 11.0(1d)]
  system:    version 11.0(1d) [build 11.0(1d)]
  BIOS compile time:       06/29/2014
  kickstart image file is: /bootflash/aci-n9000-dk9.11.0.1d.bin
 

6.  Use the APIC Admin GUI to add the Node to the ACI Fabric. 

Once the node is added to the ACI fabric, you should see something like this on the Node Console:

(none)#
Broadcast message from root@leaf (Tue Oct 14 13:44:43 2014):

This switch is now part of the ACI fabric. Please re-login with the right credentials.

leaf#

[michelvankessel]

is it correct to say that it doesn't work via SSH? I seems to work only via Console. Can you confirm that?

many thanks

Michel van Kessel

[Tomas de Leon]

Michel,

This does work via SSH up to the current release of ACI firmware.  What I mean by this is that you can initiate the RESTORE to DEFAULT via SSH.  What you can NOT do is troubleshoot the "Discovery" process unless you are on the CONSOLE.

So it is best practice to have access (physically or remotely) to console along with SSH access for management purposes.

Note: the process may change in the next major release and I will update note after it is release.

I hope this helps!

T.

[jgnoel]

Tomas, I ssh'ed from APIC GUI to each of the leafs and spines and followed your procedure.  I then simply powered cycled the boxes.  Worked great.  Thanks!  Jim

[iandjelk]

I noticed that after this procedure to reset APIC and nodes, the mgmt ip address of the nodes is preserved.

Details: None of my APICs is configured but I can still reach leaves/spines at the old mgmt ip address, just login credentials do not work anymore. Version 1.1(o)

Is that normal behavior?

[mtimm]

Yes this is normal, it allows you to erase the ACI policy config but continue to access the node via ssh.  The oob mgmt address is stored in cmos and can be set/cleared from loader or a utility called cmosio when the switch is running the ACI image when the node is not part of the fabric.  The oob mgmt address can also be set via policy on the APIC once the node is part of the fabric.

Mike

[Richard Hornbaker]

Hi, Tomas.

Thanks for this article!  I'm resetting a lab fabric (new name, different connectivity model) and seeing a curious effect on the APICs - they think their 10Gb links are down, but the leaf shows them as up.  So, not getting far on fabric discovery. ;-)

Maybe this is a side-effect of resetting the APICs before we did the switches (i.e., APIC thinks its adjacent leaf is already in a different fabric and disabled its interface)?  More work planned for tomorrow, but I'm curious if there's a recommended sequence for resetting devices. (Software is 11.1(2h), if that's important.)

Cheers,

Richard

[Tomas de Leon]

Richard,

As long as you have console access to the leaf and spine switches, Sequence shouldn't matter.  If you choose to reset the APICs first that is fine.  One suggestion is to reset all APICs via the CIMC at the same time.  Reconfigure ONLY APIC1 first via the setup script.  After you have reset APIC2 & APIC3, do not configure yet.  Simply POWER OFF the APIC2 & APIC3.  Then reset the switches and then reboot.  The Leaf node attached to APIC1 should show up in the fabric membership inventory.  Configure the Node ID and Node Name.  Once it is discovered, the Spines and rest of the Leaf nodes should start appearing.  Configure the appropriate Node IDs and Node Names.  Once the entire fabric is discovered and up, power on APIC2.  Configure APIC2 via the setup script.  Once APIC2 joins the Fabric and the cluster is "fully fit".  Power on APIC3 and repeat the configuration steps.   APIC3 should then join the Fabric and the cluster should become fully fit.

Just let us know how it goes!

T.

[Richard Hornbaker]

Hi, Tomas.

Occam's Razor... the problem was much simpler than suspected, of course.

When the gear was moved, connectivity changed to fiber (from twinax) and the crew inadvertently used FET-10G SFPs.  Of course, those are supported on the switch (for FEXs), but not on servers (APICs), causing the odd link-light behavior.  Changing to SFP-10G-SR optics cleared the problem and fabric discovery was as simple as it should have been.

For convenience, here's a cross-link to your other reply with steps for resetting the APICs to factory default.

Cheers,

Richard

[yashartp@gmail.com]

Hi Mate,

We have a couple of ACI spine and leaf switches (nexus 9300 and 9500 series) in our environment and the business changed its mind and they don't want to go with ACI anymore. Now we want to rebuild them as normal NXOS switches and use them in an ordinary core/distribution/access structure, is it doable or there is limitation in doing so? is it doable for both spine and leaf switches?