ACI Upgrade ISSUE

[JlassiAhmed0345]

Hi community

yesterday I tried to upgrade my  ACI fabric with a current version 2.1(1h)  to the recommended version 4.2(7f), so to achieve the upgrade I need to pass by an intermediate version which is the 2.2(2d).

the upgrade from version 2.1(1h) => 2.2(2d) is worked properly and all the fabric is fine. but when I tried to complete the upgrade to my target version 4.2(7f), I have faced an issue with switches of the fabric, I find that only one leaf is upgraded properly to version 14.2.7f but the other switches (LEAF2, SPINE1-2) are failed the upgrade and they stuck in the loader mode, even I tried to boot the switches manually using the CLI on the correct version with no way.

this some hardware pieces of information related to my fabric :

Two Nexus 9336PQ Switches Spines, Two Nexus 9396PX Switches Leafs,Two 2232TM-E Fabric Extenders

and this the upgrade path to the target version:

ACI version Path Upgrade duration
1 2.1(1h) → 2.2(4p) 
2 2.2(4p) → 4.2(7f) 

Target version of Switches:

Target version	supported spine hardware	supported leaf hardware	Actual device support
12.2(4)	N9K-C9336PQ	N9K-C9396PX	yes
14.2(7)	N9K-C9336PQ	N9K-C9396PX	yes

below you will find attached an output terminal of the upgrade of one switch to version 14.2(7).

is there any solution to fix this issue? 

thanks.

[Robert Burns]

For one of the switches in loader, which images show in the bootflash?

Once you know the file images available, try to boot one.

Ref: https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/6-x/troubleshooting/guide/b_Cisco_Nexus_9000_Series_NX-OS_Troubleshooting_Guide/b_Cisco_Standalone_Series_NX-OS_Troubleshooting_Guide_chapter_01.html

Robert

[JlassiAhmed0345]

Hi , in the output of show bootflash: command I see the target image is saved .

[ecsnnsls]

I don't have an answer to this question. But can you let us know the following?

Did you raise a TAC case and is this a production or lab setup?

From the logs I see that when you re-tried, the same file system consistency check script runs and then goes back to loader prompt. What happened during the last run when you tried to boot a lower version? 

[JlassiAhmed0345]

Hi,

I didn't raise a TAC case for this issue, I tried to boot to a lower version which the intermediate version but I faced the problem, the switch cannot load it.

[ecsnnsls]

While skimming through the logs, I also noticed that at one point you were able to get the normal switch prompt. What happened after that? Does it reload automatically?

I see some memory allocation related errors after the login. From the looks of it, I think this might be a bug, it is better you create a TAC case and follow up with them. Let us know what happens.

Cheers.

[Robert Burns]

When you're upgrading from such an older image to a new image, I wouldn't attempt downgrading to the older 2.x version.  Can you try the following:

1. Let the swtich fail into loader or break into it with ctrl+c while device reload
2. loader> boot aci-n9000-dk9.14.2.7f.bin
3. after boot, login as admin (quickly before it cores)
4. (none)# /bin/prepare-mfg.sh aci-n9000-dk9.14.2.7f.bin
5. (none)# vsh
6. Switch# reload

Note: typically I'd recommend you load a fresh copy of the image to a USB and try to boot the Switch off the USB instead of doing a local boot in Step 2.  If you have access to the switch on site, I'd suggest trying this first. The USB device would be usb1 or usb2 depending which slot you use. 

Robert

[vantipov]

Having similar problem where I was upgrading code on N9K-C9336PQ spines from 14.2.2f to 14.2.7f via APIC GUI.  I scheduled the upgrade with "pause on failure".  First spine failed to upgrade and went into loader prompt.  Now I cannot get it to boot to any image old or new.  Copied aci-n9000-dk9.14.2.2f.bin to USB and it reads it, but fails to boot.  Same with new code.  Also attempted tftpboot.  No luck.  Running "tcpdump net 10.100.251.0/24" on my TFTP server and I don't even see any attempts from that switch.

loader > boot tftp://10.1.1.10/aci-n9000-dk9.14.2.2f.bin

Security Lock
Booting tftp://10.1.1.10
Trying netboot
Filename:/n9000-dk9.14.2.2f.bin
Address: 10.100.251.20
Netmask: 255.255.255.0
Server: 10.1.1.10
Gateway: 10.100.251.1
Filesystem type is tftp, using whole disk
Boot failed

Error 9: Unknown boot failure

Also attempted running "cmdline init_system clear_config", system acted like it re-formated the disks, but still cannot boot from USB1 or TFTP.