yesterday I tried to upgrade my ACI fabric with a current version 2.1(1h) to the recommended version 4.2(7f), so to achieve the upgrade I need to pass by an intermediate version which is the 2.2(2d).
the upgrade from version 2.1(1h) => 2.2(2d) is worked properly and all the fabric is fine. but when I tried to complete the upgrade to my target version 4.2(7f), I have faced an issue with switches of the fabric, I find that only one leaf is upgraded properly to version 14.2.7f but the other switches (LEAF2, SPINE1-2) are failed the upgrade and they stuck in the loader mode, even I tried to boot the switches manually using the CLI on the correct version with no way.
this some hardware pieces of information related to my fabric :
Two Nexus 9336PQ Switches Spines, Two Nexus 9396PX Switches Leafs,Two 2232TM-E Fabric Extenders
and this the upgrade path to the target version:
ACI version Path Upgrade duration
1 2.1(1h) → 2.2(4p)
2 2.2(4p) → 4.2(7f)
Target version of Switches:
|Target version||supported spine hardware||supported leaf hardware||Actual|
below you will find attached an output terminal of the upgrade of one switch to version 14.2(7).
is there any solution to fix this issue?
For one of the switches in loader, which images show in the bootflash?
Once you know the file images available, try to boot one.
I don't have an answer to this question. But can you let us know the following?
Did you raise a TAC case and is this a production or lab setup?
From the logs I see that when you re-tried, the same file system consistency check script runs and then goes back to loader prompt. What happened during the last run when you tried to boot a lower version?
I didn't raise a TAC case for this issue, I tried to boot to a lower version which the intermediate version but I faced the problem, the switch cannot load it.
While skimming through the logs, I also noticed that at one point you were able to get the normal switch prompt. What happened after that? Does it reload automatically?
I see some memory allocation related errors after the login. From the looks of it, I think this might be a bug, it is better you create a TAC case and follow up with them. Let us know what happens.
When you're upgrading from such an older image to a new image, I wouldn't attempt downgrading to the older 2.x version. Can you try the following:
1. Let the swtich fail into loader or break into it with ctrl+c while device reload
2. loader> boot aci-n9000-dk18.104.22.168f.bin
3. after boot, login as admin (quickly before it cores)
4. (none)# /bin/prepare-mfg.sh aci-n9000-dk22.214.171.124f.bin
5. (none)# vsh
6. Switch# reload
Note: typically I'd recommend you load a fresh copy of the image to a USB and try to boot the Switch off the USB instead of doing a local boot in Step 2. If you have access to the switch on site, I'd suggest trying this first. The USB device would be usb1 or usb2 depending which slot you use.
Having similar problem where I was upgrading code on N9K-C9336PQ spines from 14.2.2f to 14.2.7f via APIC GUI. I scheduled the upgrade with "pause on failure". First spine failed to upgrade and went into loader prompt. Now I cannot get it to boot to any image old or new. Copied aci-n9000-dk126.96.36.199f.bin to USB and it reads it, but fails to boot. Same with new code. Also attempted tftpboot. No luck. Running "tcpdump net 10.100.251.0/24" on my TFTP server and I don't even see any attempts from that switch.
loader > boot tftp://10.1.1.10/aci-n9000-dk188.8.131.52f.bin
Filesystem type is tftp, using whole disk
Error 9: Unknown boot failure
Also attempted running "cmdline init_system clear_config", system acted like it re-formated the disks, but still cannot boot from USB1 or TFTP.