06-23-2015 03:51 AM - edited 03-01-2019 04:50 AM
Hi,
My old firmware is n9000-11.0(2j) and apic-1.0(2j).
I have done upgrade firmware apic-1.1(1j) on controller and n9000-11.1(1j) on 2 Leafs.
But when i upgrade firmware n9000-11.1(1j) on 2 Spines, it returns fail. I got some picture for faults log ( attachment ).
Can someone help me to find more detail this failed state :(.
Thanks.
Solved! Go to Solution.
06-23-2015 04:10 AM
Hi,
Here are the release notes for 1.1(1j): http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/release/notes/apic_rn_111.html
You'll notice that the supported upgrade path would involve upgrading from 1.0(2x) to 1.0(4x) before going to 1.1(1j). Have you tried upgrading the spines to 1.0(4x) code? Were the APIC upgrades to 1.1(1j) successful?
In regard to the fault, it looks like the issue may be with the configuration for the firmware and maintenance groups and not the upgrade itself. Can you double-check that you have a target firmware set-up under the firmware groups?
06-23-2015 07:57 PM
Ok.
A couple of things that I would like you perform.
1. Access console port of 9508 or ssh to spine. Verify the MD5 checksum for aci-n9000-dk9.11.1.1j.bin matches the following:
aci-spine# md5sum aci-n9000-dk9.11.1.1j.bin
7d3b141d8a39675bc088f531796ddaf2 aci-n9000-dk9.11.1.1j.bin
If the checksum matches, proceed to step 2. If different downland the image again from
https://software.cisco.com/download/release.html?mdfid=285956854&flowid=47182&softwareid=286278844&release=11.0(2m)&relind=AVAILABLE&rellifecycle=&reltype=latest
2. Download aci-n9000-dk9.11.0.4h.bin
https://software.cisco.com/download/release.html?mdfid=285956854&flowid=47182&softwareid=286278844&release=11.0(2m)&relind=AVAILABLE&rellifecycle=&reltype=latest
verify md5checksum =
6abe77ae7a92a5695e77e86edd8f08e6
3. Cleanup bootflash.
cd/bootflash
rm aci-n9000-dk9.11.0.2j.bin
rm aci-n9000-dk9.11.0.4h.bin-isan
rm aci-n9000-dk9.11.1.1j.bin-isan
rm auto-s
rm disk_log.txt
rm mem*
4. SCP or SFTP aci-n9000-dk9.11.0.4h.bin to the spine and put in bootflash. You should have 2 images: aci-n9000-dk9.11.0.4h.bin, aci-n9000-dk9.11.1.1j.bin
5. Run the following CLI commands after both images are in bootflash:
setup-bootvars.sh aci-n9000-dk9.11.0.4h.bin
setup-clean-config.sh aci-n9000-dk9.11.0.4h.bin
reboot switch
6. the switch should reload with version aci-n9000-dk9.11.0.4h.bin. Verify this is indeed the case. The reason that we are loading 11.0.4h first is for two reasons: upgrading to 11.1.1j should be performed for an existing version of 11.0.4h. Secondly, a two stage install is necessary to upgrade bios & epld fw versions if needed. Normally performed via policy uprade but this is a manual process at this point.
7. once you verified the spine is running aci-n9000-dk9.11.0.4h.bin and joined the fabric, you can then perform a policy upgrade to aci-n9000-dk9.11.1.1j.bin. remove aci-n9000-dk9.11.1.1j.bin from bootflash and then perform a policy upgrade from the GUI to 11.1.1j. During the upgrade the spine will upgrade bios and EPLDs.
8. If the first seven steps worked on first spine, repeat steps on Spine2.
06-23-2015 04:10 AM
Hi,
Here are the release notes for 1.1(1j): http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/release/notes/apic_rn_111.html
You'll notice that the supported upgrade path would involve upgrading from 1.0(2x) to 1.0(4x) before going to 1.1(1j). Have you tried upgrading the spines to 1.0(4x) code? Were the APIC upgrades to 1.1(1j) successful?
In regard to the fault, it looks like the issue may be with the configuration for the firmware and maintenance groups and not the upgrade itself. Can you double-check that you have a target firmware set-up under the firmware groups?
06-23-2015 04:26 AM
06-23-2015 04:38 AM
Hello
Just out of curiosity,
What type of spines do you have?
06-23-2015 05:32 AM
Hi Dpita,
It is N9K-C9508.
06-23-2015 05:44 AM
do you have both supervisors installed and powered on?
if so, one supervisor may have been upgraded and when rebooted, came on the warm standby which was not upgraded.
If you have external access to fabric, private message me the access information with login credentials and I can look at your spines and see what is going on.
Cheers!
T.
06-23-2015 06:24 PM
Hi Tomas De Leon,
I have some information from those commands:
-rwx------ 1 rad admin 293 Apr 22 08:00 MDS201504201837188540.lic -rw-r--r-- 1 root root 500237761 Jan 6 03:45 aci-n9000-dk9.11.0.2j.bin -rw-rw-rw- 1 root root 254275072 Jun 23 12:39 aci-n9000-dk9.11.0.4h.bin-isan -rw-rw-rw- 1 root root 591631164 Jun 23 11:14 aci-n9000-dk9.11.1.1j.bin -rw-rw-rw- 1 root root 396732732 Jun 23 11:14 aci-n9000-dk9.11.1.1j.bin-isan -rw-r--r-- 1 root root 328541633 Mar 13 04:27 auto-s -rw-r--r-- 1 root root 54 Jun 24 02:20 disk_log.txt -rw-r--r-- 1 root root 7750390 Jun 24 02:20 mem_log.txt -rw-r--r-- 1 root root 2474 Jan 6 03:50 mem_log.txt.old.gz |
Redundancy mode This supervisor (sup-27) Other supervisor (sup-28) |
Total Objects shown: 1 # firmware.Running |
Thanks!
06-23-2015 07:57 PM
Ok.
A couple of things that I would like you perform.
1. Access console port of 9508 or ssh to spine. Verify the MD5 checksum for aci-n9000-dk9.11.1.1j.bin matches the following:
aci-spine# md5sum aci-n9000-dk9.11.1.1j.bin
7d3b141d8a39675bc088f531796ddaf2 aci-n9000-dk9.11.1.1j.bin
If the checksum matches, proceed to step 2. If different downland the image again from
https://software.cisco.com/download/release.html?mdfid=285956854&flowid=47182&softwareid=286278844&release=11.0(2m)&relind=AVAILABLE&rellifecycle=&reltype=latest
2. Download aci-n9000-dk9.11.0.4h.bin
https://software.cisco.com/download/release.html?mdfid=285956854&flowid=47182&softwareid=286278844&release=11.0(2m)&relind=AVAILABLE&rellifecycle=&reltype=latest
verify md5checksum =
6abe77ae7a92a5695e77e86edd8f08e6
3. Cleanup bootflash.
cd/bootflash
rm aci-n9000-dk9.11.0.2j.bin
rm aci-n9000-dk9.11.0.4h.bin-isan
rm aci-n9000-dk9.11.1.1j.bin-isan
rm auto-s
rm disk_log.txt
rm mem*
4. SCP or SFTP aci-n9000-dk9.11.0.4h.bin to the spine and put in bootflash. You should have 2 images: aci-n9000-dk9.11.0.4h.bin, aci-n9000-dk9.11.1.1j.bin
5. Run the following CLI commands after both images are in bootflash:
setup-bootvars.sh aci-n9000-dk9.11.0.4h.bin
setup-clean-config.sh aci-n9000-dk9.11.0.4h.bin
reboot switch
6. the switch should reload with version aci-n9000-dk9.11.0.4h.bin. Verify this is indeed the case. The reason that we are loading 11.0.4h first is for two reasons: upgrading to 11.1.1j should be performed for an existing version of 11.0.4h. Secondly, a two stage install is necessary to upgrade bios & epld fw versions if needed. Normally performed via policy uprade but this is a manual process at this point.
7. once you verified the spine is running aci-n9000-dk9.11.0.4h.bin and joined the fabric, you can then perform a policy upgrade to aci-n9000-dk9.11.1.1j.bin. remove aci-n9000-dk9.11.1.1j.bin from bootflash and then perform a policy upgrade from the GUI to 11.1.1j. During the upgrade the spine will upgrade bios and EPLDs.
8. If the first seven steps worked on first spine, repeat steps on Spine2.
06-23-2015 08:25 PM
Hi Tomas De Leon,
Many thanks for your support.
I have successfully upgraded firmware on 2 Spines without firmware 1.0(4x).
I reboot Spine2 and upgrade firmware with normal way : using command line on APIC "firmware upgrade switch 11.1(1j).bin nodes 104" .
On Spine1 , I using :
setup-bootvars.sh aci-n9000-dk9.11.1.1j.bin
setup-clean-config.sh aci-n9000-dk9.11.1.1j.bin
reload
Spine1 is on firmware 11.1(1j) too. But it has some faults, i attached a picture to show that.
06-23-2015 08:31 PM
Correct.
That is why i wanted you to upgrade to 4h first. Your EPLD & Bios did not get updated. The "Policy" upgrade will perform this.
Simply repeat the steps but for 4h not 1j.
After 4h is running, perform a "policy" upgrade using the CLI or the GUI to 1j. This should resolve the mismatch.
If it does not, you may need to remove power cables from switch for at least 60 seconds. Then power back on to see if the faults go away.
T.
06-24-2015 02:07 AM
Many thanks again :)
All switchs and controllers are on firmware 1.1(1j).
06-24-2015 03:52 AM
Are all the Faults for Mismatches resolved?
Please summarize the what you did for others to reference in future.
I am glad things are resolved.
T.
06-23-2015 05:53 AM
Also, please provide the output of the following for each spine:
Thanks
T.
06-23-2015 05:42 AM
vuth60001,
You can attempt to upgrade these manually if you would like. Here are the steps to do so:
Open up the CLI of one of the spines (SSH or console)
Copy the firmware image from one of the APICs to the bootflash for the spine using SCP (assuming connectivity is working between the two)
Run "setup-bootvars.sh <image-name>"
You can check that the boot variable change was changed correctly by running "cat /mnt/cfg/0/boot/grub/menu.lst.local" which should show the image name you just selected.
Reload the switch.
Issue a "show version" to verify that the switch booted into the desired image.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide