Failed when upgrade firmware n9000-11.1(1j)

[vuth60001]

Hi,

My old firmware is n9000-11.0(2j) and apic-1.0(2j).

I have done upgrade firmware apic-1.1(1j) on controller and n9000-11.1(1j) on 2 Leafs.

But when i upgrade firmware n9000-11.1(1j) on 2 Spines, it returns fail.  I got some picture for faults log ( attachment ).

Can someone help me to find more detail this failed state :(.

Thanks.

[lpember · Accepted solution]

Hi,

Here are the release notes for 1.1(1j): http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/release/notes/apic_rn_111.html

You'll notice that the supported upgrade path would involve upgrading from 1.0(2x) to 1.0(4x) before going to 1.1(1j). Have you tried upgrading the spines to 1.0(4x) code? Were the APIC upgrades to 1.1(1j) successful?

In regard to the fault, it looks like the issue may be with the configuration for the firmware and maintenance groups and not the upgrade itself. Can you double-check that you have a target firmware set-up under the firmware groups?

[vuth60001]

Hi lpember,

All APIC and LEAF have upgraded firmware 1.1(1j). Each switch is in one firmware groups.

I have tried upgrade firmware 1.0(4x) first , but it fails too.

Some information in 2 picture.

Thanks for reply .

[dpita]

Hello

Just out of curiosity,

What type of spines do you have?

[vuth60001]

Hi Dpita,

It is N9K-C9508.

[Tomas de Leon]

do you have both supervisors installed and powered on?

if so, one supervisor may have been upgraded and when rebooted, came on the warm standby which was not upgraded.

If you have external access to fabric, private message me the access information with login credentials and I can look at your spines and see what is going on.

Cheers!

T.

[vuth60001]

Hi Tomas De Leon,

I have some information from those commands:

1. ls -al /bootflash

   -rwx------  1 rad  admin       293 Apr 22 08:00 MDS201504201837188540.lic
   -rw-r--r--  1 root root  500237761 Jan  6 03:45 aci-n9000-dk9.11.0.2j.bin
   -rw-rw-rw-  1 root root  254275072 Jun 23 12:39 aci-n9000-dk9.11.0.4h.bin-isan
   -rw-rw-rw-  1 root root  591631164 Jun 23 11:14 aci-n9000-dk9.11.1.1j.bin
   -rw-rw-rw-  1 root root  396732732 Jun 23 11:14 aci-n9000-dk9.11.1.1j.bin-isan
   -rw-r--r--  1 root root  328541633 Mar 13 04:27 auto-s
   -rw-r--r--  1 root root         54 Jun 24 02:20 disk_log.txt
   -rw-r--r--  1 root root    7750390 Jun 24 02:20 mem_log.txt
   -rw-r--r--  1 root root       2474 Jan  6 03:50 mem_log.txt.old.gz

2. show system redundancy status

   Redundancy mode
   ---------------
         administrative:   Warm
            operational:   Warm

   This supervisor (sup-27)
   -----------------------
       Redundancy state:   Active
       Supervisor state:   Active
         Internal state:   Active with warm standby

   Other supervisor (sup-28)
   ------------------------
       Redundancy state:   Standby
       Supervisor state:   Warm standby
         Internal state:   Warm standby

3. moquery -c firmwareRunning

   Total Objects shown: 1

   # firmware.Running
   biosTs         : 2014-09-10T07:00:00.000+07:00
   biosVer        : 08.06
   childAction    : 
   descr          : version 11.0(2j) [build 11.0(2j)]
   dn             : sys/fwstatuscont/running
   internalLabel  : 4755470886b309d39a3dcb8fd1e04db8a015f1f9
   ksFile         : bootflash:///aci-n9000-dk9.11.0.2j.bin
   modTs          : never
   mode           : normal
   peVer          : 1.0(2j)
   rn             : running
   status         : 
   sysFile        : bootflash:///auto-s
   ts             : 2014-11-12T01:08:58.000+07:00
   type           : switch
   version        : n9000-11.0(2j)

Thanks!

[Tomas de Leon · Accepted solution]

Ok.

A couple of things that I would like you perform.

1. Access console port of 9508 or ssh to spine.  Verify the MD5 checksum for aci-n9000-dk9.11.1.1j.bin matches the following:

aci-spine# md5sum aci-n9000-dk9.11.1.1j.bin
7d3b141d8a39675bc088f531796ddaf2  aci-n9000-dk9.11.1.1j.bin

If the checksum matches, proceed to step 2. If different downland the image again from

https://software.cisco.com/download/release.html?mdfid=285956854&flowid=47182&softwareid=286278844&release=11.0(2m)&relind=AVAILABLE&rellifecycle=&reltype=latest

2. Download aci-n9000-dk9.11.0.4h.bin

https://software.cisco.com/download/release.html?mdfid=285956854&flowid=47182&softwareid=286278844&release=11.0(2m)&relind=AVAILABLE&rellifecycle=&reltype=latest

verify md5checksum =
6abe77ae7a92a5695e77e86edd8f08e6

3. Cleanup bootflash.  

cd/bootflash
rm aci-n9000-dk9.11.0.2j.bin
rm aci-n9000-dk9.11.0.4h.bin-isan
rm aci-n9000-dk9.11.1.1j.bin-isan
rm auto-s
rm disk_log.txt
rm mem*

4. SCP or SFTP aci-n9000-dk9.11.0.4h.bin to the spine and put in bootflash. You should have 2 images:  aci-n9000-dk9.11.0.4h.bin, aci-n9000-dk9.11.1.1j.bin

5. Run the following CLI commands after both images are in bootflash:

setup-bootvars.sh aci-n9000-dk9.11.0.4h.bin
setup-clean-config.sh aci-n9000-dk9.11.0.4h.bin
reboot switch

6. the switch should reload with version aci-n9000-dk9.11.0.4h.bin.  Verify this is indeed the case.   The reason that we are loading 11.0.4h first is for two reasons: upgrading to 11.1.1j should be performed for an existing version of 11.0.4h. Secondly, a two stage install is necessary to upgrade bios & epld fw versions if needed.  Normally performed via  policy uprade but this is a manual process at this point.

7. once you verified the spine is running aci-n9000-dk9.11.0.4h.bin and joined the fabric, you can then perform a policy upgrade to aci-n9000-dk9.11.1.1j.bin. remove aci-n9000-dk9.11.1.1j.bin from bootflash and then perform a policy upgrade from the GUI to 11.1.1j.  During the upgrade the spine will upgrade bios and EPLDs.

8. If the first seven steps worked on first spine, repeat steps on Spine2.

[vuth60001]

Hi Tomas De Leon,

Many thanks for your support.

I have successfully upgraded firmware on 2 Spines without firmware 1.0(4x).

I reboot Spine2 and upgrade firmware with normal way : using command line on APIC "firmware upgrade switch 11.1(1j).bin nodes 104" .

On Spine1 , I using :

setup-bootvars.sh aci-n9000-dk9.11.1.1j.bin
setup-clean-config.sh aci-n9000-dk9.11.1.1j.bin
reload

Spine1 is on firmware 11.1(1j) too. But it has some faults, i attached a picture to show that.

[Tomas de Leon]

Correct.

That is why i wanted you to upgrade to 4h first.  Your EPLD & Bios did not get updated.  The "Policy" upgrade will perform this.

Simply repeat the steps but for 4h not 1j.

After 4h is running, perform a "policy" upgrade using the CLI or the GUI to 1j.  This should resolve the mismatch.

If it does not, you may need to remove power cables from switch for at least 60 seconds.  Then power back on to see if the faults go away.

T.

[vuth60001]

Many thanks  again :)

All switchs and controllers are on firmware 1.1(1j).

[Tomas de Leon]

Are all the Faults for Mismatches resolved?

Please summarize the what you did for others to reference in future.

I am glad things are resolved.

T. 

[Tomas de Leon]

Also, please provide the output of the following for each spine:

• ls -al /bootflash
• show system redundancy status
• moquery -c firmwareRunning
•  cat /mnt/pss/installer_detail.log | grep -E "Compatibility|New-Version|FPGA|Upgrade-Result|Success"
  

Thanks

T.

[lpember]

vuth60001,

You can attempt to upgrade these manually if you would like. Here are the steps to do so:

Open up the CLI of one of the spines (SSH or console)

Copy the firmware image from one of the APICs to the bootflash for the spine using SCP (assuming connectivity is working between the two)

Run "setup-bootvars.sh <image-name>"

You can check that the boot variable change was changed correctly by running "cat /mnt/cfg/0/boot/grub/menu.lst.local" which should show the image name you just selected. 

Reload the switch.

Issue a "show version" to verify that the switch booted into the desired image.