cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2344
Views
10
Helpful
13
Replies
Highlighted
Beginner

Failed when upgrade firmware n9000-11.1(1j)

Hi,

My old firmware is n9000-11.0(2j) and apic-1.0(2j).

I have done upgrade firmware apic-1.1(1j) on controller and n9000-11.1(1j) on 2 Leafs.

But when i upgrade firmware n9000-11.1(1j) on 2 Spines, it returns fail.  I got some picture for faults log ( attachment ).

 

Can someone help me to find more detail this failed state :(.

Thanks.

 

2 ACCEPTED SOLUTIONS

Accepted Solutions
Highlighted
Beginner

Hi,Here are the release notes

Hi,

Here are the release notes for 1.1(1j): http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/release/notes/apic_rn_111.html

You'll notice that the supported upgrade path would involve upgrading from 1.0(2x) to 1.0(4x) before going to 1.1(1j). Have you tried upgrading the spines to 1.0(4x) code? Were the APIC upgrades to 1.1(1j) successful?

In regard to the fault, it looks like the issue may be with the configuration for the firmware and maintenance groups and not the upgrade itself. Can you double-check that you have a target firmware set-up under the firmware groups?

View solution in original post

Highlighted
Cisco Employee

Ok. A couple of things that I

Ok.

 

A couple of things that I would like you perform.

1. Access console port of 9508 or ssh to spine.  Verify the MD5 checksum for aci-n9000-dk9.11.1.1j.bin matches the following:

aci-spine# md5sum aci-n9000-dk9.11.1.1j.bin
7d3b141d8a39675bc088f531796ddaf2  aci-n9000-dk9.11.1.1j.bin

If the checksum matches, proceed to step 2. If different downland the image again from

https://software.cisco.com/download/release.html?mdfid=285956854&flowid=47182&softwareid=286278844&release=11.0(2m)&relind=AVAILABLE&rellifecycle=&reltype=latest


2. Download aci-n9000-dk9.11.0.4h.bin

https://software.cisco.com/download/release.html?mdfid=285956854&flowid=47182&softwareid=286278844&release=11.0(2m)&relind=AVAILABLE&rellifecycle=&reltype=latest

verify md5checksum =
6abe77ae7a92a5695e77e86edd8f08e6


3. Cleanup bootflash.  

cd/bootflash
rm aci-n9000-dk9.11.0.2j.bin
rm aci-n9000-dk9.11.0.4h.bin-isan
rm aci-n9000-dk9.11.1.1j.bin-isan
rm auto-s
rm disk_log.txt
rm mem*


4. SCP or SFTP aci-n9000-dk9.11.0.4h.bin to the spine and put in bootflash. You should have 2 images:  aci-n9000-dk9.11.0.4h.bin, aci-n9000-dk9.11.1.1j.bin


5. Run the following CLI commands after both images are in bootflash:

setup-bootvars.sh aci-n9000-dk9.11.0.4h.bin
setup-clean-config.sh aci-n9000-dk9.11.0.4h.bin
reboot switch

 

6. the switch should reload with version aci-n9000-dk9.11.0.4h.bin.  Verify this is indeed the case.   The reason that we are loading 11.0.4h first is for two reasons: upgrading to 11.1.1j should be performed for an existing version of 11.0.4h. Secondly, a two stage install is necessary to upgrade bios & epld fw versions if needed.  Normally performed via  policy uprade but this is a manual process at this point.

 

7. once you verified the spine is running aci-n9000-dk9.11.0.4h.bin and joined the fabric, you can then perform a policy upgrade to aci-n9000-dk9.11.1.1j.bin. remove aci-n9000-dk9.11.1.1j.bin from bootflash and then perform a policy upgrade from the GUI to 11.1.1j.  During the upgrade the spine will upgrade bios and EPLDs.


8. If the first seven steps worked on first spine, repeat steps on Spine2.

View solution in original post

13 REPLIES 13
Highlighted
Beginner

Hi,Here are the release notes

Hi,

Here are the release notes for 1.1(1j): http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/release/notes/apic_rn_111.html

You'll notice that the supported upgrade path would involve upgrading from 1.0(2x) to 1.0(4x) before going to 1.1(1j). Have you tried upgrading the spines to 1.0(4x) code? Were the APIC upgrades to 1.1(1j) successful?

In regard to the fault, it looks like the issue may be with the configuration for the firmware and maintenance groups and not the upgrade itself. Can you double-check that you have a target firmware set-up under the firmware groups?

View solution in original post

Highlighted
Beginner

Hi lpember,All APIC and LEAF

Hi lpember,

All APIC and LEAF have upgraded firmware 1.1(1j). Each switch is in one firmware groups.

I have tried upgrade firmware 1.0(4x) first , but it fails too.

Some information in 2 picture.

Thanks for reply .

Highlighted
Cisco Employee

HelloJust out of curiosity

Hello

Just out of curiosity,

What type of spines do you have?

Highlighted
Beginner

Hi Dpita,It is N9K-C9508.

Hi Dpita,

It is N9K-C9508.

Highlighted
Cisco Employee

do you have both supervisors

do you have both supervisors installed and powered on?

if so, one supervisor may have been upgraded and when rebooted, came on the warm standby which was not upgraded.

 

If you have external access to fabric, private message me the access information with login credentials and I can look at your spines and see what is going on.

 

Cheers!


T.

Highlighted
Beginner

Hi Tomas De Leon,I have some

Hi Tomas De Leon,

I have some information from those commands:

  1. ls -al /bootflash
    -rwx------  1 rad  admin       293 Apr 22 08:00 MDS201504201837188540.lic
    -rw-r--r--  1 root root  500237761 Jan  6 03:45 aci-n9000-dk9.11.0.2j.bin
    -rw-rw-rw-  1 root root  254275072 Jun 23 12:39 aci-n9000-dk9.11.0.4h.bin-isan
    -rw-rw-rw-  1 root root  591631164 Jun 23 11:14 aci-n9000-dk9.11.1.1j.bin
    -rw-rw-rw-  1 root root  396732732 Jun 23 11:14 aci-n9000-dk9.11.1.1j.bin-isan
    -rw-r--r--  1 root root  328541633 Mar 13 04:27 auto-s
    -rw-r--r--  1 root root         54 Jun 24 02:20 disk_log.txt
    -rw-r--r--  1 root root    7750390 Jun 24 02:20 mem_log.txt
    -rw-r--r--  1 root root       2474 Jan  6 03:50 mem_log.txt.old.gz

     

  2. show system redundancy status

    Redundancy mode
    ---------------
          administrative:   Warm
             operational:   Warm

    This supervisor (sup-27)
    -----------------------
        Redundancy state:   Active
        Supervisor state:   Active
          Internal state:   Active with warm standby

    Other supervisor (sup-28)
    ------------------------
        Redundancy state:   Standby
        Supervisor state:   Warm standby
          Internal state:   Warm standby

     

  3. moquery -c firmwareRunning

    Total Objects shown: 1

    # firmware.Running
    biosTs         : 2014-09-10T07:00:00.000+07:00
    biosVer        : 08.06
    childAction    : 
    descr          : version 11.0(2j) [build 11.0(2j)]
    dn             : sys/fwstatuscont/running
    internalLabel  : 4755470886b309d39a3dcb8fd1e04db8a015f1f9
    ksFile         : bootflash:///aci-n9000-dk9.11.0.2j.bin
    modTs          : never
    mode           : normal
    peVer          : 1.0(2j)
    rn             : running
    status         : 
    sysFile        : bootflash:///auto-s
    ts             : 2014-11-12T01:08:58.000+07:00
    type           : switch
    version        : n9000-11.0(2j)

     

 

Thanks!

Highlighted
Cisco Employee

Ok. A couple of things that I

Ok.

 

A couple of things that I would like you perform.

1. Access console port of 9508 or ssh to spine.  Verify the MD5 checksum for aci-n9000-dk9.11.1.1j.bin matches the following:

aci-spine# md5sum aci-n9000-dk9.11.1.1j.bin
7d3b141d8a39675bc088f531796ddaf2  aci-n9000-dk9.11.1.1j.bin

If the checksum matches, proceed to step 2. If different downland the image again from

https://software.cisco.com/download/release.html?mdfid=285956854&flowid=47182&softwareid=286278844&release=11.0(2m)&relind=AVAILABLE&rellifecycle=&reltype=latest


2. Download aci-n9000-dk9.11.0.4h.bin

https://software.cisco.com/download/release.html?mdfid=285956854&flowid=47182&softwareid=286278844&release=11.0(2m)&relind=AVAILABLE&rellifecycle=&reltype=latest

verify md5checksum =
6abe77ae7a92a5695e77e86edd8f08e6


3. Cleanup bootflash.  

cd/bootflash
rm aci-n9000-dk9.11.0.2j.bin
rm aci-n9000-dk9.11.0.4h.bin-isan
rm aci-n9000-dk9.11.1.1j.bin-isan
rm auto-s
rm disk_log.txt
rm mem*


4. SCP or SFTP aci-n9000-dk9.11.0.4h.bin to the spine and put in bootflash. You should have 2 images:  aci-n9000-dk9.11.0.4h.bin, aci-n9000-dk9.11.1.1j.bin


5. Run the following CLI commands after both images are in bootflash:

setup-bootvars.sh aci-n9000-dk9.11.0.4h.bin
setup-clean-config.sh aci-n9000-dk9.11.0.4h.bin
reboot switch

 

6. the switch should reload with version aci-n9000-dk9.11.0.4h.bin.  Verify this is indeed the case.   The reason that we are loading 11.0.4h first is for two reasons: upgrading to 11.1.1j should be performed for an existing version of 11.0.4h. Secondly, a two stage install is necessary to upgrade bios & epld fw versions if needed.  Normally performed via  policy uprade but this is a manual process at this point.

 

7. once you verified the spine is running aci-n9000-dk9.11.0.4h.bin and joined the fabric, you can then perform a policy upgrade to aci-n9000-dk9.11.1.1j.bin. remove aci-n9000-dk9.11.1.1j.bin from bootflash and then perform a policy upgrade from the GUI to 11.1.1j.  During the upgrade the spine will upgrade bios and EPLDs.


8. If the first seven steps worked on first spine, repeat steps on Spine2.

View solution in original post

Highlighted
Beginner

Hi Tomas De Leon,Many thanks

Hi Tomas De Leon,

Many thanks for your support.

I have successfully upgraded firmware on 2 Spines without firmware 1.0(4x).

I reboot Spine2 and upgrade firmware with normal way : using command line on APIC "firmware upgrade switch 11.1(1j).bin nodes 104" .

On Spine1 , I using :

setup-bootvars.sh aci-n9000-dk9.11.1.1j.bin
setup-clean-config.sh aci-n9000-dk9.11.1.1j.bin
reload

Spine1 is on firmware 11.1(1j) too. But it has some faults, i attached a picture to show that.

Highlighted
Cisco Employee

Correct.That is why i wanted

Correct.

That is why i wanted you to upgrade to 4h first.  Your EPLD & Bios did not get updated.  The "Policy" upgrade will perform this.

 

Simply repeat the steps but for 4h not 1j.

After 4h is running, perform a "policy" upgrade using the CLI or the GUI to 1j.  This should resolve the mismatch.

 

If it does not, you may need to remove power cables from switch for at least 60 seconds.  Then power back on to see if the faults go away.

 

T.

 

 

 

 

Highlighted
Beginner

Many thanks  again :)All

Many thanks  again :)

All switchs and controllers are on firmware 1.1(1j).

Highlighted
Cisco Employee

Are all the Faults for

Are all the Faults for Mismatches resolved?

Please summarize the what you did for others to reference in future.

 

I am glad things are resolved.

 

T. 

Highlighted
Cisco Employee

Also, please provide the

Also, please provide the output of the following for each spine:

 

  • ls -al /bootflash
  • show system redundancy status
  • moquery -c firmwareRunning
  •  cat /mnt/pss/installer_detail.log | grep -E "Compatibility|New-Version|FPGA|Upgrade-Result|Success"

Thanks

 

T.

 

Highlighted
Beginner

vuth60001,You can attempt to

vuth60001,

You can attempt to upgrade these manually if you would like. Here are the steps to do so:

Open up the CLI of one of the spines (SSH or console)

Copy the firmware image from one of the APICs to the bootflash for the spine using SCP (assuming connectivity is working between the two)

Run "setup-bootvars.sh <image-name>"

You can check that the boot variable change was changed correctly by running "cat /mnt/cfg/0/boot/grub/menu.lst.local" which should show the image name you just selected. 

Reload the switch.

Issue a "show version" to verify that the switch booted into the desired image.