Nexus 9504 Dual Supervisor Upgrade

[russell.sage]

I have a Nexus 9504 with dual supervisors I am unclear having read the release notes for 9.3(10) as to how the upgrade works. There is no concept of a slave supervisor looking at the directory structure. The release notes state to conduct the upgrade on the active supervisor but at no point discuss what to do with the standby supervisor. I presume I have to copy the software onto the standby supervisor or does the upgrade process take care of this?

Relative newbie on Nexus.

[Christopher Hart · Accepted solution]

Hi Russell!

When a disruptive upgrade is initiated on the active supervisor of modular Nexus 9500 switches with dual supervisors with the install all command, the NX-OS binary image file is automatically copied to the bootflash of the standby supervisor as part of the upgrade process.

To give you some extra confidence, I performed an NX-OS upgrade from 9.3(7) to 9.3(10) on a Nexus 9504 with dual supervisors to show you the process. 

N9K-C9504# show module 
<snip>
Mod Ports             Module-Type                      Model           Status
--- ----- ------------------------------------- --------------------- ---------
1    32   32x40/100G Ethernet Module            N9K-X9732C-FX         ok        
2    36   36x100G Ethernet Module               N9K-X9736C-EX         ok        
22   0    4-slot Fabric Module                  N9K-C9504-FM-E        ok        
23   0    4-slot Fabric Module                  N9K-C9504-FM-E        ok        
24   0    4-slot Fabric Module                  N9K-C9504-FM-E        ok        
26   0    4-slot Fabric Module                  N9K-C9504-FM-E        ok        
27   0    Supervisor Module                     N9K-SUP-B+            active *  
28   0    Supervisor Module                     N9K-SUP-B+            ha-standby
29   0    System Controller                     N9K-SC-A              standby   
30   0    System Controller                     N9K-SC-A              active    

Mod  Sw                       Hw    Slot
---  ----------------------- ------ ----
1    9.3(7)                   1.0    LC1 
2    9.3(7)                   1.3    LC2 
22   9.3(7)                   1.3    FM2 
23   9.3(7)                   1.3    FM3 
24   9.3(7)                   1.3    FM4 
26   9.3(7)                   1.3    FM6 
27   9.3(7)                   1.0    SUP1
28   9.3(7)                   1.0    SUP2
29   9.3(7)                   2.0    SC1 
30   9.3(7)                   2.0    SC2 

As you can see, the NX-OS binary image file for NX-OS 9.3(10) (nxos.9.3.10.bin) is present on the bootflash of the active supervisor.

N9K-C9504# dir bootflash: | include nxos.9.3.10.bin
 1966000640    Aug 19 00:53:47 2022  nxos.9.3.10.bin

However, this same nxos.9.3.10.bin file is not present on the standby supervisor's bootflash.

N9K-C9504# dir bootflash://sup-standby/ | include nxos.9.3.10.bin
N9K-C9504# 
N9K-C9504# dir bootflash://sup-standby/
       4096    Jan 17 09:00:45 2023  .rpmstore/
       4096    Jan 17 17:12:22 2022  .swtam/
        172    Jan 14 11:04:10 2023  20230114_110342_poap_4579_init.log
        172    Jan 16 00:18:45 2023  20230116_001817_poap_4559_init.log
        172    Jan 16 03:40:16 2023  20230116_033948_poap_4531_init.log
        172    Jan 16 19:47:06 2023  20230116_194637_poap_4555_init.log
        172    Jan 17 09:24:08 2023  20230117_090408_poap_1590_init.log
       1639    Jan 16 17:11:57 2023  bgp.1.0
          0    Jan 17 07:31:59 2023  bootflash_sync_list
      64507    May 28 02:42:50 2021  d
       4096    Jun 23 19:38:27 2020  eem_snapshots/
       4096    Apr 29 07:04:17 2022  evt_log_snapshot/
       4096    Oct 31 00:58:23 2020  home/
       4096    Nov 22 22:13:57 2022  intersight/
       4096    Dec 28 03:36:05 2022  lost+found/
       2849    Jan 17 07:29:29 2023  lpssutil_lpss_log
 1011037696    Dec 28 02:45:56 2022  nxos.7.0.3.I7.6.bin
 1973174784    Jan 13 15:16:42 2023  nxos.9.3.6.bin
 1959441408    May 25 19:17:06 2021  nxos.9.3.7.bin
 1965225472    Nov 23 22:06:55 2022  nxos.9.3.7a.bin
 1856263168    Dec 20 08:24:46 2022  nxos64.10.2.1.F.bin
       4096    Aug 24 06:15:06 2018  onep/
          0    Jan 10 12:47:21 2023  platform-sdk.cmd
       4096    Feb 18 03:45:30 2022  ptp_autolog/
       4096    Apr 29 03:41:14 2022  scripts/
       4096    Jan 10 12:42:21 2023  virt_strg_pool_bf_vdc_1/
       4096    Jan 10 12:42:31 2023  virtual-instance/
       4096    Jan 10 12:43:06 2023  virtual-instance-stby-sync/
        120    Jan 17 09:04:21 2023  virtual-instance.conf

Usage for bootflash://sup-standby
 8858894336 bytes used
12872318976 bytes free
21731213312 bytes total

If we initiate the NX-OS upgrade process with the install all nxos bootflash:nxos.9.3.10.bin command on the active supervisor, we can see that the NX-OS binary image is copied to the bootflash of the standby supervisor during the upgrade process. I've highlighted this step for you below in red.

N9K-C9504# install all nxos bootflash:nxos.9.3.10.bin
Installer will perform compatibility check first. Please wait. 
Installer is forced disruptive

Verifying image bootflash:/nxos.9.3.10.bin for boot variable "nxos".
[####################] 100% -- SUCCESS

Verifying image type.
[####################] 100% -- SUCCESS

Preparing "bios" version info using image bootflash:/nxos.9.3.10.bin.
[####################] 100% -- SUCCESS

Preparing "nxos" version info using image bootflash:/nxos.9.3.10.bin.
[####################] 100% -- SUCCESS

Performing module support checks.
[####################] 100% -- SUCCESS

Notifying services about system upgrade.
[####################] 100% -- SUCCESS



Compatibility check is done:
Module  bootable          Impact  Install-type  Reason
------  --------  --------------  ------------  ------
     1       yes      disruptive         reset  default upgrade is not hitless
     2       yes      disruptive         reset  default upgrade is not hitless
    22       yes      disruptive         reset  default upgrade is not hitless
    23       yes      disruptive         reset  default upgrade is not hitless
    24       yes      disruptive         reset  default upgrade is not hitless
    26       yes      disruptive         reset  default upgrade is not hitless
    27       yes      disruptive         reset  default upgrade is not hitless
    28       yes      disruptive         reset  default upgrade is not hitless
    29       yes      disruptive         reset  default upgrade is not hitless
    30       yes      disruptive         reset  default upgrade is not hitless



Images will be upgraded according to following table:
Module       Image                  Running-Version(pri:alt)           New-Version  Upg-Required
------  ----------  ----------------------------------------  --------------------  ------------
     1       lcn9k                                    9.3(7)               9.3(10)           yes
     1        bios     v03.48(04/24/2018):v03.48(04/24/2018)    v03.48(04/24/2018)            no
     2       lcn9k                                    9.3(7)               9.3(10)           yes
     2        bios     v03.48(04/24/2018):v03.48(04/24/2018)    v03.48(04/24/2018)            no
    22       lcn9k                                    9.3(7)               9.3(10)           yes
    22        bios     v03.48(04/24/2018):v03.48(04/24/2018)    v03.48(04/24/2018)            no
    23       lcn9k                                    9.3(7)               9.3(10)           yes
    23        bios     v03.48(04/24/2018):v03.48(04/24/2018)    v03.48(04/24/2018)            no
    24       lcn9k                                    9.3(7)               9.3(10)           yes
    24        bios     v03.48(04/24/2018):v03.48(04/24/2018)    v03.48(04/24/2018)            no
    26       lcn9k                                    9.3(7)               9.3(10)           yes
    26        bios     v03.48(04/24/2018):v03.48(04/24/2018)    v03.48(04/24/2018)            no
    27        nxos                                    9.3(7)               9.3(10)           yes
    27        bios     v05.47(04/28/2022):v05.24(08/02/2017)    v05.47(04/28/2022)            no
    28        nxos                                    9.3(7)               9.3(10)           yes
    28        bios     v05.46(01/13/2022):v05.24(08/02/2017)    v05.47(04/28/2022)           yes
    29       lcn9k                                    9.3(7)               9.3(10)           yes
    29        bios                       v01.48(00:v01.48(00             v01.48(00            no
    30       lcn9k                                    9.3(7)               9.3(10)           yes
    30        bios                       v01.48(00:v01.48(00             v01.48(00            no


Switch will be reloaded for disruptive upgrade.
Do you want to continue with the installation (y/n)?  [n] y


Install is in progress, please wait.

Performing runtime checks.
[####################] 100% -- SUCCESS

Syncing image bootflash:/nxos.9.3.10.bin to standby.
[####################] 100% -- SUCCESS

Setting boot variables.
[####################] 100% -- SUCCESS

Performing configuration copy.
[####################] 100% -- SUCCESS

Module 1: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% -- SUCCESS

Module 2: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% -- SUCCESS

Module 22: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% -- SUCCESS

Module 23: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% -- SUCCESS

Module 24: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% -- SUCCESS

Module 26: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% -- SUCCESS

Module 27: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% -- SUCCESS

Module 28: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% -- SUCCESS

Module 29: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% -- SUCCESS

Module 30: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% -- SUCCESS
2023 Jan 17 09:48:51 N9K-C9504 %$ VDC-1 %$ %VMAN-2-ACTIVATION_STATE: Successfully deactivated virtual service 'guestshell+'  


Finishing the upgrade, switch will reboot in 10 seconds.
N9K-C9504# 

Once the upgrade process has completed, we can see that both supervisors are now running NX-OS 9.3(10) as expected. This is highlighted in red below.

N9K-C9504# show module
<snip>
Mod Ports             Module-Type                      Model           Status
--- ----- ------------------------------------- --------------------- ---------
1    32   32x40/100G Ethernet Module            N9K-X9732C-FX         ok        
2    36   36x100G Ethernet Module               N9K-X9736C-EX         ok        
22   0    4-slot Fabric Module                  N9K-C9504-FM-E        ok        
23   0    4-slot Fabric Module                  N9K-C9504-FM-E        ok        
24   0    4-slot Fabric Module                  N9K-C9504-FM-E        ok        
26   0    4-slot Fabric Module                  N9K-C9504-FM-E        ok        
27   0    Supervisor Module                     N9K-SUP-B+            active *  
28   0    Supervisor Module                     N9K-SUP-B+            ha-standby
29   0    System Controller                     N9K-SC-A              standby   
30   0    System Controller                     N9K-SC-A              active    

Mod  Sw                       Hw    Slot
---  ----------------------- ------ ----
1    9.3(10)                  1.0    LC1 
2    9.3(10)                  1.3    LC2 
22   9.3(10)                  1.3    FM2 
23   9.3(10)                  1.3    FM3 
24   9.3(10)                  1.3    FM4 
26   9.3(10)                  1.3    FM6 
27   9.3(10)                  1.0    SUP1
28   9.3(10)                  1.0    SUP2
29   9.3(10)                  2.0    SC1 
30   9.3(10)                  2.0    SC2 

 I hope this helps - thank you!

-Christopher

[russell.sage]

Chris 

Many thanks - great clarification

[Mike Ruiz]

Hello, Christopher. Taking advantage of the fact that I came across your post, I would like to ask you about the process for replacing the standby supervisor. In this case, should we update it to the version of the active supervisor, or is there a synchronization that occurs after inserting the new module? Additionally, would this process have any impact on the network?

[rohit.sharma3]

Hello, I need some assistance to replace current SUP-B with SUP-B+ supervisor card. Can you please advise what are specific steps to follow as this is not getting clear in the document provided by Cisco.