Solved: Upgrading EPLD on N9k

sschubert · ‎09-29-2020

Hi,

I'm in the process of upgrading to the latest EPLD and I came to this instruction that states to use this command if you don't need to upgrade the bios "boot nxos bootflash:n9000-dk9.9.3.5.bin." I don't have this file in bootflash and I'm not sure where I'd get it. Can someone explain if this command?

Thank you,

Steve Schubert

Christopher Hart · ‎09-29-2020

Hi Steve!

Most Nexus 9000 switches are equipped with two FPGAs - the IO FPGA, and the MI FPGA. Each FPGA has two memory regions to store its firmware - the Primary region, and the Golden region. The idea behind this is that in the rare event that one of the regions is corrupted, the FPGA would continue to function by booting firmware from the other region.

The install all epld command upgrades the Primary region of both FPGAs. After reloading, the switch will boot from the Golden region so that NX-OS can verify the Primary region was not corrupted during the upgrade. As a result, the output of show version module 1 epld will show the MI FPGA and IO FPGA versions of the Golden region, not the Primary region that was just updated. I performed an EPLD upgrade on a Nexus 93180YC-FX in my lab environment to demonstrate this:

N9K# show module 
Mod Ports             Module-Type                      Model           Status
--- ----- ------------------------------------- --------------------- ---------
1    54   48x10/25G/32G + 6x40/100G Ethernet/FC N9K-C93180YC-FX       active *  

Mod  Sw                       Hw    Slot
---  ----------------------- ------ ----
1    9.3(5)                   1.1    NA  

N9K# show version module 1 epld


EPLD Device                     Version
---------------------------------------
MI FPGA                          0x10
IO FPGA                          0x17

N9K# install epld bootflash:n9000-epld.9.3.5.img module all
Digital signature verification is successful
Compatibility check:
Module        Type         Upgradable        Impact   Reason
------  -----------------  ----------    ----------   ------
     1            SUP           Yes       disruptive   Module Upgradable

Retrieving EPLD versions.... Please wait.
Images will be upgraded according to following table:
Module  Type   EPLD              Running-Version   New-Version  Upg-Required
------  ----  -------------      ---------------   -----------  ------------
     1   SUP  MI FPGA                   0x10        0x10             No
     1   SUP  IO FPGA                   0x17        0x22            Yes
The above modules require upgrade.
The switch will be reloaded at the end of the upgrade
Do you want to continue (y/n) ?  [n] y

Proceeding to upgrade Modules.

 Starting Module 1 EPLD Upgrade

Module 1 : IO FPGA [Programming] : 100.00% (     64 of      64 sectors)
Module 1 EPLD upgrade is successful.
Module        Type  Upgrade-Result
------  ------------------  --------------
     1         SUP         Success

Module 1 EPLD upgrade is successful.


Reseting Active SUP (Module 1) FPGAs. Please wait... 

<snip, switch reloaded>

N9K# show version module 1 epld


EPLD Device                     Version
---------------------------------------
MI FPGA                          0x10
IO FPGA                          0x17

N9K# show logging logfile | include FPGA
2020 Sep 29 17:07:21 N9K %CARDCLIENT-2-FPGA_BOOT_GOLDEN: IOFPGA booted from Golden
2020 Sep 29 17:07:21 N9K %CARDCLIENT-2-FPGA_BOOT_GOLDEN: MIFPGA booted from Golden
2020 Sep 29 17:07:21 N9K %CARDCLIENT-5-MOD_BOOT_GOLDEN: Module 1 IOFPGA booted from Golden
2020 Sep 29 17:07:21 N9K %CARDCLIENT-5-MOD_BOOT_GOLDEN: Module 1 MIFPGA booted from Golden

In order to upgrade the Golden region, you will need to use the install epld bootflash:n9000-epld.9.3.5.img module all golden command to upgrade the Golden region as well. This will cause the switch to reload once more, after which it will boot from the Primary region with the upgraded EPLD versions as expected.

N9K# install epld bootflash:n9000-epld.9.3.5.img module all golden
Digital signature verification is successful
Compatibility check:
Module        Type         Upgradable        Impact   Reason
------  -----------------  ----------    ----------   ------
     1            SUP           Yes       disruptive   Module Upgradable

Retrieving EPLD versions.... Please wait.
Images will be upgraded according to following table:
Module  Type   EPLD              Running-Version   New-Version  Upg-Required
------  ----  -------------      ---------------   -----------  ------------
     1   SUP  MI FPGA                   0x10        0x10            Yes
     1   SUP  IO FPGA                   0x17        0x22            Yes
The above modules require upgrade.
The switch will be reloaded at the end of the upgrade
Do you want to continue (y/n) ?  [n] y

Proceeding to upgrade Modules.

 Starting Module 1 EPLD Upgrade

Module 1 : MI FPGA [Programming] : 100.00% (     64 of      64 sectors)
Module 1 : IO FPGA [Programming] : 100.00% (     64 of      64 sectors)
Module 1 EPLD upgrade is successful.
Module        Type  Upgrade-Result
------  ------------------  --------------
     1         SUP         Success

Module 1 EPLD upgrade is successful.


Reseting Active SUP (Module 1) FPGAs. Please wait... 

<snip, switch reloaded>

N9K# show version module 1 epld 


EPLD Device                     Version
---------------------------------------
MI FPGA                          0x10
IO FPGA                          0x22
N9K# show logging logfile | include FPGA
2020 Sep 29 19:11:47 N9K %CARDCLIENT-2-FPGA_BOOT_PRIMARY: IOFPGA booted from Primary
2020 Sep 29 19:11:47 N9K %CARDCLIENT-2-FPGA_BOOT_PRIMARY: MIFPGA booted from Primary
2020 Sep 29 19:11:47 N9K %CARDCLIENT-5-MOD_BOOT_PRIMARY: Module 1 IOFPGA booted from Primary
2020 Sep 29 19:11:47 N9K %CARDCLIENT-5-MOD_BOOT_PRIMARY: Module 1 MIFPGA booted from Primary

As a side note, I filed documentation bug CSCvv91381 (which may not yet be public-facing to you) this morning to update this documentation across the board to be more accurate and centralized. I will make sure that the procedure for upgrading the Golden region is also included as part of this update.

I hope this helps - thank you!

-Christopher

View solution in original post

Christopher Hart · ‎09-29-2020

Hi Steve!

This appears to be an error in our documentation. The command should be boot nxos bootflash:nxos.9.3.5.bin, which is the NX-OS binary image file for software release 9.3(5) that is available for download from Cisco's Software Download website. I will work internally to have this documentation corrected across all NX-OS FPGA/EPLD release notes.

Assuming that you are already running the NX-OS software release that you want to upgrade EPLD to (e.g. NX-OS software release 9.3(5)), the only command you need to run is install all epld bootflash:n9000-epld.9.3.5.img module all. Note that this command is disruptive, as the switch will reload to load the new EPLD images. The majority of the commands in the documented procedure appear to be unnecessary - I will work internally to have this corrected as well.

I hope this helps - thank you!

-Christopher

sschubert · ‎09-29-2020

Hi Christopher,

I thought this may be the case. However, the one thing that confused me was after I performed the upgrade and then ran the two commands to verify it worked. It appeared that the upgrade was successful with this commmand, "show install epld status" but unsuccessful with this command, "show version module 1 epld"

Am I missing anything?

N9236C-B# show version module 1 epld

EPLD Device Version

---------------------------------------

MI FPGA 0x9

IO FPGA 0x15

N9236C-B# show install epld status

1) Module 1 upgraded on Mon Sep 28 22:11:03 2020 (179390 us)

EPLD Install Image: EPLD image file 9.3.5. built on Mon Jul 20 22:37:31 2020

Status: EPLD Upgrade was Successful

EPLD Curr Ver Old Ver

-------------------------------------------------------

IO FPGA 0x17 0x16

2) Module 1 upgraded on Mon Sep 28 22:11:03 2020 (179326 us)

EPLD Install Image: EPLD image file 9.3.5. built on Mon Jul 20 22:37:31 2020

Status: EPLD Upgrade was Successful

EPLD Curr Ver Old Ver

-------------------------------------------------------

IO FPGA 0x17 0x16

3) Module 1 upgraded on Wed Apr 24 22:19:29 2019 ( 60873 us)

EPLD Install Image: EPLD image file 9.2.3.I9.1. built on Sun Feb 17 04:25:02 2019

Status: EPLD Upgrade was Successful

EPLD Curr Ver Old Ver

-------------------------------------------------------

IO FPGA 0x16 0x15

4) Module 1 upgraded on Wed Apr 24 22:19:29 2019 ( 60840 us)

EPLD Install Image: EPLD image file 9.2.3.I9.1. built on Sun Feb 17 04:25:02 2019

Status: EPLD Upgrade was Successful

EPLD Curr Ver Old Ver

-------------------------------------------------------

IO FPGA 0x16 0x15

Christopher Hart · ‎09-29-2020

Hi Steve!

Most Nexus 9000 switches are equipped with two FPGAs - the IO FPGA, and the MI FPGA. Each FPGA has two memory regions to store its firmware - the Primary region, and the Golden region. The idea behind this is that in the rare event that one of the regions is corrupted, the FPGA would continue to function by booting firmware from the other region.

The install all epld command upgrades the Primary region of both FPGAs. After reloading, the switch will boot from the Golden region so that NX-OS can verify the Primary region was not corrupted during the upgrade. As a result, the output of show version module 1 epld will show the MI FPGA and IO FPGA versions of the Golden region, not the Primary region that was just updated. I performed an EPLD upgrade on a Nexus 93180YC-FX in my lab environment to demonstrate this:

N9K# show module 
Mod Ports             Module-Type                      Model           Status
--- ----- ------------------------------------- --------------------- ---------
1    54   48x10/25G/32G + 6x40/100G Ethernet/FC N9K-C93180YC-FX       active *  

Mod  Sw                       Hw    Slot
---  ----------------------- ------ ----
1    9.3(5)                   1.1    NA  

N9K# show version module 1 epld


EPLD Device                     Version
---------------------------------------
MI FPGA                          0x10
IO FPGA                          0x17

N9K# install epld bootflash:n9000-epld.9.3.5.img module all
Digital signature verification is successful
Compatibility check:
Module        Type         Upgradable        Impact   Reason
------  -----------------  ----------    ----------   ------
     1            SUP           Yes       disruptive   Module Upgradable

Retrieving EPLD versions.... Please wait.
Images will be upgraded according to following table:
Module  Type   EPLD              Running-Version   New-Version  Upg-Required
------  ----  -------------      ---------------   -----------  ------------
     1   SUP  MI FPGA                   0x10        0x10             No
     1   SUP  IO FPGA                   0x17        0x22            Yes
The above modules require upgrade.
The switch will be reloaded at the end of the upgrade
Do you want to continue (y/n) ?  [n] y

Proceeding to upgrade Modules.

 Starting Module 1 EPLD Upgrade

Module 1 : IO FPGA [Programming] : 100.00% (     64 of      64 sectors)
Module 1 EPLD upgrade is successful.
Module        Type  Upgrade-Result
------  ------------------  --------------
     1         SUP         Success

Module 1 EPLD upgrade is successful.


Reseting Active SUP (Module 1) FPGAs. Please wait... 

<snip, switch reloaded>

N9K# show version module 1 epld


EPLD Device                     Version
---------------------------------------
MI FPGA                          0x10
IO FPGA                          0x17

N9K# show logging logfile | include FPGA
2020 Sep 29 17:07:21 N9K %CARDCLIENT-2-FPGA_BOOT_GOLDEN: IOFPGA booted from Golden
2020 Sep 29 17:07:21 N9K %CARDCLIENT-2-FPGA_BOOT_GOLDEN: MIFPGA booted from Golden
2020 Sep 29 17:07:21 N9K %CARDCLIENT-5-MOD_BOOT_GOLDEN: Module 1 IOFPGA booted from Golden
2020 Sep 29 17:07:21 N9K %CARDCLIENT-5-MOD_BOOT_GOLDEN: Module 1 MIFPGA booted from Golden

In order to upgrade the Golden region, you will need to use the install epld bootflash:n9000-epld.9.3.5.img module all golden command to upgrade the Golden region as well. This will cause the switch to reload once more, after which it will boot from the Primary region with the upgraded EPLD versions as expected.

N9K# install epld bootflash:n9000-epld.9.3.5.img module all golden
Digital signature verification is successful
Compatibility check:
Module        Type         Upgradable        Impact   Reason
------  -----------------  ----------    ----------   ------
     1            SUP           Yes       disruptive   Module Upgradable

Retrieving EPLD versions.... Please wait.
Images will be upgraded according to following table:
Module  Type   EPLD              Running-Version   New-Version  Upg-Required
------  ----  -------------      ---------------   -----------  ------------
     1   SUP  MI FPGA                   0x10        0x10            Yes
     1   SUP  IO FPGA                   0x17        0x22            Yes
The above modules require upgrade.
The switch will be reloaded at the end of the upgrade
Do you want to continue (y/n) ?  [n] y

Proceeding to upgrade Modules.

 Starting Module 1 EPLD Upgrade

Module 1 : MI FPGA [Programming] : 100.00% (     64 of      64 sectors)
Module 1 : IO FPGA [Programming] : 100.00% (     64 of      64 sectors)
Module 1 EPLD upgrade is successful.
Module        Type  Upgrade-Result
------  ------------------  --------------
     1         SUP         Success

Module 1 EPLD upgrade is successful.


Reseting Active SUP (Module 1) FPGAs. Please wait... 

<snip, switch reloaded>

N9K# show version module 1 epld 


EPLD Device                     Version
---------------------------------------
MI FPGA                          0x10
IO FPGA                          0x22
N9K# show logging logfile | include FPGA
2020 Sep 29 19:11:47 N9K %CARDCLIENT-2-FPGA_BOOT_PRIMARY: IOFPGA booted from Primary
2020 Sep 29 19:11:47 N9K %CARDCLIENT-2-FPGA_BOOT_PRIMARY: MIFPGA booted from Primary
2020 Sep 29 19:11:47 N9K %CARDCLIENT-5-MOD_BOOT_PRIMARY: Module 1 IOFPGA booted from Primary
2020 Sep 29 19:11:47 N9K %CARDCLIENT-5-MOD_BOOT_PRIMARY: Module 1 MIFPGA booted from Primary

As a side note, I filed documentation bug CSCvv91381 (which may not yet be public-facing to you) this morning to update this documentation across the board to be more accurate and centralized. I will make sure that the procedure for upgrading the Golden region is also included as part of this update.

I hope this helps - thank you!

-Christopher