IO FPGA epld fails upgrade on Cisco Nexus 9k (C93180YC-EX)

[will]

Basically upgrading EPLD on nexus 9k platform:

install epld bootflash:n9000-epld.9.3.9.img module all

Shows I need to do the upgrade and then runs through it and reboots switch.

When switch comes back up, it still runs at same (old 0x14) level IO FPGA as before.

Do i need a hard power cycle for this to take effect?

Apparently some net app users found this problem too, but I don't see an answer:
IO FPGA fails to upgrade on Cisco Nexus switches - NetApp Knowledge Base

https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Systems/Fabric%2C_Interconnect_and_Management_Switches/IO_FPGA_fails_to_upgrade_on_Cisco_Nexus_switches

thx in advance.

[Christopher Hart · Accepted solution]

Hi Will!

Nexus 9000 switches have two "regions" that hold the EPLD FPGA firmware - a "Primary" region, and a "Golden" region. Your switch most likely booted into the Golden region - if you run the show logging logfile | include FPGA command, you may see a syslog similar to the following:

%CARDCLIENT-5-MOD_BOOT_GOLDEN: Module 28 IOFPGA booted from Golden

This is done by the switch so that you can address a known Secure Boot security vulnerability CVE-2019-1649. To fix this, you need to upgrade the Golden region with the below command:

switch# install epld bootflash:n9000-epld.9.3.9.img module 1 golden

Note the "golden" keyword at the end of this command, which indicates the Golden region should be updated. Also note that this command is hidden, so you will need to type it in exactly as shown in order for it to execute. This command will cause the switch to update the Golden region and reboot, after which it should boot into the Primary region once more.

This is documented in the "Cisco Secure Boot Hardware Tampering Vulnerability - Remediation Steps" section of the 9.3(x) Cisco Nexus 9000 Series FPGA/EPLD Upgrade Release Notes.

I hope this helps - thank you!

-Christopher

[will]

thx chris! that appears to have done the trick, with an interesting side-twist:

2 of my 4 devices had booted to golden, base on the show log file command. and 2 had booted to primary.

I upgraded the two golden ones to new EPLD with the golden hidden switch.

the other 2 which booted to primary still didnt take the upgrade the normal way. I added the golden switch and they upgraded and then booted to the golden after the upgrade?? Not sure whats going here, but im all upgraded.

Will the device stay on the golden until the next epld upgrade? do these things flop back between golden and primary only on the reboot after the epld upgrade?