problem with a 9372TX-E that was working with no error before upgrading from 9.3.7 to 9.3.14, it restart all the time on final stage with error "switch %$ VDC-1 %$ %MODULE-2-MOD_DIAG_FAIL: Module 1 (Serial number: SAL2023RCXL) reported failure on ports Ethernet1/1-48 (Ethernet) due to DEVICE POST/DIAG FAILURE in device DEV_BCM56850_ASIC (device error 0x2)"... it was upgraded from 9.3.1 perfectly fine with no error. I tried going back to nxos.7.0.3.I7.10, 9.3.1 or 9.3.7, but same error stop the installation... there was no disruption, no physical or any other software changes. I know this model reach EOL but it was a perfectly fine hardware up until with absolute no reason it display and report "device failure".
console display output:
CISCO SWITCH Ver7.69
Switch G1
Device detected on 0:6:0 after 0 msecs
Device detected on 0:1:1 after 0 msecs
Device detected on 0:1:0 after 0 msecs
MCFrequency 1333Mhz
Relocated to memory
Time: 3/23/2025 15:55:26
Detected CISCO IOFPGA
Donner Present
MIFPGA Present
Code Signing Results: 0x0
Using Upgrade FPGA
Checking and setting PSU fan directions
Booting from Primary Bios
FPGA Revison : 0x6
FPGA ID : 0x1399840
FPGA Date : 0x20160719
Power Debug Register: 0x0
Reset Cause Register: 0x80000000
Boot Ctrl Register : 0xe0ff
FPGA Update Status : 0x20
FPGA SPI Flash Micron(Numonyx) N25Q064A
Detected CISCO MIFPGA
FPGA Update Status : 0x20
Version 2.16.1240. Copyright (C) 2013 American Megatrends, Inc.
Board type 2
IOFPGA @ 0xc8000000
SLOT_ID @ 0xf
Standalone chassis
check_bootmode: grub: Continue grub
Trying to read config file /boot/grub/menu.lst.local from (hd0,0)
Filesystem type is fat, partition type 0xb
Trying to read config file /boot/grub/menu.lst.local from (hd1,4)
Filesystem type is ext2fs, partition type 0x83
Trying to read config file /boot/grub/menu.lst.local from (hd1,5)
Filesystem type is ext2fs, partition type 0x83
No autoboot or failed autoboot. falling to loader
Loader Version 7.69
loader > cmdline recoverymode=1
loader > boot usb1:nxos.9.3.14.bin
Security Lock
Booting usb1:nxos.9.3.14.bin
Trying diskboot
Filesystem type is fat, partition type 0xb
Image valid
Image Signature verification was Successful.
Boot Time: 3/23/2025 15:56:41
Security Lock
Installing klm_card_index
done
Linking n9k flash devices
creating flash devices BOOT_DEV= sda
INIT: version 2.88 booting
Installing ata_piix module ... done.
Unsquashing rootfs ...
vm.min_free_kbytes = 174080
Enabling 8250 serial driver spurious INTs workaround
Installing isan procfs ... done.
is_lxc: is_titan_eor: is_stby: suffix: klm_ftrace: /isanboot/lib/modules/klm_ftrace.o
Installing ftrace in non-lxc mode done
Installing SSE module with card index 21027 ... done.
Creating SSE device node 247 ... done.
Loading I2C driver ... done.
Loading MEM scrub driver ... done.
Installing CCTRL driver for card_type 21 without NEED_GEM ... done.
Loading IGB driver ... done.
Checking SSD firmware ...
Model Number: Micron_M600_MTFDDAT064MBF
Serial Number: MSA20090GBE
Firmware Revision: MC04
Checking all filesystems.Warning: switch is starting up with default configuration
/bin/mount /dev/sdb1 /usbslot1 2>/dev/null
Extracting rpms from image...
2916585 blocks
/
Unmounting USB drive
Installing SPROM driver ... 21027 IS_N9K done.
Installing pfmsvcs module ...done.
Installing nvram module ... done.
Installing if_index module with port mode 6 ... done.
Calling install_tor for TOR
Installing fcfwd
Installing RNI lcnd ... done
Installing LC netdev ... done
Installing psdev module ... done.
Installing veobc module ... done.
Checking SR card
Card Index is 21027
Inserting eMMC module ...
Inserting mtdphysmap module...Inserting OBFL module ... done.
Making OBFL character devices
mounting plog for N9k!
Mounting OBFL pstore for mtd
Inserting kernel_services module ... done.
Making kernel_services character devices
cgroups initialized
Removing any system startup links for cgroups-init ...
Adding system startup for /etc/init.d/cgroups-init.
Removing any system startup links for docker ...
Running groupadd commands...
NOTE: docker: Performing groupadd with [ -r docker]
Running useradd commands...
NOTE: docker: Performing useradd with [ -r -U -s /bin/false dockremap]
update-alternatives: Linking /bin/vi to /usr/bin/vim.vim
update-alternatives: Linking /usr/bin/vim to /usr/bin/vim.vim
update-alternatives: Linking /usr/bin/xxd to /usr/bin/xxd.vim
exit code: 1
mkdir: cannot create directory '/lc': File exists
tune2fs 1.42.9 (28-Dec-2013)
Setting reserved blocks percentage to 0% (0 blocks)
Starting portmap daemon...
creating NFS state directory: done
starting 8 nfsd kernel threads: done
starting mountd: done
starting statd: done
Saving image for img-sync ...
Loading system software
Patch Repository Setup completed successfully
INIT: Sending processes the KILL signal
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (C) 2002-2024, Cisco and/or its affiliates.
All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under their own
licenses, such as open source. This software is provided "as is," and unless
otherwise stated, there is no warranty, express or implied, including but not
limited to warranties of merchantability and fitness for a particular purpose.
Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or
GNU General Public License (GPL) version 3.0 or the GNU
Lesser General Public License (LGPL) Version 2.1 or
Lesser General Public License (LGPL) Version 2.0.
A copy of each such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://opensource.org/licenses/gpl-3.0.html and
http://www.opensource.org/licenses/lgpl-2.1.php and
http://www.gnu.org/licenses/old-licenses/library.txt.
switch(boot)# init system
This command is going to erase your startup-config, licenses as well as the contents of your bootflash:.
Do you want to continue? (y/n) [n] y
Initializing the system ...
Checking flash ...
Backing up boot image ...
Erasing Flash ...
Partitioning ...
Partitioning successfull
Formatting bootloader: ...
mke2fs 1.42.9 (28-Dec-2013)
Initializing startup-config and licenses ...
Formatting cfg0: ...
mke2fs 1.42.9 (28-Dec-2013)
Formatting cfg1: ...
mke2fs 1.42.9 (28-Dec-2013)
Formatting pss: ...
mke2fs 1.42.9 (28-Dec-2013)
Formatting plog: ...
mke2fs 1.42.9 (28-Dec-2013)
Formatting bootflash: ...
mke2fs 1.42.9 (28-Dec-2013)
Formatting logflash: ...
mke2fs 1.42.9 (28-Dec-2013)
Mounting ...
mount: can't find /mnt/plog in /etc/fstab
ERROR: cannot mount filesystem
Restoring boot image...
Resetting CMOS to default configuration ...
Reinitializing NVRAM contents ...
Initialization completed.
switch(boot)# load-nxos
Unsquashing rootfs ...
Total size needed in bootflash is 113348
check bootflash : OK
Total size needed in bootflash is 41340
check bootflash : OK
Creating /dev/mcelog
Starting mcelog daemon
INIT: Switching to runlevel: 3
INIT: Sending processes the TERM signal
INIT: (boot)#
Running S93thirdparty-script...
Populating conf files for hybrid sysmgr ...
Starting hybrid sysmgr ...
done
Netbroker support IS present in the kernel.
done
Skipping EMON in TOR ...
Executing Prune clis.
ethernet switching mode Sun Mar 23 15:59:18 UTC 2025
Mar 23 15:59:21 %FW_APP-2-FIRMWARE_IMAGE_LOAD_SUCCESS No Firmware needed for Non SR card.
2025 Mar 23 15:59:26 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: <<%USBHSD-2-MOUNT>> logflash: online - usbhsd
2025 Mar 23 15:59:26 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: <<%USBHSD-2-USB_SWAP>> USB insertion or removal detected - usbhsd
2025 Mar 23 15:59:26 %$ VDC-1 %$ %PLATFORM-2-PS_FAIL: Power supply 1 failed or shut down (Serial number DCB2022X3NM)
2025 Mar 23 15:59:26 %$ VDC-1 %$ %PLATFORM-2-PS_OK: Power supply 2 ok (Serial number DCB2022X3NO)
2025 Mar 23 15:59:26 %$ VDC-1 %$ %PLATFORM-2-PS_FANOK: Fan in Power supply 2 ok
2025 Mar 23 15:59:26 %$ VDC-1 %$ %PLATFORM-2-PS_ABSENT: Power supply 1 is absent/shutdown, ps-redundancy might be affected
2025 Mar 23 15:59:26 %$ VDC-1 %$ %PLATFORM-2-PS_RED_MODE_CHG: Power supply operational redundancy mode changed to non-redundant
2025 Mar 23 15:59:26 %$ VDC-1 %$ %PLATFORM-2-FANMOD_FAN_OK: Fan module 1 (Fan1(sys_fan1) fan) ok
2025 Mar 23 15:59:26 %$ VDC-1 %$ %PLATFORM-2-FANMOD_FAN_OK: Fan module 2 (Fan2(sys_fan2) fan) ok
2025 Mar 23 15:59:26 %$ VDC-1 %$ %PLATFORM-2-FANMOD_FAN_OK: Fan module 3 (Fan3(sys_fan3) fan) ok
2025 Mar 23 15:59:26 %$ VDC-1 %$ %PLATFORM-2-FANMOD_FAN_OK: Fan module 4 (Fan4(sys_fan4) fan) ok
2025 Mar 23 15:59:26 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: <<%USBHSD-2-MOUNT>> USB1: online - usbhsd
2025 Mar 23 15:59:30 %$ VDC-1 %$ netstack: Registration with cli server complete
2025 Mar 23 15:59:31 %$ VDC-1 %$ %USER-1-SYSTEM_MSG: ERROR: NXOS RPMs list cannot be empty - loadrpm.sh - root
2025 Mar 23 15:59:31 %$ VDC-1 %$ %PLUGIN-2-PLUGIN_LOAD_INTERNAL_ERROR: Plugin has internal error during load operation. Reason: plugin_extract_img: script '/isan/bin/loadrpm.sh' returned failure. Refer to /var/log/rpm_install.log for error logs
System is coming up ... Please wait ...
....System is coming up ... Please wait ...
....System is coming up ... Please wait ...
....System is coming up ... Please wait ...
....System is coming up ... Please wait ...
....System is coming up ... Please wait ...
....System is coming up ... Please wait ...
....System is coming up ... Please wait ...
....System is coming up ... Please wait ...
....System is coming up ... Please wait ...
....System is coming up ... Please wait ...
....System is coming up ... Please wait ...
..bash-4.3# reboot -f
CISCO SWITCH Ver7.69
Switch G1
Device detected on 0:6:0 after 0 msecs
Device detected on 0:1:1 after 0 msecs
Device detected on 0:1:0 after 0 msecs
MCFrequency 1333Mhz
Relocated to memory
Time: 3/23/2025 16:5:56
Detected CISCO IOFPGA
Donner Present
MIFPGA Present
Code Signing Results: 0x0
Using Upgrade FPGA
Checking and setting PSU fan directions
Booting from Primary Bios
FPGA Revison : 0x6
FPGA ID : 0x1399840
FPGA Date : 0x20160719
Power Debug Register: 0x0
Reset Cause Register: 0x4
Boot Ctrl Register : 0xe0ff
FPGA Update Status : 0x20
Fixing BIOS Info
FPGA SPI Flash Micron(Numonyx) N25Q064A
Detected CISCO MIFPGA
FPGA Update Status : 0x20
Version 2.16.1240. Copyright (C) 2013 American Megatrends, Inc.
Board type 2
IOFPGA @ 0xc8000000
SLOT_ID @ 0xf
Standalone chassis
check_bootmode: grub: Continue grub
Trying to read config file /boot/grub/menu.lst.local from (hd0,0)
Filesystem type is fat, partition type 0xb
Trying to read config file /boot/grub/menu.lst.local from (hd1,4)
Filesystem type is ext2fs, partition type 0x83
Trying to read config file /boot/grub/menu.lst.local from (hd1,5)
Filesystem type is ext2fs, partition type 0x83
No autoboot or failed autoboot. falling to loader
Loader Version 7.69
loader > boot usb1:nxos.9.3.14.bin
Security Lock
Booting usb1:nxos.9.3.14.bin
Trying diskboot
Filesystem type is fat, partition type 0xb
Image valid
Image Signature verification was Successful.
Boot Time: 3/23/2025 16:7:28
Security Lock
Installing klm_card_index
done
Linking n9k flash devices
creating flash devices BOOT_DEV= sda
INIT: version 2.88 booting
Installing ata_piix module ... done.
Unsquashing rootfs ...
Total size needed in bootflash is 113348
check bootflash : OK
Total size needed in bootflash is 41340
check bootflash : OK
vm.min_free_kbytes = 174080
Enabling 8250 serial driver spurious INTs workaround
Installing isan procfs ... done.
Installing SSE module with card index 21027 ... done.
Creating SSE device node 247 ... done.
Inserting kernel_services module ... done.
Making kernel_services character devices
cgroups initialized
is_lxc: is_titan_eor: is_stby: suffix: klm_ftrace: /isanboot/lib/modules/klm_ftrace.o
Installing ftrace in non-lxc mode done
Checking SSD firmware ...
Loading I2C driver ... Model Number: Micron_M600_MTFDDAT064MBF
Serial Number: MSA20090GBE
Firmware Revision: MC04
done.
Loading MEM scrub driver ... done.
Installing CCTRL driver for card_type 21 without NEED_GEM ... done.
Loading IGB driver ...
Checking all filesystems.Warning: switch is starting up with default configuration
/bin/mount /dev/sdb1 /usbslot1 2>/dev/null
done.
Installing SPROM driver ... 21027 IS_N9K done.
Installing pfmsvcs module ...done.
Installing nvram module ... done.
Installing if_index module with port mode 6 ... done.
Installing fcfwd
Installing RNI lcnd ... done
Installing LC netdev ... done
Installing psdev module ... done.
Extracting rpms from image...
Installing veobc module ... done.
Checking SR card
Card Index is 21027
Inserting eMMC module ...
Inserting mtdphysmap module...Inserting OBFL module ... done.
Making OBFL character devices
mounting plog for N9k!
Mounting OBFL pstore for mtd
/
Unmounting USB drive
Calling install_tor for TOR
tune2fs 1.42.9 (28-Dec-2013)
Setting reserved blocks percentage to 0% (0 blocks)
Erasing 128 Kibyte @ 0 -- 0 % complete Removing any system startup links for cgroups-init ...
Adding system startup for /etc/init.d/cgroups-init.
Removing any system startup links for docker ...
Running groupadd commands...
NOTE: docker: Performing groupadd with [ -r docker]
Running useradd commands...
NOTE: docker: Performing useradd with [ -r -U -s /bin/false dockremap]
Erasing 128 Kibyte @ 20000 -- 100 % complete
Erasing 128 Kibyte @ 0 -- 0 % complete update-alternatives: Linking /bin/vi to /usr/bin/vim.vim
update-alternatives: Linking /usr/bin/vim to /usr/bin/vim.vim
update-alternatives: Linking /usr/bin/xxd to /usr/bin/xxd.vim
Erasing 128 Kibyte @ 60000 -- 75 % complete Starting portmap daemon...
creating NFS state directory: done
starting 8 nfsd kernel threads: done
starting mountd: done
starting statd: done
exit code: 1
Erasing 128 Kibyte @ 60000 -- 100 % complete
Saving image for img-sync ...
Installing local RPMS
Loading system software
Patch Repository Setup completed successfully
Creating /dev/mcelog
Starting mcelog daemon
INIT: Entering runlevel: 3
Running S93thirdparty-script...
/etc/init.d/rc:
Populating conf files for hybrid sysmgr ...
Starting hybrid sysmgr ...
done
Netbroker support IS present in the kernel.
done
Skipping EMON in TOR ...
Executing Prune clis.
ethernet switching mode Sun Mar 23 16:08:38 UTC 2025
Mar 23 16:08:41 %FW_APP-2-FIRMWARE_IMAGE_LOAD_SUCCESS No Firmware needed for Non SR card.
2025 Mar 23 16:08:45 %$ VDC-1 %$ %PLATFORM-2-PS_FAIL: Power supply 1 failed or shut down (Serial number DCB2022X3NM)
2025 Mar 23 16:08:45 %$ VDC-1 %$ %PLATFORM-2-PS_OK: Power supply 2 ok (Serial number DCB2022X3NO)
2025 Mar 23 16:08:45 %$ VDC-1 %$ %PLATFORM-2-PS_FANOK: Fan in Power supply 2 ok
2025 Mar 23 16:08:45 %$ VDC-1 %$ %PLATFORM-2-PS_ABSENT: Power supply 1 is absent/shutdown, ps-redundancy might be affected
2025 Mar 23 16:08:45 %$ VDC-1 %$ %PLATFORM-2-PS_RED_MODE_CHG: Power supply operational redundancy mode changed to non-redundant
2025 Mar 23 16:08:45 %$ VDC-1 %$ %PLATFORM-2-FANMOD_FAN_OK: Fan module 1 (Fan1(sys_fan1) fan) ok
2025 Mar 23 16:08:45 %$ VDC-1 %$ %PLATFORM-2-FANMOD_FAN_OK: Fan module 2 (Fan2(sys_fan2) fan) ok
2025 Mar 23 16:08:45 %$ VDC-1 %$ %PLATFORM-2-FANMOD_FAN_OK: Fan module 3 (Fan3(sys_fan3) fan) ok
2025 Mar 23 16:08:45 %$ VDC-1 %$ %PLATFORM-2-FANMOD_FAN_OK: Fan module 4 (Fan4(sys_fan4) fan) ok
2025 Mar 23 16:08:46 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: <<%USBHSD-2-MOUNT>> logflash: online - usbhsd
2025 Mar 23 16:08:46 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: <<%USBHSD-2-USB_SWAP>> USB insertion or removal detected - usbhsd
2025 Mar 23 16:08:47 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: <<%USBHSD-2-MOUNT>> USB1: online - usbhsd
2025 Mar 23 16:08:50 %$ VDC-1 %$ netstack: Registration with cli server complete
2025 Mar 23 16:09:07 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: ssnmgr_app_init called on ssnmgr up - aclmgr
2025 Mar 23 16:09:08 %$ VDC-1 %$ %VMAN-2-INSTALL_STATE: Installing virtual service 'guestshell+'
2025 Mar 23 16:09:17 %$ VDC-1 %$ %USER-0-SYSTEM_MSG: end of default policer - copp
2025 Mar 23 16:09:17 %$ VDC-1 %$ %COPP-2-COPP_NO_POLICY: Control-plane is unprotected.
2025 Mar 23 16:09:20 %$ VDC-1 %$ %CARDCLIENT-2-FPGA_BOOT_PRIMARY: IOFPGA booted from Primary
2025 Mar 23 16:09:20 %$ VDC-1 %$ %CARDCLIENT-2-FPGA_BOOT_PRIMARY: MIFPGA booted from Primary
2025 Mar 23 16:09:22 %$ VDC-1 %$ %VDC_MGR-2-VDC_ONLINE: vdc 1 has come online
Waiting for system online status before starting POAP ...
Waiting for system online status before starting POAP ...
2025 Mar 23 16:09:52 switch %$ VDC-1 %$ %VMAN-2-INSTALL_STATE: Install success virtual service 'guestshell+'; Activating
2025 Mar 23 16:09:52 switch %$ VDC-1 %$ %VMAN-2-ACTIVATION_STATE: Activating virtual service 'guestshell+'
Waiting for system online status before starting POAP ...
2025 Mar 23 16:10:35 switch %$ VDC-1 %$ %USER-1-SYSTEM_MSG: SWINIT failed. devid:241 inst:0 - t2usd
2025 Mar 23 16:10:35 switch %$ VDC-1 %$ %MODULE-2-MOD_DIAG_FAIL: Module 1 (Serial number: SAL2023RCXL) reported failure on ports Ethernet1/1-48 (Ethernet) due to DEVICE POST/DIAG FAILURE in device DEV_BCM56850_ASIC (device error 0x2)
