Nexus 3000 C3064PQ crashes after upgrade

[Surgeon]

Hi all,

Please help me with an issue after upgrading my Nexus 3000 N3K-C3064PQ-10GX

I've already upgrades some of them and I found no issue. I am followed exactly the same steps on all switches. All are same model and hardware revision. However I am facing boot loop after upgrading two of them.

Original version running nxos.7.0.3.I7.9.bin

Target version: nxos.9.3.7.bin

I compact image on USB flash drive and then install compacted image to the switch. After reboot the switch falls into boot loop with following error

switch login: 2021 Oct  9 05:39:27 switch %$ VDC-1 %$ %USER-2-SYSTEM_MSG: - E - Failed Opening Device rc=-4 - neutron_usd
2021 Oct  9 05:39:42 switch %$ VDC-1 %$ %USER-2-SYSTEM_MSG:  retry failed. Exiting. retry_count =5 - neutron_usd
2021 Oct  9 05:39:42 switch %$ VDC-1 %$ %USER-2-SYSTEM_MSG: neutron_chip_init failed. Retry after  neutron_reset rc=-1 - neutron_usd
2021 Oct  9 05:39:47 switch %$ VDC-1 %$ %USER-2-SYSTEM_MSG: - E - Failed Opening Device rc=-4 - neutron_usd
2021 Oct  9 05:40:02 switch %$ VDC-1 %$ %USER-2-SYSTEM_MSG:  retry failed. Exiting. retry_count =5 - neutron_usd
2021 Oct  9 05:40:02 switch %$ VDC-1 %$ %USER-2-SYSTEM_MSG: neutron_chip_init failed again. Aborting. - neutron_usd
2021 Oct  9 05:40:03 switch %$ VDC-1 %$ %SYSMGR-SLOT1-2-SERVICE_CRASHED: Service "neutron_usd" (PID 13778) hasn't caught signal 6 (core will be saved).
2021 Oct  9 05:40:03 switch %$ VDC-1 %$ %SYSMGR-SLOT1-2-HAP_FAILURE_SUP_RESET: Service "neutron_usd" in vdc 1 has had a hap failure
2021 Oct  9 05:40:03 switch %$ VDC-1 %$ %SYSMGR-SLOT1-2-LAST_CORE_BASIC_TRACE: fsm_action_become_offline: PID 6943 with message Could not turn off console logging on vdc 1 error: mts req-response with syslogd in vdc 1 failed (0xFFFFFFFF)  .
2021 Oct  9 05:40:03 switch %$ VDC-1 %$ %SYSMGR-SLOT1-2-HAP_FAILURE_SUP_RESET: Service "neutron_usd" in vdc 1 has had a hap failure
2021 Oct  9 05:40:03 switch %$ VDC-1 %$ %SYSMGR-SLOT1-2-LAST_CORE_BASIC_TRACE: fsm_action_become_offline: PID 6943 with message Could not turn off console logging on vdc 1 error: mts req-response with syslogd in vdc 1 failed (0xFFFFFFFF)  .
2021 Oct  9 05:40:27 switch %$ VDC-1 %$ %SYSMGR-SLOT1-2-LAST_CORE_BASIC_TRACE: core_client_main: PID 14133 with message filename = 0x102_neutron_usd_log.13778.tar.gz .

Please help to understand what is going on.

Regards,

[Sergiu.Daniluk]

Try getting into loader, boot up previous image and see if the switch comes up.

If yes, then you are facing a bug. Check what is different between the failing switches and the working ones (hardware, config differences etc).

Stay safe.

Sergiu

[Surgeon]

It is working fine with older version. It is not config issue, we tried to reset config completely. I checked and find some hardware difference

This is older one is working fine with 9.3.7

Switch is booted up
  Switch type is : Nexus3000 C3064PQ Chassis
  Model number is N3K-C3064PQ-10GX
  H/W version is 1.3
  Part Number is 68-4363-03
  Part Revision is F0

 The new one does not want to run with 9.3.7

Switch type is : Nexus3000 C3064PQ Chassis
  Model number is N3K-C3064PQ-10GX
  H/W version is 1.6
  Part Number is 68-4363-03
  Part Revision is M0

I could not find anything related neither in release notes nor in bug search. Looks like a bug