N7K Line cards error prompted "Reset (powered-down) because module does not boot"

[JohnJacob26941]

Hello all,

Line cards error prompted "Reset (powered-down) because module does not boot"
-Swapped supervisors and fabric card modules with N7K-C7010-FAB-2 to N7K-C7010-FAB-1
-Power down after 10 minutes without any reason

-It may be caused by one of the following bug or actual faulty units linked below

https://us-west-2.protection.sophos.com?d=cisco.com&u=aHR0cHM6Ly93d3cuY2lzY28uY29tL2MvZW4vdXMvc3VwcG9ydC9kb2NzL2ZpZWxkLW5vdGljZXMvNjM3L2ZuNjM3NTEuaHRtbA==&i=NWQ0YWJlOTM5ODg3ODgxMWM0YWNhMTMz&t=YWpkTW8xYWlDSU5ZNWdBeitXdUppa3oxdXFnT2pyV2RJOG55ZXBXOVJybz...

https://us-west-2.protection.sophos.com?d=cisco.com&u=aHR0cHM6Ly9ic3QuY2xvdWRhcHBzLmNpc2NvLmNvbS9idWdzZWFyY2gvYnVnL0NTQ3V1NTg3NTAvP3Jmcz1pcXZyZWQ=&i=NWQ0YWJlOTM5ODg3ODgxMWM0YWNhMTMz&t=N2xiYzZHVXdvNjNsbWZXMThqZVcvbXk4eHlEV0hDS2dZTXRzQU5ibHByZz0=&h=49c...

Output from CLI

switch# show mod
Mod Ports Module-Type Model Status
--- ----- ----------------------------------- ------------------ ----------
1 32 10 Gbps Ethernet Module N7K-M132XP-12 powered-dn
2 32 10 Gbps Ethernet Module N7K-M132XP-12 powered-dn
3 48 10/100/1000 Mbps Ethernet Module N7K-M148GT-11 powered-dn
4 48 10/100/1000 Mbps Ethernet Module N7K-M148GT-11 powered-dn
6 0 Supervisor module-2 N7K-SUP2 active *
7 48 10/100/1000 Mbps Ethernet Module N7K-M148GT-11 powered-dn
8 48 10/100/1000 Mbps Ethernet Module N7K-M148GT-11 powered-dn

Mod Power-Status Reason
--- ------------ ---------------------------
1 powered-dn Reset (powered-down) because module does not boot
2 powered-dn Reset (powered-down) because module does not boot
3 powered-dn Reset (powered-down) because module does not boot
4 powered-dn Reset (powered-down) because module does not boot
7 powered-dn Reset (powered-down) because module does not boot
8 powered-dn Reset (powered-down) because module does not boot

Mod Sw Hw
--- -------------- ------
6 6.1(2) 3.0

Mod MAC-Address(es) Serial-Num
--- -------------------------------------- ----------
1 00-00-00-00-00-00 to 00-00-00-00-00-00 JAF1408BGAN
2 00-00-00-00-00-00 to 00-00-00-00-00-00 JAF1408AKRT
3 00-00-00-00-00-00 to 00-00-00-00-00-00 JAF1413CJCN
4 00-00-00-00-00-00 to 00-00-00-00-00-00 JAF1417AJDE
6 e4-c7-22-19-37-4c to e4-c7-22-19-37-5e JAF1804AHJR
7 00-00-00-00-00-00 to 00-00-00-00-00-00 JAF1414CMSK
8 00-00-00-00-00-00 to 00-00-00-00-00-00 JAF1419DEMA

Mod Online Diag Status
--- ------------------
6 Pass

Xbar Ports Module-Type Model Status
--- ----- ----------------------------------- ------------------ ----------
1 0 Fabric Module 1 N7K-C7010-FAB-1 ok
2 0 Fabric Module 1 N7K-C7010-FAB-1 ok
3 0 Fabric Module 1 N7K-C7010-FAB-1 ok
4 0 Fabric Module 1 N7K-C7010-FAB-1 ok
5 0 Fabric Module 1 N7K-C7010-FAB-1 ok

Is there a hardware/software compatibility issue that I'm missing?

When I power them up a few minutes later I'm prompted with the following errors;

switch# 2021 Jun 24 17:40:57 switch %$ VDC-1 %$ %PLATFORM-2-MOD_DETECT: Module 1 detected (Serial number JAF1408BGAN
) Module-Type 10 Gbps Ethernet Module Model N7K-M132XP-12
2021 Jun 24 17:40:57 switch %$ VDC-1 %$ %PLATFORM-2-MOD_PWRUP: Module 1 powered up (Serial number JAF1408BGAN)
2021 Jun 24 17:41:04 switch %$ VDC-1 %$ %PLATFORM-2-MOD_DETECT: Module 2 detected (Serial number JAF1408AKRT) Module -Type 10 Gbps Ethernet Module Model N7K-M132XP-12
2021 Jun 24 17:41:04 switch %$ VDC-1 %$ %PLATFORM-2-MOD_PWRUP: Module 2 powered up (Serial number JAF1408AKRT)
2021 Jun 24 17:45:59 switch %$ VDC-1 %$ %MODULE-2-MOD_FAIL: Initialization of module 1 (Serial number: JAF1408BGAN) failed
2021 Jun 24 17:45:59 switch %$ VDC-1 %$ last message repeated 1 time
2021 Jun 24 17:45:59 switch %$ VDC-1 %$ %PLATFORM-2-MOD_PWRDN: Module 1 powered down (Serial number JAF1408BGAN)
2021 Jun 24 17:46:06 switch %$ VDC-1 %$ %MODULE-2-MOD_FAIL: Initialization of module 2 (Serial number: JAF1408AKRT) failed
2021 Jun 24 17:46:07 switch %$ VDC-1 %$ last message repeated 1 time
2021 Jun 24 17:46:07 switch %$ VDC-1 %$ %PLATFORM-2-MOD_PWRDN: Module 2 powered down (Serial number JAF1408AKRT)

Any suggestions?

[marce1000]

                                  >Any suggestions?

  Ref : https://www.cisco.com/c/en/us/support/docs/field-notices/637/fn63751.html

 >...

    Workaround/Solution

Normal SmartNet and warranty entitlement rules remain in place and are applied by the Cisco Technical Assistance Center (TAC) if you experience a failure in one of the products listed in the Field Notice. In other circumstances, such as out of warranty or out of contract, Cisco encourages you to raise your concern directly with your Cisco account team.

Fix on Failure Replacement Guidelines: Request Return Material Authorization (RMA) product through normal service support channels.

If you need assistance in order to determine which hardware part(s) might need replacement, consult the error messages documented in the Problem Symptom section.

For assistance with replacement part disposition, reference this table. In cases where replacing the memory DIMM and/or daughter card is not a viable option, a request may be made to replace the entire card.

Refer to this documentation for assistance on memory modules replacement:

>...

 M.

[Peter Paluch]

John,

It would be too soon to assume that you are hitting a field notice. The fact that all your 6 cards are affected at once makes it simply too suspicious.

Can you please clarify what exactly did you do with the switch between the last time the switch worked, and the first time it failed like this? If I understand you correctly, you replaced all FAB2 modules with FAB1 and that seems to have triggered the problem - is that correct? If not, please clarify.

I would like to ask you to share a number of outputs from your switch - they will be long and they need to be captured to a file and attached here. I strongly recommend capturing them while accessing the switch through the mgmt0 connection because the CLI output could take minutes, if not hours, to complete. They are:

terminal length 0
terminal width 511
show version
show inventory
show logging log
show logging nvram
show logging onboard
show logging onboard module 1
show module internal exceptionlog
show module internal all
show hardware internal error all
terminal no width
terminal no length

This would be a good place to start.

Best regards,
Peter