Re: Nexus 7700 100G CPAK Issue

John.Mayer · ‎08-16-2020

Dear All

Hello

actually we have to use 100G module on Cisco Nexus 7710 with two SUP2E, but module not boot properly and shows powered-dn in show module

module shows below logs when we put it on chassis

020 Jul 29 08:35:40 switch %$ VDC-1 %$ %USER-2-SYSTEM_MSG: <<%EPLD_AUTO-2-AUTO_UPGRADE_CHECK>> Automatic EPLD upgrade check for module 1: EPLD versions are up to date. - epld_auto
2020 Jul 29 08:36:25 switch %$ VDC-1 %$ %MODULE-2-MOD_SOMEPORTS_FAILED: Module 1 (Serial number: JAE203306ZT) reported failure on ports Ethernet1/41-48 (Ethernet) due to non-fatal error in device DEV_FLN_FWD (device error 0xcbb05200)
2020 Jul 29 08:36:25 switch %$ VDC-1 %$ %MODULE-2-MOD_SOMEPORTS_FAILED: Module 1 (Serial number: JAE203306ZT) reported failure on ports Ethernet1/25-32 (Ethernet) due to non-fatal error in device DEV_FLN_FWD (device error 0xcbb03200)
2020 Jul 29 08:36:53 switch %$ VDC-1 %$ %MODULE-2-MOD_DIAG_FAIL: Module 1 (Serial number: JAE203306ZT) reported failure Ethernet1/1-8due to Local serial link syncing exception in device DEV_LOCAL_SAC_ASIC (device error 0xc910020c)
2020 Jul 29 08:36:55 switch %$ VDC-1 %$ %MODULE-2-MOD_FAIL: Initialization of module 1 (Serial number: 0) failed
2020 Jul 29 08:36:55 switch %$ VDC-1 %$ %MODULE-2-MOD_FAIL: Initialization of module 1 (Serial number: 0) failed
2020 Jul 29 08:36:55 switch %$ VDC-1 %$ %PLATFORM-2-MOD_PWRDN: Module 1 powered down (Serial number 0)

Extra Info:
Module Part Number: N77-F312CK-26
Sup: SUP2E
Chassis Part Number: Nexus 7710
Number of PWR: 6
Number of FAN: 3
Number of FAB: 4

I try to upgrade NX-OS and EPLD and try different version of the from 6 to 8, but result was the same

does anyone have the same problem on 100G CPAK cards

Thank You In Advance

John

Sergiu.Daniluk · ‎08-16-2020

Hi @John.Mayer

The log errors indicate an issue with both ASICs of your 100G Module, as well as with the XBAR:

2020 Jul 29 08:36:53 switch %$ VDC-1 %$ %MODULE-2-MOD_DIAG_FAIL: Module 1 (Serial number: JAE203306ZT) reported failure Ethernet1/1-8due to Local serial link syncing exception in device DEV_LOCAL_SAC_ASIC (device error 0xc910020c)
N7K-1# show system error-id 0xc910020c
Error Description: Device Name:[Sacramento Xbar ASIC] Instance:[0] Error Type:[hw error] code:[12]

2020 Jul 29 08:36:25 switch %$ VDC-1 %$ %MODULE-2-MOD_SOMEPORTS_FAILED: Module 1 (Serial number: JAE203306ZT) reported failure on ports Ethernet1/41-48 (Ethernet) due to non-fatal error in device DEV_FLN_FWD (device error 0xcbb05200)
N7K-1# show system error-id 0xcbb03200
Error Description: Device Name:[Flanker Fwd Driver] Instance:[3] Error Type:[hw error] code:[0]

How many line cards and how many xbars do you have inserted in your N7700 chassis?

Was the 100G module working before?

What changes have you done before issue started?

Can you share the full show module output? (feel free to remove the SNs for the safe of privacy)

Stay safe,

Sergiu

John.Mayer · ‎08-17-2020

Dear Sergiu

Hello

first of all thanks for you fast reply

second, this is my first two 100G module and not used before on this chassis. i try to install 100G modules on two different chassis

btw, please check the show module and show environment below:

switch# show module
Mod Ports Module-Type Model Status
--- ----- ----------------------------------- ------------------ ----------
1 12 100 Gbps Ethernet Module N77-F312CK-26 powered-dn
5 0 Supervisor Module-2 N77-SUP2E active *

Mod Power-Status Reason
--- ------------ ---------------------------
1 powered-dn Reset (powered-down) because module does not boot

Mod Sw Hw
--- --------------- ------
5 7.3(3)D1(1) 1.1

Mod MAC-Address(es) Serial-Num
--- -------------------------------------- ----------
1 00-00-00-00-00-00 to 00-00-00-00-00-00 0000000
5 3c-0e-23-c4-10-ab to 3c-0e-23-c4-10-bd 0000000

Mod Online Diag Status
--- ------------------
5 Pass

Xbar Ports Module-Type Model Status
--- ----- ----------------------------------- ------------------ ----------
3 0 Fabric Module 2 N77-C7710-FAB-2 ok
4 0 Fabric Module 2 N77-C7710-FAB-2 ok
5 0 Fabric Module 2 N77-C7710-FAB-2 ok
6 0 Fabric Module 2 N77-C7710-FAB-2 ok

Xbar Sw Hw
--- --------------- ------
3 NA 1.1
4 NA 1.1
5 NA 1.1
6 NA 1.1

Xbar MAC-Address(es) Serial-Num
--- -------------------------------------- ----------
3 NA 0000000
4 NA 0000000
5 NA 0000000
6 NA 0000000

* this terminal session
switch#

switch# show environment
Power Supply:
Voltage: 50 Volts
Power Actual Total
Supply Model Output Capacity Status
(Watts ) (Watts )
------- ------------------- ----------- ----------- --------------
1 N77-AC-3KW 258 W 3000 W Ok
2 N77-AC-3KW 0 W 0 W Shutdown
3 N77-AC-3KW 0 W 0 W Shutdown
4 N77-AC-3KW 0 W 0 W Shutdown
5 N77-AC-3KW 261 W 3000 W Ok
6 N77-AC-3KW 0 W 0 W Shutdown
7 ------------ 0 W 0 W Absent
8 ------------ 0 W 0 W Absent

Actual Power
Module Model Draw Allocated Status
(Watts ) (Watts )
------- ------------------- ----------- ----------- --------------
1 N77-F312CK-26 N/A 0 W Powered-Dn
5 N77-SUP2E 139 W 265 W Powered-Up
6 supervisor N/A 0 W Absent
Xb1 xbar N/A 150 W Absent
Xb2 xbar N/A 150 W Absent
Xb3 N77-C7710-FAB-2 57 W 150 W Powered-Up
Xb4 N77-C7710-FAB-2 57 W 150 W Powered-Up
Xb5 N77-C7710-FAB-2 58 W 150 W Powered-Up
Xb6 N77-C7710-FAB-2 58 W 150 W Powered-Up
fan1 N77-C7710-FAN 40 W 600 W Powered-Up
fan2 N77-C7710-FAN 45 W 600 W Powered-Up
fan3 N77-C7710-FAN 45 W 600 W Powered-Up

N/A - Per module power not available

Power Usage Summary:
--------------------
Power Supply redundancy mode (configured) Non-Redundant(combined)
Power Supply redundancy mode (operational) Non-Redundant(combined)

Total Power Capacity (based on configured mode) 6000 W
Total Power of all Inputs (cumulative) 6000 W
Total Power Output (actual draw) 519 W
Total Power Allocated (budget) 3230 W
Total Power Available for additional modules 2770 W

Clock:
----------------------------------------------------------
Clock Model Hw Status
----------------------------------------------------------
A Clock Module -- NotSupported/None
B Clock Module -- NotSupported/None

Fan:
------------------------------------------------------
Fan Model Hw Status
------------------------------------------------------
Fan1(sys_fan1) N77-C7710-FAN 1.0 Ok
Fan2(sys_fan2) N77-C7710-FAN 1.0 Ok
Fan3(sys_fan3) N77-C7710-FAN 1.0 Ok
Fan_in_PS1 -- -- Ok
Fan_in_PS2 -- -- Shutdown
Fan_in_PS3 -- -- Shutdown
Fan_in_PS4 -- -- Shutdown
Fan_in_PS5 -- -- Ok
Fan_in_PS6 -- -- Shutdown
Fan_in_PS7 -- -- Absent
Fan_in_PS8 -- -- Absent
Fan Zone Speed: Zone 1: 0x5e

Temperature:
--------------------------------------------------------------------
Module Sensor MajorThresh MinorThres CurTemp Status
(Celsius) (Celsius) (Celsius)
--------------------------------------------------------------------
5 Inlet (s1) 60 42 25 Ok
5 Crossbar(s2) 125 115 82 Ok
5 L2L3Dev1(s3) 125 110 51 Ok
5 Arbiter (s4) 125 105 62 Ok
5 CPU1CORE1(s5) 85 75 51 Ok
5 CPU1CORE2(s6) 85 75 49 Ok
5 CPU1CORE3(s7) 85 75 50 Ok
5 CPU1CORE4(s8) 85 75 52 Ok
5 CPU2CORE1(s9) 85 75 35 Ok
5 CPU2CORE2(s10) 85 75 31 Ok
5 CPU2CORE3(s11) 85 75 34 Ok
5 CPU2CORE4(s12) 85 75 34 Ok
5 DDR3DIMM1(s13) 95 85 43 Ok
5 DDR3DIMM2(s14) 95 85 41 Ok
5 DDR3DIMM4(s16) 95 85 32 Ok
5 DDR3DIMM5(s17) 95 85 31 Ok
xbar-3 Crossbar1(s1) 125 115 39 Ok
xbar-3 Crossbar2(s2) 125 115 38 Ok
xbar-4 Crossbar1(s1) 125 115 43 Ok
xbar-4 Crossbar2(s2) 125 115 40 Ok
xbar-5 Crossbar1(s1) 125 115 45 Ok
xbar-5 Crossbar2(s2) 125 115 39 Ok
xbar-6 Crossbar1(s1) 125 115 45 Ok
xbar-6 Crossbar2(s2) 125 115 42 Ok

John.Mayer · ‎08-22-2020

@Sergiu.Daniluk

Sergiu.Daniluk · ‎08-23-2020

Hi @John.Mayer

I can see that the xbars are installed like this:

Xb1 xbar N/A 150 W Absent
Xb2 xbar N/A 150 W Absent
Xb3 N77-C7710-FAB-2 57 W 150 W Powered-Up
Xb4 N77-C7710-FAB-2 57 W 150 W Powered-Up
Xb5 N77-C7710-FAB-2 58 W 150 W Powered-Up
Xb6 N77-C7710-FAB-2 58 W 150 W Powered-Up

Could you move xbar4 or xbar 6 to slot1?

If there are only three fabric modules to install, install them in fabric slots 1, 3, and 5.

Ref: https://www.cisco.com/c/en/us/td/docs/switches/datacenter/hw/nexus7000/installation/guide/b_n7710_hardware_install_guide/b_n7710_hardware_install_guide_chapter_011.html

After that, reseat the module1 and see if this changes anything.

You also mentioned that you tried the module in different chassis, right? Did you noticed the same results (logs)?

Cheers.

Sergiu

John.Mayer · ‎08-23-2020

Hi @Sergiu.Daniluk

as i remember i test the xbars on different slots, but let me check again

and about the different chassis, yes the result was the same