Re: Nexus 2148 ASIC failure

Marko Pribanic · ‎02-28-2011

Hello,

I have one problem concerning FEX which is single homed to one Nexus 5020.

Recently I did the upgrade to latest (5.0(2)N2(1)) NX-OS version.

I did the upgrade on two 55020 and six 2148 which are connected to 5k’s.

I wasn’t able to do ISSU because LACP fast timers misconfig, so I did the disruptive upgrade.

Everything went fine, but after the upgrade on FEX 106 I noticed the following log:

2011 Feb 24 14:35:13 swDC-NX5k-SS4-R5OR3-2 %SATCTRL-FEX106-2-SATCTRL: FEX-106 Module 1: Cold boot

2011 Feb 24 14:35:16 swDC-NX5k-SS4-R5OR3-2 %SATCTRL-FEX106-2-SOHMS_DIAG_ERROR: FEX-106 Module 1: Runtime diag detected major event: Forwarding ASIC failure: Ethernet106/1/10 Ethernet106/1/9 Ethernet106/1/12 Ethernet106/1/11 Ethernet106/1/14 Ethernet106/1/13 Ethernet106/1/16 Ethernet106/1/15

2011 Feb 24 14:40:08 swDC-NX5k-SS4-R5OR3-2 %SATCTRL-FEX106-2-SOHMS_DIAG_ERROR: FEX-106 Module 1: Bootup diag detected major event: Forwarding ASIC failure: Ethernet106/1/10 Ethernet106/1/9 Ethernet106/1/12 Ethernet106/1/11 Ethernet106/1/14 Ethernet106/1/13 Ethernet106/1/16 Ethernet106/1/15

2011 Feb 24 14:40:08 swDC-NX5k-SS4-R5OR3-2 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel205: Ethernet102/1/5 is up

2011 Feb 24 14:40:08 swDC-NX5k-SS4-R5OR3-2 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel205: first operational port changed from none to Ethernet102/1/5

2011 Feb 24 14:40:08 swDC-NX5k-SS4-R5OR3-2 %ETHPORT-5-IF_UP: Interface Ethernet102/1/5 is up in mode access

2011 Feb 24 14:40:08 swDC-NX5k-SS4-R5OR3-2 %ETHPORT-5-IF_UP: Interface port-channel205 is up in mode access

2011 Feb 24 14:40:13 swDC-NX5k-SS4-R5OR3-2 %ETHPORT-5-IF_DOWN_HW_FAILURE: Interface Ethernet106/1/9 is down (Hardware Failure)

2011 Feb 24 14:40:13 swDC-NX5k-SS4-R5OR3-2 %ETHPORT-5-IF_DOWN_HW_FAILURE: Interface Ethernet106/1/10 is down (Hardware Failure)

2011 Feb 24 14:40:13 swDC-NX5k-SS4-R5OR3-2 %ETHPORT-5-IF_DOWN_HW_FAILURE: Interface Ethernet106/1/11 is down (Hardware Failure)

2011 Feb 24 14:40:13 swDC-NX5k-SS4-R5OR3-2 %ETHPORT-5-IF_DOWN_HW_FAILURE: Interface Ethernet106/1/12 is down (Hardware Failure)

2011 Feb 24 14:40:13 swDC-NX5k-SS4-R5OR3-2 %ETHPORT-5-IF_DOWN_HW_FAILURE: Interface Ethernet106/1/13 is down (Hardware Failure)

2011 Feb 24 14:40:13 swDC-NX5k-SS4-R5OR3-2 %ETHPORT-5-IF_DOWN_HW_FAILURE: Interface Ethernet106/1/14 is down (Hardware Failure)

2011 Feb 24 14:40:13 swDC-NX5k-SS4-R5OR3-2 %ETHPORT-5-IF_DOWN_HW_FAILURE: Interface Ethernet106/1/15 is down (Hardware Failure)

2011 Feb 24 14:40:13 swDC-NX5k-SS4-R5OR3-2 %ETHPORT-5-IF_DOWN_HW_FAILURE: Interface Ethernet106/1/16 is down (Hardware Failure)

After that the ports 106/1/9-16 were not operational:

swDC-NX5k-SS4-R5OR3-2# sh diagnostic result fex 106

FEX-106: N2K-C2148T-1GE/Supervisor SerialNo: JAF1419BLGQ

Overall Diagnostic Result for FEX-106 : OK

Test results: (. = Pass, F = Fail, U = Untested)

TestPlatform:

0) SPROM: ---------------> .

1) MV88E6095: ---------------> .

2) Fan: ---------------> .

3) Power Supply: ---------------> .

4) Temperature Sensor: ---------------> .

TestForwardingPorts:

Eth 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Port ------------------------------------------------------------------------

. . . . . . . . F F F F F F F F . . . . . . . .

Eth 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

Port ------------------------------------------------------------------------

. . . . . . . . . . . . . . . . . . . . . . . .

TestFabricPorts:

Fabric 1 2 3 4

Port ------------

. . . .

Then I did the reboot of FEX 106 and everything went back to normal:

FEX-106: N2K-C2148T-1GE/Supervisor SerialNo: JAF1419BLGQ

Overall Diagnostic Result for FEX-106 : OK

Test results: (. = Pass, F = Fail, U = Untested)

TestPlatform:

0) SPROM: ---------------> .

1) MV88E6095: ---------------> .

2) Fan: ---------------> .

3) Power Supply: ---------------> .

4) Temperature Sensor: ---------------> .

TestForwardingPorts:

Eth 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Port ------------------------------------------------------------------------

. . . . . . . . . . . . . . . . . . . . . . . .

Eth 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

Port ------------------------------------------------------------------------

. . . . . . . . . . . . . . . . . . . . . . . .

TestFabricPorts:

Fabric 1 2 3 4

Port ------------

. . . .

So I am wondering if I am experiencing a hardware failure, because ASIC failures are generally related to hardware failures, or this is just a consequence of a software upgread?

Are there some other useful “show” or “debug” commands in discovering the cause(s) of this issue?

Tnx

Leo Laohoo · ‎02-28-2011

Contact Cisco TAC and arrange for an RMA. Nothing else you can do.

vdsudame · ‎03-01-2011

yes, it could be a transient hardware problem with some fex ports, it might seem coincidental that upgrade could have caused it.

Marko Pribanic · ‎03-07-2011

Thank you for your suggestions.

I was hoping to avoid opening the TAC case for this issue, or doing the RMA.

For the past 7 days I didn’t have any problems regarding this FEX, so I will leave this setup as is and in the case that this problem reoccurs then I will contact the TAC.

jkarson · ‎06-29-2011

Hi Marko,

Great assistance here. I followed your lead and a reboot was the resolution. The shortcoming was less than 48 hours later for us the problem began again. We'll need to replace this fex with an RMA but do appreciate the info you've provided.

Thanks,

Jay K.