Solved: NEXUS 7000 Octopus internal error in device 78 message

Rosa Ladeira · ‎11-13-2012

Has someone heard of NEXUS 7000 Octopus internal error in device 78 message. I got this on NEXUS 7000 logs:

Nov 12 22:05:14 smale-outside : 2012 Nov 12 21:05:14 BRST: %MODULE-2-MOD_DIAG_FAIL: Module 2 (serial: JAF1548AMKB) reported failure on ports 2/1-2/32 (Ethernet) due to Octopus internal error in device 78 (device error 0xc4e0025b)

Nov 12 22:05:17 smale-outside : 2012 Nov 12 21:05:17 BRST: %MODULE-2-MOD_FAIL: Initialization of module 2 (serial: JAF1548AMKB) failed

Nov 12 22:05:18 smale-outside : 2012 Nov 12 21:05:18 BRST: %MODULE-2-MOD_FAIL: Initialization of module 2 (serial: JAF1548AMKB) failed

and :

nx7000-wc0# sh module

Mod Ports Module-Type Model Status

--- ----- -------------------------------- ------------------ ------------

1 32 10 Gbps Ethernet Module N7K-M132XP-12 ok

2 32 10 Gbps Ethernet Module N7K-M132XP-12 powered-dn

5 0 Supervisor module-1X N7K-SUP1 active *

10 48 10/100/1000 Mbps Ethernet Module N7K-M148GT-11 ok

Mod Power-Status Reason

--- ------------ ---------------------------

2 powered-dn Reset (powered-down) because module does not boot

nx7000-wc0# sh version

Cisco Nexus Operating System (NX-OS) Software

TAC support: http://www.cisco.com/tac

Documents: http://www.cisco.com/en/US/products/ps9372/tsd_products_support_serie

s_home.html

The copyrights to certain works contained in this software are

owned by other third parties and used and distributed under

license. Certain components of this software are licensed under

the GNU General Public License (GPL) version 2.0 or the GNU

Lesser General Public License (LGPL) Version 2.1. A copy of each

such license is available at

http://www.opensource.org/licenses/gpl-2.0.php and

http://www.opensource.org/licenses/lgpl-2.1.php

Software

BIOS: version 3.22.0

kickstart: version 5.1(1)

system: version 5.1(1)

BIOS compile time: 02/20/10

kickstart image file is: bootflash:///n7000-s1-kickstart-npe.5.1.1.bin

kickstart compile time: 12/25/2020 12:00:00 [10/24/2010 08:25:25]

system image file is: bootflash:///n7000-s1-dk9-npe.5.1.1.bin

system compile time: 9/2/2010 19:00:00 [10/24/2010 09:34:48]

Hardware

cisco Nexus7000 C7010 (10 Slot) Chassis ("Supervisor module-1X")

Intel(R) Xeon(R) CPU with 4109596 kB of memory.

Processor Board ID JAF1444BLKS

Device name: nx7000-wc0

bootflash: 2029608 kB

slot0: 0 kB (expansion flash)

Kernel uptime is 222 day(s), 4 hour(s), 9 minute(s), 22 second(s)

Last reset

Reason: Unknown

System version: 5.1(1)

Service:

plugin

Core Plugin, Ethernet Plugin

CMP (Module 5) ok

CMP Software

CMP BIOS version: 02.01.05

CMP Image version: 5.1(1) [build 5.0(0.66)]

CMP BIOS compile time: 8/ 4/2008 19:39:40

CMP Image compile time: 9/2/2010 19:00:00

nx7000-wc0#

I only have found:

http://www.cisco.com/web/software/datacenter/N7K/nxos_521_sw_advisory.html

which does not match reality : No one was configuring Vlans.

Arumugam Muthaiah · ‎11-13-2012

Hi Rosa,

You are having same problem, please try to reload the module 2 and check the performance.

Regards,

Aru

*** Please rate if the post is useful ***

Regards, Aru *** Please rate if the post useful ***

View solution in original post

Rosa Ladeira · ‎11-13-2012

In fact, a new Vlan has been added ! But version is 5.1

Problem Description

For Cisco Nexus 7000 series switches running NX-OS 5.2(1), M1-series modules can reset or link state across multiple ports can flap after configuring a new VLAN with SPAN. A full list of all M1-series modules is provided below:
N7K-M148GT-11=
N7K-M132XP-12=
N7K-M148GS-11=
N7K-M148GS-11L=
N7K-M108X2-12L=
N7K-M132XP-12L=
N7K-M148GT-11L=
Please note that this is a software caveat (CSCtt43115) and not a hardware issue on the M1-series modules.

Background

This issue only occurs on Cisco Nexus 7000 series switches running NX-OS 5.2(1) when a TX SPAN session is configured with the destination port as a trunk port. The SPAN destination port could be in either an M1 or F1 module.

Problem Symptoms

The following error messages can be seen when the issue occurs:

%MODULE-2-MOD_DIAG_FAIL: Module X (serial: ) reported failure on ports X/1-X/48 (Ethernet) due to Octopus internal
 error in device 78  (device error [ErrCode])

Workaround/Solution

There are two methods to workaround this issue:
1) Remove all the monitor sessions, configure VLAN(s) and restore all monitor session configurations.
2) Configure the SPAN destination port as an access port instead of a trunk port.
Either of the above options should be repeated every time a new VLAN needs to be configured.
This issue is applicable only to the NX-OS 5.2(1) release. This issue is NOT applicable to pre-5.2 releases or 5.2(3a) and later.
Additional Workaround Information:
If the affected Nexus 7000 module(s) are powered down after a few reloads, then follow the steps mentioned below to recover the modules (relevant CLI commands are provided):
1) Remove the newly added VLAN.

2) Remove the monitor session(s).
N7K-CORE(config)#no monitor session 1
3) Manually reload the M1 module(s) from the default VDC.
N7K#reload module [mod#]
4) Once the modules are up, add the new VLANs.

DDTS

To follow the bug ID link below and see detailed bug information, you must be a registered customer and you must be logged in.

DDTS Description
{CSCtt43115} (registered customers only) N7K/5.2(1) Adding a new vlan reloads M1-series linecards

In this case

Software

BIOS: version 3.22.0

kickstart: version 5.1(1)

system: version 5.1(1)

BIOS compile time: 02/20/10

kickstart image file is: bootflash:///n7000-s1-kickstart-npe.5.1.1.bin

kickstart compile time: 12/25/2020 12:00:00 [10/24/2010 08:25:25]

system image file is: bootflash:///n7000-s1-dk9-npe.5.1.1.bin

system compile time: 9/2/2010 19:00:00 [10/24/2010 09:34:48]

Hardware

cisco Nexus7000 C7010 (10 Slot) Chassis ("Supervisor module-1X")

Intel(R) Xeon(R) CPU with 4109596 kB of memory.

Processor Board ID JAF1444BLKS

Arumugam Muthaiah · ‎11-13-2012

Hi Rosa,

As you mentioned previously, this issue is applicable only to 5.2(1) release not in 5.1(1)

Octopus is one of the internal ASIC avialble on module N7K-M132XP-12. There are two Octopus ASICs are mapped to multiple interfaces.

1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31 - All even ports are mapped to Octopus 1

2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32 - All odd ports are mapped to Octopus 0

Our case, we are seeing problem on all the ports

Nov 12 22:05:14 smale-outside : 2012 Nov 12 21:05:14 BRST: %MODULE-2-MOD_DIAG_FAIL: Module 2 (serial: JAF1548AMKB) reported failure on ports 2/1-2/32 (Ethernet) due to Octopus internal error in device 78 (device error 0xc4e0025b)

Nexus7010# sh system error-id 0xc4e0025b

Error Description: Device Name:[Octopus] Instance:[0] Error Type:[hw error] code:[91]

Example:
Nexus#show module internal exceptionlog module 10

********* Exception info for module 10 ********

exception information --- exception instance 1 ----
Module Slot Number: 10
Device Id         : 78
Device Name       : Octopus
Device Errorcode : 0xc4e01266
Device ID         : 78 (0x4e)
Device Instance   : 01 (0x01)
Dev Type (HW/SW) : 02 (0x02)
ErrNum (devInfo) : 102 (0x66)
System Errorcode : 0x410b003a Octopus internal error
Error Type        : Minor error
PhyPortLayer      : Ethernet
Port(s) Affected : 31
Error Description : OC_DF_TX1_INT_TX_TIMER_EXP_INT
DSAP              : 0 (0x0)
UUID              : 0 (0x0)
Time              : Mon Nov 1 08:21:47 2010
                    (Ticks: 4CCE789B jiffies)

This is indicates that this module is experiencing hardware problem
Could you perform soft or hard reload of this module 2, then check the performance
If you see the same issue still, i would recommend to replace the module

Regards,

Aru

*** Please rate if the post is useful ***

Regards, Aru *** Please rate if the post useful ***

Rosa Ladeira · ‎11-13-2012

Error Description: Device Name:[Octopus] Instance:[0] Error Type:[hw error] code:[91]

nx7000-wc0# show module internal exceptionlog module 2

********* Exception info for module 2 ********

exception information --- exception instance 1 ----

Module Slot Number: 2

Device Id : 78

Device Name : Octopus

Device Errorcode : 0xc4e00266

Device ID : 78 (0x4e)

Device Instance : 00 (0x00)

Dev Type (HW/SW) : 02 (0x02)

ErrNum (devInfo) : 102 (0x66)

System Errorcode : 0x410b003a Octopus internal error

Error Type : FATAL error

PhyPortLayer : Ethernet

Port(s) Affected : 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32

Error Description : OC_DF_TX1_INT_TX_TIMER_EXP_INT

DSAP : 0 (0x0)

UUID : 0 (0x0)

Time : Tue Nov 13 13:27:04 2012

(Ticks: 50A274D8 jiffies)

exception information --- exception instance 2 ----

Module Slot Number: 2

Device Id : 78

Device Name : Octopus

Device Errorcode : 0xc4e00266

Device ID : 78 (0x4e)

Device Instance : 00 (0x00)

Dev Type (HW/SW) : 02 (0x02)

ErrNum (devInfo) : 102 (0x66)

System Errorcode : 0x410b003a Octopus internal error

Error Type : FATAL error

PhyPortLayer : Ethernet

Port(s) Affected : 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32

Error Description : OC_DF_TX1_INT_TX_TIMER_EXP_INT

DSAP : 0 (0x0)

UUID : 0 (0x0)

Time : Tue Nov 13 12:10:42 2012

(Ticks: 50A262F2 jiffies)

exception information --- exception instance 3 ----

Module Slot Number: 2

Device Id : 78

Device Name : Octopus

Device Errorcode : 0xc4e00266

Device ID : 78 (0x4e)

Device Instance : 00 (0x00)

Dev Type (HW/SW) : 02 (0x02)

ErrNum (devInfo) : 102 (0x66)

System Errorcode : 0x410b003a Octopus internal error

Error Type : FATAL error

PhyPortLayer : Ethernet

Port(s) Affected : 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32

Error Description : OC_DF_TX1_INT_TX_TIMER_EXP_INT

DSAP : 0 (0x0)

UUID : 0 (0x0)

Time : Tue Nov 13 11:53:54 2012

(Ticks: 50A25F02 jiffies)

exception information --- exception instance 4 ----

Module Slot Number: 2

Device Id : 78

Device Name : Octopus

Device Errorcode : 0xc4e0025b

Device ID : 78 (0x4e)

Device Instance : 00 (0x00)

Dev Type (HW/SW) : 02 (0x02)

ErrNum (devInfo) : 91 (0x5b)

System Errorcode : 0x410b003a Octopus internal error

Error Type : FATAL error

PhyPortLayer : Ethernet

Port(s) Affected : 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32

Error Description : OC_DF_TX0_INT_TX_TIMER_EXP_INT

DSAP : 0 (0x0)

UUID : 0 (0x0)

Time : Tue Nov 13 11:14:59 2012

(Ticks: 50A255E3 jiffies)

exception information --- exception instance 5 ----

Module Slot Number: 2

Device Id : 78

Device Name : Octopus

Device Errorcode : 0xc4e0025b

Device ID : 78 (0x4e)

Device Instance : 00 (0x00)

Dev Type (HW/SW) : 02 (0x02)

ErrNum (devInfo) : 91 (0x5b)

System Errorcode : 0x410b003a Octopus internal error

Error Type : FATAL error

PhyPortLayer : Ethernet

Port(s) Affected : 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32

Error Description : OC_DF_TX0_INT_TX_TIMER_EXP_INT

DSAP : 0 (0x0)

UUID : 0 (0x0)

Time : Mon Nov 12 21:05:14 2012

(Ticks: 50A18EBA jiffies)

exception information --- exception instance 6 ----

Module Slot Number: 2

Device Id : 78

Device Name : Octopus

Device Errorcode : 0xc4e0025b

Device ID : 78 (0x4e)

Device Instance : 00 (0x00)

Dev Type (HW/SW) : 02 (0x02)

ErrNum (devInfo) : 91 (0x5b)

System Errorcode : 0x410b003a Octopus internal error

Error Type : FATAL error

PhyPortLayer : Ethernet

Port(s) Affected : 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32

Error Description : OC_DF_TX0_INT_TX_TIMER_EXP_INT

DSAP : 0 (0x0)

UUID : 0 (0x0)

Time : Sat Nov 10 03:40:36 2012

(Ticks: 509DF6E4 jiffies)

exception information --- exception instance 7 ----

Module Slot Number: 2

Device Id : 78

Device Name : Octopus

Device Errorcode : 0xc4e0025b

Device ID : 78 (0x4e)

Device Instance : 00 (0x00)

Dev Type (HW/SW) : 02 (0x02)

ErrNum (devInfo) : 91 (0x5b)

System Errorcode : 0x410b003a Octopus internal error

Error Type : FATAL error

PhyPortLayer : Ethernet

Port(s) Affected : 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32

Error Description : OC_DF_TX0_INT_TX_TIMER_EXP_INT

DSAP : 0 (0x0)

UUID : 0 (0x0)

Time : Thu Nov 8 16:55:28 2012

(Ticks: 509C0E30 jiffies)

exception information --- exception instance 8 ----

Module Slot Number: 2

Device Id : 96

Device Name : R2D2

Device Errorcode : 0x41830059

Device ID : 24 (0x18)

Device Instance : 48 (0x30)

Dev Type (HW/SW) : 00 (0x00)

ErrNum (devInfo) : 89 (0x59)

System Errorcode : 0x4183003d Loopback test failed. Packets lost on the LC at the MAC ASIC

Error Type : Warning

PhyPortLayer : Ethernet

Port(s) Affected : 8

DSAP : 0 (0x0)

UUID : 0 (0x0)

Time : Tue Feb 14 18:51:50 2012

(Ticks: 4F3AD776 jiffies)

exception information --- exception instance 9 ----

Module Slot Number: 2

Device Id : 123

Device Name : Platform

Device Errorcode : 0xc7b00227

Device ID : 123 (0x7b)

Device Instance : 00 (0x00)

Dev Type (HW/SW) : 02 (0x02)

ErrNum (devInfo) : 39 (0x27)

System Errorcode : 0x40380027 LC/SUP powered down due to both ejectors popped open

Error Type : FATAL error

PhyPortLayer :

Port(s) Affected :

DSAP : 39 (0x27)

UUID : 24 (0x18)

Time : Thu Oct 20 15:04:17 2011

(Ticks: 4EA05491 jiffies)

exception information --- exception instance 10 ----

Module Slot Number: 2

Device Id : 70

Device Name : Santa-Cruz-Module

Device Errorcode : 0xc4600248

Device ID : 70 (0x46)

Device Instance : 00 (0x00)

Dev Type (HW/SW) : 02 (0x02)

ErrNum (devInfo) : 72 (0x48)

System Errorcode : 0x40420009 X-bar Interface ASIC Error

Error Type : Warning

PhyPortLayer :

Port(s) Affected :

DSAP : 0 (0x0)

UUID : 254 (0xfe)

Time : Thu Oct 20 15:04:17 2011

(Ticks: 4EA05491 jiffies)

exception information --- exception instance 11 ----

Module Slot Number: 2

Device Id : 70

Device Name : Santa-Cruz-Module

Device Errorcode : 0xc4600248

Device ID : 70 (0x46)

Device Instance : 00 (0x00)

Dev Type (HW/SW) : 02 (0x02)

ErrNum (devInfo) : 72 (0x48)

System Errorcode : 0x40420009 X-bar Interface ASIC Error

Error Type : Warning

PhyPortLayer :

Port(s) Affected :

DSAP : 0 (0x0)

UUID : 254 (0xfe)

Time : Thu Oct 20 15:04:17 2011

(Ticks: 4EA05491 jiffies)

exception information --- exception instance 12 ----

Module Slot Number: 2

Device Id : 70

Device Name : Santa-Cruz-Module

Device Errorcode : 0xc4600248

Device ID : 70 (0x46)

Device Instance : 00 (0x00)

Dev Type (HW/SW) : 02 (0x02)

ErrNum (devInfo) : 72 (0x48)

System Errorcode : 0x40420009 X-bar Interface ASIC Error

Error Type : Warning

PhyPortLayer :

Port(s) Affected :

DSAP : 0 (0x0)

UUID : 254 (0xfe)

Time : Thu Oct 20 15:04:17 2011

(Ticks: 4EA05491 jiffies)

exception information --- exception instance 13 ----

Module Slot Number: 2

Device Id : 70

Device Name : Santa-Cruz-Module

Device Errorcode : 0xc4600248

Device ID : 70 (0x46)

Device Instance : 00 (0x00)

Dev Type (HW/SW) : 02 (0x02)

ErrNum (devInfo) : 72 (0x48)

System Errorcode : 0x40420009 X-bar Interface ASIC Error

Error Type : Warning

PhyPortLayer :

Port(s) Affected :

DSAP : 0 (0x0)

UUID : 254 (0xfe)

Time : Thu Oct 20 15:04:17 2011

(Ticks: 4EA05491 jiffies)

exception information --- exception instance 14 ----

Module Slot Number: 2

Device Id : 70

Device Name : Santa-Cruz-Module

Device Errorcode : 0xc4600248

Device ID : 70 (0x46)

Device Instance : 00 (0x00)

Dev Type (HW/SW) : 02 (0x02)

ErrNum (devInfo) : 72 (0x48)

System Errorcode : 0x40420009 X-bar Interface ASIC Error

Error Type : Warning

PhyPortLayer :

Port(s) Affected :

DSAP : 0 (0x0)

UUID : 254 (0xfe)

Time : Thu Oct 20 15:04:17 2011

(Ticks: 4EA05491 jiffies)

exception information --- exception instance 15 ----

Module Slot Number: 2

Device Id : 70

Device Name : Santa-Cruz-Module

Device Errorcode : 0xc4600248

Device ID : 70 (0x46)

Device Instance : 00 (0x00)

Dev Type (HW/SW) : 02 (0x02)

ErrNum (devInfo) : 72 (0x48)

System Errorcode : 0x40420009 X-bar Interface ASIC Error

Error Type : Warning

PhyPortLayer :

Port(s) Affected :

DSAP : 0 (0x0)

UUID : 254 (0xfe)

Time : Thu Oct 20 15:04:17 2011

(Ticks: 4EA05491 jiffies)

exception information --- exception instance 16 ----

Module Slot Number: 2

Device Id : 123

Device Name : Platform

Device Errorcode : 0xc7b02202

Device ID : 123 (0x7b)

Device Instance : 02 (0x02)

Dev Type (HW/SW) : 02 (0x02)

ErrNum (devInfo) : 02 (0x02)

System Errorcode : 0x40380026 One ejector is open (Ejector instance=ErrNum)

Error Type : Minor error

PhyPortLayer :

Port(s) Affected :

DSAP : 39 (0x27)

UUID : 24 (0x18)

Time : Thu Oct 20 15:04:16 2011

(Ticks: 4EA05490 jiffies)

Arumugam Muthaiah · ‎11-13-2012

Hi Rosa,

You are having same problem, please try to reload the module 2 and check the performance.

Regards,

Aru

*** Please rate if the post is useful ***

Regards, Aru *** Please rate if the post useful ***

Rosa Ladeira · ‎11-13-2012

In addition, I should tell you that:

1. there are 2 DELL 6224 :

Eth2/26 and

Eth2/30

2. when 10 Gbps Ethernet Module N7K-M132XP-12 powered-dn

6224 fibers where detached

3. I have executed:

conf t

no poweroff module 2

attach module 2

exit

4. I ahve attache 6224 to Eth2/26

5. The arrangement is running fine for more than an hour.

6. I will wait a little bit longer to attach Eth2/30