Syslog error - WAAS-SYS-3-900000: e1000bp: eth5: e1000bp_clean_tx_irq: Detected Tx Unit Hang

cwehrstedt · ‎03-10-2011

I am receiving this error in my syslog, every couple weeks or so. It causes a flap between our two inline WAEs. Have an idea what can be causing this? syslog is below, start from the bottom up.

2011-03-10 10:30:08 Kernel Error 10.0.100.26 2011 Mar 10 15: kernel: %WAAS-SYS-3-900000: e1000bp: eth4: e1000bp_watchdog: NIC Link is Up 1000 Mbps Full Duplex

2011-03-10 10:30:06 Kernel Error 10.0.100.26 2011 Mar 10 15: kernel: %WAAS-SYS-3-900000: e1000bp: eth5: e1000bp_watchdog: NIC Link is Up 1000 Mbps Full Duplex

2011-03-10 10:30:02 Kernel Warning 10.0.100.26 2011 Mar 10 15: kernel: %WAAS-SYS-4-900000: next_to_watch.status <0>

2011-03-10 10:30:02 Kernel Warning 10.0.100.26 2011 Mar 10 15: kernel: %WAAS-SYS-4-900000: jiffies <2b900a9e1>

2011-03-10 10:30:02 Kernel Warning 10.0.100.26 2011 Mar 10 15: kernel: %WAAS-SYS-4-900000: next_to_watch <91>

2011-03-10 10:30:02 Kernel Warning 10.0.100.26 2011 Mar 10 15: kernel: %WAAS-SYS-4-900000: time_stamp <2b9008c71>

2011-03-10 10:30:02 Kernel Warning 10.0.100.26 2011 Mar 10 15: kernel: %WAAS-SYS-4-900000: buffer_info[next_to_clean]

2011-03-10 10:30:02 Kernel Warning 10.0.100.26 2011 Mar 10 15: kernel: %WAAS-SYS-4-900000: next_to_clean <91>

2011-03-10 10:30:02 Kernel Warning 10.0.100.26 2011 Mar 10 15: kernel: %WAAS-SYS-4-900000: next_to_use <7e>

2011-03-10 10:30:02 Kernel Warning 10.0.100.26 2011 Mar 10 15: kernel: %WAAS-SYS-4-900000: TDT <7e>

2011-03-10 10:30:02 Kernel Warning 10.0.100.26 2011 Mar 10 15: kernel: %WAAS-SYS-4-900000: TDH <91>

2011-03-10 10:30:02 Kernel Warning 10.0.100.26 2011 Mar 10 15: kernel: %WAAS-SYS-4-900000: Tx Queue <0>

2011-03-10 10:30:02 Kernel Error 10.0.100.26 2011 Mar 10 15: kernel: %WAAS-SYS-3-900000: e1000bp: eth5: e1000bp_clean_tx_irq: Detected Tx Unit Hang

Bhavin Yadav · ‎03-10-2011

Hi Cora,

Do you see any amber lights on the front panel?

Do you see any alarms on the WAVE?

Can you post "sh alarm hist detail " cli output from the WAE?

Thanks,

Bhavin Yadav · ‎03-10-2011

Can you also post output of following cli from WAE?

WAE7341-1#find-pattern match "NMI" syslog.txt

Thanks.

cwehrstedt · ‎03-11-2011

Nothing too much from the "find-pattern" cmd. a bunch from the 11-14 of december when I was rebuilding it (the

device was RMA'd for the battery, maybe you remember)

2010 Dec 14 19:14:19 NO-HOSTNAME syslog_bootup_msgs: %WAAS-SYS-5-900001: <5>Kernel command line: root=/dev/ram0 ramdisk_start=
0x6000000 ramdisk_size=100000 pci=nommconf nmi_watchdog=0 reboot=a panic=2 console=ttyS0,9600n8 crashkernel=32M@16M
2010 Dec 14 19:14:19 NO-HOSTNAME syslog_bootup_msgs: %WAAS-SYS-5-900001: <6>oprofile: using NMI interrupt.

these are the last two entries, nothing more recent.

drlechowicz · ‎08-21-2014

Did you figure out what was causing this issue? I am having the exact same issue.

Thanks!
Dirk

Abhishek Purohit · ‎09-03-2014

Hi Dirk,

An RMA for the Inline card is needed for this issue.

In case even after the RMA, the issue remains, reimage the WAAS with the same version or higher.

Hope that helps.

Regards,

Abhishek

CCIES 35269

Regards, Abhishek Purohit CCIE-S- 35269

cwehrstedt · ‎03-11-2011

Here is the output from the show alarms hist, also there are no visual LED problem with the device.

     Op Sev Alarm ID             Module/Submodule     Instance
     -- --- -------------------- -------------------- --------------------
   1 C Mi servicedead          nodemgr              cms_ce
     Mar 7 15:00:41.435 EST, Processing Error Alarm, #000008, 2000:330004
     nodemgr: The cms_ce service died.

   2 R Mi servicedead          nodemgr              cms_ce
     Mar 7 15:00:31.233 EST, Processing Error Alarm, #000008, 2000:330004
     nodemgr: The cms_ce service died.

   3 C Mi servicedead          nodemgr              device_mgr
     Feb 2 11:31:45.484 EST, Processing Error Alarm, #000007, 2000:330004
     nodemgr: The device_mgr service died.

   4 R Mi servicedead          nodemgr              device_mgr
     Feb 2 11:31:35.282 EST, Processing Error Alarm, #000007, 2000:330004
     nodemgr: The device_mgr service died.

   5 C Mi servicedead          nodemgr              device_mgr
     Jan 22 20:39:47.480 EST, Processing Error Alarm, #000006, 2000:330004
     nodemgr: The device_mgr service died.

   6 R Mi servicedead          nodemgr              device_mgr
     Jan 22 20:39:37.276 EST, Processing Error Alarm, #000006, 2000:330004
     nodemgr: The device_mgr service died.

Bhavin Yadav · ‎03-11-2011

Hmm. nothing is coming out as eye catching but i will look at thsi during weekend and update you accordingly. You might have to open a TAC case for further deep analysis.

Regards.

cwehrstedt · ‎05-02-2011

I did an RMA on the Inline card, but I am still recieving this error, and getting a drop in service.