Catalyst 6509 "Module 9 Error counter exceeds threshold..."

[nnraymond]

Every 1-4 hours we see messages like this:

*Mar 11 16:04:56.532: %CONST_DIAG-4-ERROR_COUNTER_WARNING: Module 9 Error counter exceeds threshold, system operation continue.
*Mar 11 16:04:56.532: %CONST_DIAG-4-ERROR_COUNTER_DATA: ID:60 IN:0 PO:255 RE:5212 RM:255 DV:2 EG:2 CF:10 TF:233180

Module 9 is a 48 port 10/100/1000 IEEE 802.3AT, model WS-X6148E-GE-45AT. I have moved everything off that module so that no ethernet devices are plugged into it and the "Module 9 Error counter exceeds threshold" continue. I performed a reset of that module, and this is what was logged:

*Mar 11 16:33:46.752: %C6KPWR-4-DISABLED: power to module in slot 9 set off (User Request)
*Mar 11 16:33:46.788: %SNMP-5-MODULETRAP: Module 9 [Down] Trap
*Mar 11 16:35:39.504: %DIAG-6-RUN_MINIMUM: Module 9: Running Minimal Diagnostics...
*Mar 11 16:37:59.432: %DIAG-6-DIAG_OK: Module 9: Passed Online Diagnostics
*Mar 11 16:38:00.800: %SNMP-5-MODULETRAP: Module 9 [Up] Trap
*Mar 11 16:38:00.804: %OIR-6-SP_INSCARD: Card inserted in slot 9, interfaces are now online
*Mar 11 16:52:12.912: %CONST_DIAG-4-ERROR_COUNTER_WARNING: Module 9 Error counter exceeds threshold, system operation continue.
*Mar 11 16:52:12.912: %CONST_DIAG-4-ERROR_COUNTER_DATA: ID:60 IN:0 PO:255 RE:5212 RM:255 DV:5 EG:2 CF:10 TF:10

What is the cause/fix for this?

[balaji.bandi]

you done soft reset or hard reseated the module (i mean took out and re-insertion ?)

can you post show modules and show version

after reseat the error still show, time for TAC for RMA

[nnraymond]

I did a soft reset. We can't do TAC or RMA because we don't have any paid support in place for this equipment (yes it is the heart of our network, but we are a school district). Here is the output of show module:

Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
1 48 48 port 10/100/1000 IEEE 802.3AT WS-X6148E-GE-45AT SAL17215R2D
2 48 48 port 10/100/1000 IEEE 802.3AT WS-X6148E-GE-45AT SAL17215R13
3 8 DCEF2T 8 port 10GE WS-X6908-10G SAL17215FQU
4 8 DCEF2T 8 port 10GE WS-X6908-10G SAL17215FS4
5 5 Supervisor Engine 2T 10GE w/ CTS (Acti VS-SUP2T-10G SAL17204Z8G
6 5 Supervisor Engine 2T 10GE w/ CTS (Hot) VS-SUP2T-10G SAL17204Z8A
7 48 48 port 10/100/1000 IEEE 802.3AT WS-X6148E-GE-45AT SAL17215R3P
8 48 48 port 10/100/1000 IEEE 802.3AT WS-X6148E-GE-45AT SAL17215R04
9 48 48 port 10/100/1000 IEEE 802.3AT WS-X6148E-GE-45AT SAL17215R2X

Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
1 bc16.650f.e4a0 to bc16.650f.e4cf 1.4 8.4(1) 15.0(1)SY4 Ok
2 bc16.658e.1b30 to bc16.658e.1b5f 1.4 8.4(1) 15.0(1)SY4 Ok
3 0006.f6b2.4648 to 0006.f6b2.464f 1.2 12.2(50r)SYL 15.0(1)SY4 Ok
4 0006.f6b2.45b8 to 0006.f6b2.45bf 1.2 12.2(50r)SYL 15.0(1)SY4 Ok
5 2c54.2dc3.968e to 2c54.2dc3.9695 1.5 12.2(50r)SYS 15.0(1)SY4 Ok
6 2c54.2dc3.96f0 to 2c54.2dc3.96f7 1.5 12.2(50r)SYS 15.0(1)SY4 Ok
7 e4d3.f1f6.2bd0 to e4d3.f1f6.2bff 1.4 8.4(1) 15.0(1)SY4 Ok
8 bc16.650f.e9b0 to bc16.650f.e9df 1.4 8.4(1) 15.0(1)SY4 Ok
9 bc16.650f.dd50 to bc16.650f.dd7f 1.4 8.4(1) 15.0(1)SY4 Ok

Mod Sub-Module Model Serial Hw Status
---- --------------------------- ------------------ ----------- ------- -------
1 IEEE Voice Daughter Card WS-F6K-48-AT SAL17215MKK 1.0 Ok
2 IEEE Voice Daughter Card WS-F6K-48-AT SAL17215ML1 1.0 Ok
3 Distributed Forwarding Card WS-F6K-DFC4-E SAL17194JYW 1.2 Ok
4 Distributed Forwarding Card WS-F6K-DFC4-E SAL17194JZ0 1.2 Ok
5 Policy Feature Card 4 VS-F6K-PFC4 SAL17204RQU 2.1 Ok
5 CPU Daughterboard VS-F6K-MSFC5 SAL17204YWM 2.0 Ok
6 Policy Feature Card 4 VS-F6K-PFC4 SAL17204RQ4 2.1 Ok
6 CPU Daughterboard VS-F6K-MSFC5 SAL17204Z05 2.0 Ok
7 IEEE Voice Daughter Card WS-F6K-48-AT SAL17215MGX 1.0 Ok
8 IEEE Voice Daughter Card WS-F6K-48-AT SAL17215MHS 1.0 Ok
9 IEEE Voice Daughter Card WS-F6K-48-AT SAL17215MGP 1.0 Ok

Mod Online Diag Status
---- -------------------
1 Pass
2 Pass
3 Pass
4 Pass
5 Pass
6 Pass
7 Pass
8 Pass
9 Minor Error

Here is the output of show version:

Cisco IOS Software, s2t54 Software (s2t54-ADVIPSERVICESK9-M), Version 15.0(1)SY4, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2013 by Cisco Systems, Inc.
Compiled Mon 18-Mar-13 00:27 by prod_rel_team

ROM: System Bootstrap, Version 12.2(50r)SYS3, RELEASE SOFTWARE (fc1)

MDF-6509 uptime is 44 weeks, 1 day, 14 hours, 56 minutes
Uptime for this control processor is 44 weeks, 1 day, 14 hours, 56 minutes
System returned to ROM by reload at 03:11:10 UTC Thu May 6 2021
System image file is "bootdisk:s2t54-advipservicesk9-mz.SPA.150-1.SY4.bin"
Last reload reason: Reload Command




This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C6509-E (M8572) processor (revision ) with 1769472K/262144K bytes of memory.
Processor board ID SMG1716N04G
CPU: MPC8572_E, Version: 2.2, (0x80E80022)
CORE: E500, Version: 3.0, (0x80210030)
CPU:1500MHz, CCB:600MHz, DDR:600MHz
L1: D-cache 32 kB enabled
I-cache 32 kB enabled

Last reset from s/w reset
18 Virtual Ethernet interfaces
246 Gigabit Ethernet interfaces
20 Ten Gigabit Ethernet interfaces
2543K bytes of non-volatile configuration memory.

Configuration register is 0x2102

[Leo Laohoo]

Re-seat that line card.

If the error message returns, RMA it.

[marce1000]

 - https://bst.cloudapps.cisco.com/bugsearch/bug/CSCsh37268  , if you note the content of the bug report , then it also is always better to have support for equipment , servicing business operations.

 M.

[nnraymond]

When I try to go to that link Cisco asks me to log in and when I do, all I get is "HTTP Status 400 – Bad Request". Can you copy the contents of that page here so I can read it?

I know it is better to have support, but I've only been in my position at this organization for a few years, and my predecessor did not purchase the equipment with support, he is not with the organization anymore (and did not document anything and refused to give anyone any passwords), I am doing the best I can with what I've got. We have a plan in place to purchase new equipment but that is weeks away before we receive it. I have a problem right now where every 1-2 minutes devices on one our VLANs are not able to receive any traffic and I'm trying to isolate that problem, i.e. is our Catalyst 6509 causing this problem? I need to get to the bottom of it ASAP.

[marce1000]

 - For the bug report  , use another browser and try again. For the rest of your reply , I can only  say that failing hardware = failing hardware and that's it. There are no other options then to repair (requires support indeed). Or to replace.  With due respect. And or also I would not dig to deep into those other observed issues , if your core 6509 is indicating a hardware problem as it did.

 M. 

[nnraymond]

Thanks, Apple Safari worked for that link - Google Chrome isn't working. Now that I can read it, it just says that the bad hardware should be replaced. In this case, our module 9 (WS-X6148E-GE-45AT) has nothing plugged into it, so I can just remove it. Hopefully that resolves this issue, and it's not the chassis itself that is the source of the problem.

[marce1000]

 - Yeah , I have been having the same issues with Chrome , when reading bug reports from Cisco , for  them I switch to Firefox. You can indeed remove the module if it is not used. Attach a console when doing so , follow-up on console messages. For the rest of the issues follow-up on logs afterwards on the 6509 or use a syslog-server (the last option is the best).

 M.

[nnraymond]

Logs show:

*Mar 14 16:09:38.182: %SNMP-5-MODULETRAP: Module 9 [Down] Trap
*Mar 14 16:09:38.210: %OIR-6-SP_REMCARD: Card removed from slot 9, interfaces disabled

No new "%CONST_DIAG-4-ERROR_COUNTER_WARNING" messages yet. I'll keep monitoring.

[nnraymond]

No new error messages in the log. So far since removing that module, the periodic and serious issues with traffic on VLAN 8 haven't recurred, but I will continue to monitor that situation.

In case you are wondering why schools end up with Cisco equipment without support, it's because there is federal money for network hardware, but there is not federal money for network hardware support. So schools can often afford Cisco hardware thanks to those federal dollars, but don't necessarily have the budget for the support. I've been pushing hard to avoid the situation where our school ends up with hardware with no support, so that the school doesn't end up up a creek without a paddle, so to speak, and just not buy hardware they don't have the money to buy support for, because eventually, it will be a problem (especially because schools tend to have upgrade cycles of about 10 years).

[marce1000]

 - I acknowledge your predicament , but equipment servicing business operations ,should always have support. Then it comes down to the question 'can the school as-a-business cope with critical failure or not...' , if not then.....

 M.