Hi folks,
I have a 6509 that had two different modules powercycle themselves about 18 hours apart. It could be coincidence, but I've never had it happen before so I suspect it's related. Unfortunately I haven't had any luck so far identifying a root cause, so I figured I'd post here for help. Below I've pasted the logs, diagnostic messages, module info, and switch version info. Any insight is appreciated.
Thanks!
FIRST EVENT
Sep 22 09:23:54: %MFIB_CONST_RP-6-REPLICATION_MODE_CHANGE: Replication Mode Change Detected. Current system replication mode is Egress
Sep 22 09:23:54: %SNMP-5-MODULETRAP: Module 6 [Down] Trap
Sep 22 09:23:54: %MLSM-6-MMLS_MODE_CHANGE: MMLS: Replication Mode Changed to Egress mode (Oir event or MMLS enabled)
Sep 22 09:24:19: %C6KPWR-SP-4-DISABLED: power to module in slot 6 set off (Reset)
Sep 22 09:26:07: %DIAG-SP-6-RUN_MINIMUM: Module 6: Running Minimal Diagnostics...
Sep 22 09:26:14: %SVCLC-5-FWVTPMODE: VTP mode is set to non-transparent
Sep 22 09:26:14: %MFIB_CONST_RP-6-REPLICATION_MODE_CHANGE: Replication Mode Change Detected. Current system replication mode is Ingress
Sep 22 09:26:14: %MLSM-6-MMLS_MODE_CHANGE: MMLS: Replication Mode Changed to Ingress mode (Oir event or MMLS enabled)
Sep 22 09:26:14: %SNMP-5-MODULETRAP: Module 6 [Up] Trap
Sep 22 09:26:14: %DIAG-SP-6-DIAG_OK: Module 6: Passed Online Diagnostics
Sep 22 09:26:18: %OIR-SP-6-INSCARD: Card inserted in slot 6, interfaces are now online
switch#show diagnostic events module 6 | b 09/22
09/22 09:26:12.247 E [6] check_ether_packet [6/1]: newpak is NULL!
09/22 09:26:12.375 E [6] komodo_plus_lb_pc_port[6/1]: check_ether_packet f
ailed (err-code 0)
09/22 09:26:12.379 E [6] komodo_plus_lb_pc_port[6/1]: check_ether_packet f
ailed
09/22 09:26:12.439 E [6] komodo_plus_test_loopback [6/1]: PC loopback test
with ether type 0xA501 failed
09/22 09:26:12.455 E [6] komodo_plus_test_loopback : loopback_port failed
09/22 09:26:13.199 E [6] check_ether_packet [6/4]: newpak is NULL!
09/22 09:26:13.319 E [6] komodo_plus_lb_pc_port[6/4]: check_ether_packet f
ailed (err-code 0)
09/22 09:26:13.327 E [6] komodo_plus_lb_pc_port[6/4]: check_ether_packet f
ailed
09/22 09:26:13.383 E [6] komodo_plus_test_loopback [6/4]: PC loopback test
with ether type 0xA601 failed
09/22 09:26:14.195 I [6] Diagnostics Passed
SECOND EVENT
Sep 23 18:20:42: %SNMP-5-MODULETRAP: Module 7 [Down] Trap
Sep 23 18:20:42: %OIR-SP-3-PWRCYCLE: Card in module 7, is being power-cycled 'off (Reset - Module Reloaded During Download)'
Sep 23 18:20:42: %C6KPWR-SP-4-DISABLED: power to module in slot 7 set off (Reset - Module Reloaded During Download)
Sep 23 18:20:56: %OIR-SP-3-PWRCYCLE: Card in module 7, is being power-cycled 'off (Module not responding to Keep Alive polling)'
Sep 23 18:20:56: %C6KPWR-SP-4-DISABLED: power to module in slot 7 set off (Module not responding to Keep Alive polling)
Sep 23 18:25:41: %DIAG-SP-6-RUN_MINIMUM: Module 7: Running Minimal Diagnostics...
Sep 23 18:25:44: %SVCLC-5-SVCLCVTPMODE: VTP mode is set to non-transparent
Sep 23 18:25:44: %SNMP-5-MODULETRAP: Module 7 [Up] Trap
Sep 23 18:25:44: %DIAG-SP-6-DIAG_OK: Module 7: Passed Online Diagnostics
Sep 23 18:25:44: %OIR-SP-6-INSCARD: Card inserted in slot 7, interfaces are now online
switch#show diagnostic events module 7 | b 09/23
09/23 18:19:00.261 E [7] diag_asic_read_reg16[7/1/42]: set_get_asic_regs()
failed (get)
09/23 18:19:00.265 E [7] queryHyperionSynched[7]: diag_asic_read_reg16 fai
led
09/23 18:19:00.269 E [7] test_asic_sync[7]: queryHyperionSynched failed
09/23 18:20:46.294 E [7] TestAsicSync Failed
09/23 18:20:46.298 E [7] diag_get_current_running_test_name [7]: diag_get_
run_info_reg failed
09/23 18:20:46.306 E [7] diag_pf_const2_test_has_failed[7/-1]: reg_invoke_
diag_get_test_attribute_by_name failed
09/23 18:25:44.222 I [7] Diagnostics Passed
SWITCH INFO
switch#show module all
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
1 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAL1107H180
2 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAL10382FWV
3 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAL1106GG6A
4 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAD114102ZZ
5 2 Supervisor Engine 720 (Active) WS-SUP720-3B SAL1105FUCK
6 6 Firewall Module WS-SVC-FWM-1 SAD110203XT
7 1 Application Control Engine Module ACE20-MOD-K9 SAD110104KW
8 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAD115100Y7
Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
1 001a.e27a.2790 to 001a.e27a.27bf 2.5 12.2(14r)S5 12.2(33)SXI3 Ok
2 0018.bacf.998c to 0018.bacf.99bb 2.5 12.2(14r)S5 12.2(33)SXI3 Ok
3 001a.e2eb.6ad0 to 001a.e2eb.6aff 2.5 12.2(14r)S5 12.2(33)SXI3 Ok
4 001d.45fd.20de to 001d.45fd.210d 2.6 12.2(14r)S5 12.2(33)SXI3 Ok
5 0017.9444.6010 to 0017.9444.6013 5.3 8.4(2) 12.2(33)SXI3 Ok
6 0019.e78f.5a70 to 0019.e78f.5a77 4.1 7.2(1) 3.2(1) Ok
7 0030.f276.145a to 0030.f276.1461 2.0 8.6(0.252-En 3.0(0)A1(4a) Ok
8 001e.13cc.c438 to 001e.13cc.c467 2.7 12.2(14r)S5 12.2(33)SXI3 Ok
Mod Sub-Module Model Serial Hw Status
---- --------------------------- ------------------ ----------- ------- -------
1 Centralized Forwarding Card WS-F6700-CFC SAD110707K5 3.1 Ok
2 Centralized Forwarding Card WS-F6700-CFC SAL1052C0SQ 3.1 Ok
3 Centralized Forwarding Card WS-F6700-CFC SAL1051BP7Y 3.1 Ok
4 Centralized Forwarding Card WS-F6700-CFC SAD113904FV 4.0 Ok
5 Policy Feature Card 3 WS-F6K-PFC3B SAL1103E9VS 2.3 Ok
5 MSFC3 Daughterboard WS-SUP720 SAL1104FB3U 2.6 Ok
8 Centralized Forwarding Card WS-F6700-CFC SAL1207GEYP 4.0 Ok
Mod Online Diag Status
---- -------------------
1 Pass
2 Pass
3 Pass
4 Pass
5 Pass
6 Pass
7 Pass
8 Pass
switch#show ver
Cisco IOS Software, s72033_rp Software (s72033_rp-ADVIPSERVICESK9_WAN-M), Version 12.2(33)SXI3, RELEASE SOFTWARE (fc2)
<snip>
cisco WS-C6509-E (R7000) processor (revision 1.3) with 458720K/65536K bytes of memory.
Processor board ID SMG1109N3BK
SR71000 CPU at 600Mhz, Implementation 0x504, Rev 1.2, 512KB L2 Cache
Last reset from s/w reset
27 Virtual Ethernet interfaces
248 Gigabit Ethernet interfaces
1 Ten Gigabit Ethernet interface
1917K bytes of non-volatile configuration memory.
8192K bytes of packet buffer memory.
65536K bytes of Flash internal SIMM (Sector size 512K).
Configuration register is 0x2102
