EarlRecoveryPatch Reset on module

[morrisce]

Hello,

We are currently getting the errors below in our log on our 6509 with a SUP 720.  Module 3 is a 16 port gigabit GBIC switch.  I checked this log recently because our edge switches are randoming dropping offline at their uplink ports.  The only thing that will bring them back is a reboot of the switch.  Any thoughts?

Feb  3 15:04:21: %C6KPWR-SP-4-DISABLED: power to module in slot 3 set off (EarlRecoveryPatch Reset)
Feb  3 15:05:00: %DIAG-SP-6-RUN_MINIMUM: Module 3: Running Minimum Diagnostics...
Feb  3 15:05:12: %DIAG-SP-6-DIAG_OK: Module 3: Passed Online Diagnostics
Feb  3 15:05:13: %OIR-SP-6-INSCARD: Card inserted in slot 3, interfaces are now online
Feb  3 15:31:04: %SYS-5-CONFIG_I: Configured from console by vty0 (10.110.43.7)
Feb  3 15:31:43: %C6KPWR-SP-4-DISABLED: power to module in slot 3 set off (Reset)
Feb  3 15:32:26: %DIAG-SP-6-RUN_COMPLETE: Module 3: Running Complete Diagnostics...
Feb  3 15:32:49: %DIAG-SP-6-DIAG_OK: Module 3: Passed Online Diagnostics
Feb  3 15:32:50: %OIR-SP-6-INSCARD: Card inserted in slot 3, interfaces are now online
Feb  3 23:16:09: %EARL-SP-3-RESET_LC: Resetting module in slot 3. (Errorcode 1)
Feb  3 23:16:09: %OIR-SP-3-PWRCYCLE: Card in module 3, is being power-cycled off (EarlRecoveryPatch Reset)
Feb  3 23:16:09: %C6KPWR-SP-4-DISABLED: power to module in slot 3 set off (EarlRecoveryPatch Reset)
Feb  3 23:16:48: %DIAG-SP-6-RUN_COMPLETE: Module 3: Running Complete Diagnostics...
Feb  3 23:17:09: %DIAG-SP-6-DIAG_OK: Module 3: Passed Online Diagnostics
Feb  3 23:17:10: %OIR-SP-6-INSCARD: Card inserted in slot 3, interfaces are now online

[Matthew Blanshard]

Hello Chad,

Can you attach a show version for the 6500?  Typically though this condition is related to either a mis-seated module or a faulty module.

-Matt

[morrisce]

Here it is!  Thanks for your response!  I've got another module on the way in case that is the issue.

ACS6509#show version
Cisco Internetwork Operating System Software
IOS (tm) s72033_rp Software (s72033_rp-PSV-M), Version 12.2(18)SXD7a, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2006 by cisco Systems, Inc.
Compiled Tue 12-Sep-06 17:42 by tinhuang
Image text-base: 0x4002100C, data-base: 0x42040000

ROM: System Bootstrap, Version 12.2(17r)S2, RELEASE SOFTWARE (fc1)
BOOTLDR: s72033_rp Software (s72033_rp-PSV-M), Version 12.2(18)SXD7a, RELEASE SOFTWARE (fc1)

ACS6509 uptime is 13 weeks, 1 day, 1 hour, 42 minutes
Time since ACS6509 switched to active is 13 weeks, 1 day, 1 hour, 41 minutes
System returned to ROM by unknown reload cause - suspect boot_data[BOOT_COUNT] 0x0, BOOT_COUNT 0, BOOTDATA 19 (SP by power-on)
System restarted at 11:05:12 EST Thu Nov 4 2010
System image file is "sup-bootflash:s72033-psv-mz.122-18.SXD7a.bin"

cisco WS-C6509 (R7000) processor (revision 3.0) with 983008K/65536K bytes of memory.
Processor board ID SCA061000XM
SR71000 CPU at 600Mhz, Implementation 0x504, Rev 1.2, 512KB L2 Cache
Last reset from power-on
X.25 software, Version 3.0.0.
Bridging software.
11 Virtual Ethernet/IEEE 802.3  interface(s)
48 FastEthernet/IEEE 802.3 interface(s)
66 Gigabit Ethernet/IEEE 802.3 interface(s)
1917K bytes of non-volatile configuration memory.
8192K bytes of packet buffer memory.

65536K bytes of Flash internal SIMM (Sector size 512K).
Configuration register is 0x2102

[Matthew Blanshard]

Because of the IOS you are running it's going to be very difficult (if impossible) to troubleshoot this beyond a hardware replacement.  You really need to upgrade your code to at least 12.2(18)SXF3 or higher based off of this article:

https://www.ciscotaccc.com/kaidara-advisor/lanswitching/showcase?case=K38164613

Fabric problems are one of the causes of the earl patch.  Has the card been re-seated yet or have you just rebooted when the issue happens?

-Matt

[morrisce]

I have yet to do a reboot.  Haven't had a good time for it just yet.  I'm going to attempt this evening along with reseating the module.  Crossing my fingers!

I do have a spare on the way though.  Thanks for the response again!

[Leo Laohoo]

System returned to ROM by unknown reload cause - suspect boot_data[BOOT_COUNT] 0x0, BOOT_COUNT 0, BOOTDATA 19 (SP by power-on)

What the ... ?

Upgrade the IOS as recommended.

[morrisce]

After resetting the 6509 and reseating the module I cleared up that error.  I was having sporadic outages on the network with uplink ports going down randomly.  It was very strange.  I found there was port flapping on our 3550s and could not clear up that either. 

Finally I launched Cisco Network Assistant and it started sending me alerts and sent me the one below.  I've unplugged the cables in these ports and all is well again.  Any thoughts why this error would occur?

Device: RH-MDF-C2950-2
000141: *Mar  1 01:16:59: %SPANTREE-CLUSTER_MEMBER_1-2-UNBLOCK_CONSIST_PORT: Unblocking FastEthernet0/34 on VLAN0004. Port consistency restored.

Thanks!