[Urgent] Problem of 7600 ES20 card

CSCO11777723 · ‎06-06-2012

hello,

I meet a very strange problem about the cisco 7600 ES20 line card.

I use:

- Sup 720-3BXL(MSFC3)

- ES20(20x1GE) 3CXL

Services:

-EVC 400(production)

-SVI terminate the EVC links

- MPLS VPN(20-30)

The problem we met was that the ES20 crash very frequantly.

===============================================================================================================

*Jun 7 01:25:49.127 CET: %DIAG-SP-6-TEST_RUNNING: Module 6: Running TestFabricSnakeForward{ID=24} ...

7k1.edge.nc#

*Jun 7 01:25:54.383 CET: %DIAG-SP-3-TEST_FAIL: Module 6: TestFabricSnakeForward{ID=24} has failed. Error code = 0x1 (DIAG_FAILURE)

7k1.edge.nc#

*Jun 7 01:25:54.663 CET: %C6KPWR-SP-4-DISABLED: power to module in slot 2 set off (Fabric channel errors)

==================================================================================================

*Jun 7 01:38:41.939 CET: %ICC-5-HUGE_BUFFER: Class [RPC ] with Request id 0 requested a huge buffer of Size 65016.

===============================================================================================================

*Jun 7 01:22:17.699 CET: %ESM20-DFC2-3-UNEXPECTED_GLOBAL_INT: Unexpected Global Interrupt: Xchip_1 Error

Please who know meet the same problem or have the solution?

Thanks a lot

Giuseppe Larosa · ‎06-07-2012

Hello Vincent,

try to reseat the module because it shows fabric errors

*Jun 7 01:25:54.663 CET: %C6KPWR-SP-4-DISABLED: power to module in slot 2 set off (Fabric channel errors)

If this does not work, if you have a free slot you can try to insert it in another slot.

If error keeps occurring is high time to open an RMA with TAC

Hope to help

Giuseppe

CSCO11777723 · ‎06-07-2012

Hi Giuseppe,

Thanks for your reply.

The card reboots itself several seconds after and then comes back after 5 minutes. We tried to change the slot, change the card with a spare one. Put a new chassis to load balance but no luck.

May 11 09:36:47.108 CET: %INTR_MGR-DFC1-3-INTR: Queueing Engine (Blackwater) [0]: EPMC Correctable ECC error

May 11 09:38:52.449 CET: %INTR_MGR-DFC1-3-INTR: Queueing Engine (Blackwater) [0]: EPMC Uncorrectable ECC error

May 11 09:38:52.449 CET: %ESM20-DFC1-3-UNEXPECTED_GLOBAL_INT: Unexpected Global Interrupt: Blackwater_0/Icewater_0 Error

May 11 09:38:52.453 CET: %DFCWLC-DFC1-2-UNRECOVERABLE_FAILURE: DFC WAN Line Card Unrecoverable Failure for Device: Queueing Engine (Blackwater)

%Software-forced reload

Strangely, the problem seams to append periodically. When the chassis is not rebooted we can see that the card crash precisely 10 or 40 or 90 or 120 minutes :

#dir dfc#1-disk0:

Directory of dfc#1-disk0:/

...

11 -rw- 812366 May 11 2012 11:23:50 +02:00 crashinfo_20120511-092351-CET

12 -rw- 802602 May 11 2012 11:38:52 +02:00 crashinfo_20120511-093852-CET

13 -rw- 797126 May 14 2012 17:00:02 +02:00 crashinfo_20120514-150002-CET

14 -rw- 953210 May 16 2012 16:28:32 +02:00 crashinfo_20120516-162827-CET

15 -rw- 955152 May 16 2012 17:39:34 +02:00 crashinfo_20120516-173931-CET

16 -rw- 957863 May 30 2012 18:54:46 +02:00 crashinfo_20120530-185442-CET

17 -rw- 943013 May 30 2012 19:04:46 +02:00 crashinfo_20120530-190442-CET

18 -rw- 973486 May 30 2012 19:20:28 +02:00 crashinfo_20120530-192024-CET

19 -rw- 948453 May 30 2012 19:44:46 +02:00 crashinfo_20120530-194442-CET

20 -rw- 958010 Jun 6 2012 12:59:42 +02:00 crashinfo_20120606-125938-CET

21 -rw- 951897 Jun 6 2012 14:59:42 +02:00 crashinfo_20120606-145939-CET

22 -rw- 943127 Jun 6 2012 16:29:38 +02:00 crashinfo_20120606-162939-CET

Everything works beautifully for months with the exact same ios and configuration. Upgrading ios didn't solve the problem.

Vincent

Giuseppe Larosa · ‎06-08-2012

Hello Vincent,

open a TAC service request to have them to analyze your issue.

It is quite complex to troubleshoot and you have done all the reasonable attempts to check if it is an HW problem and it looks like it isn't, or at least not in the affected linecard as you have changed slot you have changed the linecard.

Hope to help

Giuseppe