Need help to determine cause of error message?

dgj1 · ‎06-21-2009

%EARL_L2_ASIC-SW1_SPSTBY-4-DBUS_HDR_ERR: EARL L2 ASIC #0: Dbus

Hdr. Error occurred. Ctrl1 0xB08D0E3D

We recently implemented VSS 1440 on a pair of 6509E chasis's. Burned in for 3 weeks and tested for 3 weeks, no errors were reported. Two days after converting all our access switches to the VSS 6509's we started getting this error message constantly. I opened a ticket with TAC and the recommendation was to replace the SUP. After replacing the SUP, the same error is still consistenly being reported about 1 per minute. The ticket has been escalated, yet no solution has been provided. Has anyone seen this error message, or does anyone have any idea what could be causing it? All the hardware is brand new. We are running IOS version 12.2(33)SXI.

Any suggestions?

Reza Sharifi · ‎06-21-2009

dgi1,

Do you get the same error message if you fail over to the stand-by box?

We are also running VSS with IOS 12.2(33)SXI in our environment, but I have not seen this error message before. Just make sure all your hardware components are exactly the same. We had an issue with PFCs not being the same on both Sup modules and it took us some times to find the problem. We had to pull the Sups out of both boxes and do a comparison on the components and found out that one Sup had PFC3 and the other one had PFC3-XL.

HTH

Reza

dgj1 · ‎06-21-2009

Hi Reza,

Yes, we get the same error message when we fail over to the stand-by box. Also another interesting symptom after the supervisor was replaced on switch 1 -

everything came back up normally except for two portchannels. Everything was working as expected on switch 2 during the supervisor change which I replaced on switch 1, however when switch 1 came back online 2 port-channels would not send traffic beyond the VSS switch. I resolved this problem by unplugging the Ethernet cable on one of them and plugging it back in on switch 1. On the other one I resolved it by shutting down the interface and then doing a no shut on the interface on switch 1.

Note before doing this, I could only ping the SVIs located on the VSS switch. I could not ping anything beyond the VSS switch from the access switches associated with these 2 port-channels. The trunks and portchannels showed active with no errors on both sides. Not exactly the failover/stability I was hoping for with VSS. Thanks for the advisement on the PFC3 vs PFC3-XL - I'll check it out.

Reza Sharifi · ‎06-21-2009

Deanna,

When you say:

Port-channels would not send traffic beyond the VSS switch; do you mean the VSL links or port-channels going to your access box? You may already know this, but the port-channel you create for the VSL link has to be in different port-channel numbers. I.e., if on switch-1 the port-channel is 10, on switch-2 has to be 11 or any other number as long as they are not the same.

Can you post your VSS config?

Also, can you post the output of sh switch virtual role and sh switch virtual link

Thanks,

Reza

dgj1 · ‎06-21-2009

The port-channels going to my access switches.

I can't post the VSS config, or the output today. I'm suppose to be on vacation, but having trouble getting this off my mind. Although I might be able to pull it out of the sho tech output I posted in my Cisco ticket.

jimmysands73_2 · ‎06-21-2009

This is a little beyond my knowledge, but when I searched for your error (I do not have access to any Cisco CCO stuff yet) on the internet I found nothing, but when I searched for

EARL_L2_ASIC

I got this

CSCsd08468 cat6000 c7600-mpls SP crash at %EARL_L2_ASIC-SP-4-L2L3_SEQ_ERR due to invalid

packets

on Cisco site:

http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SXF/native/release/notes/OL_4164.html (its a 7.1 MB pdf)

&

This fourm seems to have your exact error also...or close to it. He replaced the SUP also, didnt fix error.

http://www.gossamer-threads.com/lists/engine?do=post_view_printable;post=98195;list=cisco

This guy below had the same error you had also,

http://markmail.org/message/ji3g4sjlonxvs4lr

It beyond my skill set to fix your issue, but I can search and maybe find others out there that can.

Again, just a stab but thats how I learn,

Jimmy

dgj1 · ‎06-21-2009

Jimmy,

Thanks for your input. Please see my recent response to Reza. Also the response I received from Cisco was:

This message indicates that the EARL L2 ASIC detected an invalid header in

the DBUS. This signifies that some malformed packet was received on the data

lookup bus by the supervisor.

Obviously I'm very concerned about things getting worse since I can't seem to find the root cause. If you come across anything I would really appreciate the help. This is a new installation and it has a lot of visibility right now if you know what I mean.

Thanks,

Deanna

jimmysands73_2 · ‎06-21-2009

From:

http://puck.nether.net/pipermail/cisco-nsp/2008-October/055670.html

Hi

We had a similar error in conjunction with an es20 and a RSP720. We had

to replace the ES20 card because it suddenly stopped to forward L2

traffic.

Jul 11 17:08:17.212: %EARL_L2_ASIC-DFC1-4-DBUS_HDR_ERR: EARL L2 ASIC #0: