SG200 high "Packets With Errors"

[mschubert1990]

Hi, I have a SG200-18 switch that's been in production for about 6 months. About a month ago we had a severe ice storm and everything lost power including the switch. Afterwards I started to see very high "Packets With Errors" in the interface statistics in the web gui. When looking at the Ethernet statistics, the errors are mainly "Internal MAC Receive Error" with only a few "Frame Check Sum" errors.

Before the power outtage I was seeing about 0.035% packets with errors on the heaviest loaded interface. After the power outtage I saw 1.60% packets with errors on the same interface. Other busy interfaces saw similar raise in packets with errors. Again these errors were mainly "Internal MAC Receive Error" with only a few "Frame Check Sum" errors.

I contacted Cisco and was told that the firmware could be corrupted and that I should update to the latest firmware version: 1.3.5.58 (I was running 1.3.0.62). I updated to the latest firmware (including updating the boot code). Verified that the md5 check sums listed on the web gui match that of the firmware files that I loaded onto the switch.

It's been a couple days since upgrading the switch firmware and I've noticed that the packets with errors has gone down; now around 0.3% on busiest interface. Should I be concerned by this number of packets with errors? What does "Internal MAC Receive Error" even mean?

Thank you.

[Tom Watts]

Hi Marc, you shouldn't have any errors.

But I've also found upgrading to 1.3.5.58 there is a lot of tail error in some situations and a factory reset and manual configuration clears those errors entirely.

It may be something to consider if you're seeing a noticeable performance hit.

-Tom
Please mark answered for helpful posts

[Andrew Bailey]

Tom,

I've seen a similar issue on SG300 switches.

While I haven't completely eliminatated cabling issues (I can only continuity check the cables which are ducted and perviously terminiated and the continutity check is ok) I have had an increase in "Internal MAC Receive Error" counts following upgrades to 1.3.5.58 and 1.3.7.18 firmware versions.

In my case I have four switches connected in a star topology around a central switch and interconnected via LAG groups- three swtiches are showing error counts on some ports (a mix of LAG and single ports) and one has remained error free.

At least one LAG group is showing "Internal MAC Receive Errors" at one end of the LAG but not the other- this isn't the behaviour I'd expect from a cable fault.

The "Internal MAC Receive Error" counts appear to increase disproportionally when the port or LAG is under load (eg speed testing LAN connections between Ruckus AP and Zonedirector).

Is this type of behaviour consistent with what you have seen?

In your last posted you said:-

"But I've also found upgrading to 1.3.5.58 there is a lot of tail error  in some situations and a factory reset and manual configuration clears  those errors entirely"

Are you able to definie those situations any more? Is factory defaulting the switch and restoring a backup a workable sloution or should it be a completely manual restore (ie re-creating the configuration)?

I can of course try a factory default and manual configurationre- but it will take some time and planning to implement!

Kind Regards,

Andy.

[Andrew Bailey]

Marc,

Were you able to try Tom's advice?

Did it help with your symptoms?

Kind Regards,

Andy.

[mschubert1990]

Things were okay for a while - switch was still showing packet loss but the performance was acceptable.

On the weekend I noticed an extreme decrease in internet performance (from 100+Mbps download to around 10) and began troubleshooting - determined the issue to be the switch (connecting my computer directly to modem, or directly to router did not yield the issue). Strangely though, the upload is not affected... I'm getting the speed that I should be when uploading: around 20Mbps.

I've performed the following troubleshooting to no avail: tried different ports on router and switch, factory reset the switch, upgraded firmware from 1.3.5.58 to 1.3.7.18, factory reset again...

I've gone ahead and ordered an HP switch as I know that something must have broken on this switch and I can't wait for Cisco to do all their troubleshooting and "repair at factory", etc..

I still have some more troubleshooting to do; the last thing would be to hook up my equipment to an unmanaged gigabit switch and confirm the issue is with the SG200.

I really am astonished that I'm having these issues - I could understand if I had gone to BestBuy or Futuershop and picked up a D-Link or TrendNet but Cisco... I always thought Cisco meant quality and reliability, I was sadly mistaken.

[Andrew Bailey]

Tom,

Were you able to add anything further?

Have you been able to review the comments Marc or myself have made?

Kind Regards,

Andy.

[Tom Watts]

Internal MAC receive error is defined as

"The number of frames that fail reception due to an internal MAC sublayer  receive error. This number does not include alignment, FCS, or  frame-too-long errors."

So I guess the question becomes - If you observe this behavior, is it a common link (an aggregating link to like a Router or Server) that most devices use to access something? Or are these errors occurring on the access ports where hosts connect?

If you disconnect a port that is incrementing errors then reconnect it, does the issue persist? If you take a 'known working device' and move it to the 'bad port', does this known working device also take errors?

-Tom
Please mark answered for helpful posts
http://blogs.cisco.com/smallbusiness/

[Andrew Bailey]

Tom,

Thanks for your reply.

In my network of 4 switches I have both examples. I have access ports and truck LAGs that are showing these "internal MAC receive" errors.

In one access case I can shift the accessing device (a UK SkyTV box AND it's existing Ethernet cable) to a different switch port (port configured identically) and see the "internal MAC receive" errors increment.

On one of the LAG links I seem to have there "internal MAC errors" in one direction (ie being reported at one end of the LAG but none in the other.

To requote what you said to Marc:-

"But I've also found upgrading to 1.3.5.58 there is a lot of tail  error  in some situations and a factory reset and manual configuration  clears  those errors entirely"

My question was:-

Are  you able to definie those situations any more? Is factory defaulting  the switch and restoring a backup a workable sloution or should it be a  completely manual restore (ie re-creating the configuration)?

In my situation I'm pretty clear that what I'm seeing is either failed hardware (possibly at a port level) or a bug. I doubt the Cisco hardware is likely to fail as much as my error rates would suggest so that leaves me contemplating a bug (as your response to Marc suggested).

What would you recommend as next steps?

Andy.

[Tom Watts]

Hi Andrew this is a relatively isolated incident (in fact I've heard this only 4 times counting you two). It wouldn't call this a bug but I would call it a nuissance. Like most if not all small business products, for some reason when changing firmware there is almost always some sort of weird software corruption. I don't know if it is due to code overlap or what may be the cause.

I think you're fine to make config files then just default the switch and reload the config files.

To directly address your question, it has been reported tail drops for general traffic right after upgrade without reset of the unit. A tail drop is essentially when queue becomes full and drops until the queue is free again. This was resolved by defaulting the switch.

To direct a comment to Mr. Schubert, it unfortunate you ran in to any problems. You are one of a few incidents and esentially it made you fed up. The Cisco SB switches have sold over 200,000,000 ports (200 million). If the product was unreliable or didn't meet the majority of expectations I do assure there would be a tremendous outcry.

I do hope you will find any future endeavors more pleasant.

-Tom
Please mark answered for helpful posts
http://blogs.cisco.com/smallbusiness/