Internal MAC Receive Errors + Packets with Errors

[Filomena]

Hi,

I have one SG350X 10Gb switch trunked from its 10Gb SFP+ XG04 port to an SG350 1Gb LAN switch port (GE07)

I have a SMB server running behind the SG350X in a dedicated VLAN. InterVLAN routing is done at the SG350 level.

The setup is working perfectly since more than a year.

Since a few weeks, I was experiencing important slowdowns in my SMB transfers but only in one way: read from SMB server to any client connected through the SG350 was down to 10 Mbps. Write speed to the SMB server was normal at +800 Mbps

Clients connected directly to the SG350X were not experiencing the issue and had read/writes of 800Mbps and iperf+ at the rated link speed (1Gbps to 10 Gbps depending on the clients)

I spent a lot of time trying to tweak my SMB server and clients without luck until I noticed that the Trunk port GE07 interface on the SG350 was showing a high count of both "Internal MAC Receive Errors" and "Packets with Errors".

The only recent change on switches was updating the firmware on both switches from 2.5.7.85 -> 2.5.8.12

I tried all the following steps in this order:

1. I hard reset the 2 switches and restored the configuration following advice in this post: https://community.cisco.com/t5/small-business-switches/sg200-high-quot-packets-with-errors-quot/td-p/2355916 but no help
2. I reverted the firmware on the 2 switches: no help. I kept the old firmware for the next steps
3. I changed the port on the SG350 to another port different from GE07: same issue
4. I enabled flow control on both the XG04 port on the SG350X and the GE7 port on the SG350 but without help.
5. I used a direct new LAN cable to exclude a wall cable issue between the two ports, but it did not help
6. I used a new Tranceiver on the XG04 port, but same issue.
7. I switched to the second SFP+ 10G XG03 port: same issue
8. I properly set manually the XG03 and XG04 ports to 1000M speed since they do not support Auto Negotiation: this makes the SG350X unreachable from devices connected on the SG350!! 

Finally, I switched to the LAN 10Gb XG01 and XG02 ports on the SG350X and both worked properly this time. No errros when reading from clients connected to the SG350. These are regular RJ45 10Gb ports without a transceiver and they do both support Auto Negotiation. Also, testing on regular RJ45 1Gb ports on the SG350X fixes the issue.

Anyone can advise about the cause ?

- Cisco Transceivers cannot be set to 1000 Mb ? I doubt it, but who knows

- A bug in the SFP+ XG3/4 ports with transceivers not properly setting the port speed to 1000M ?

- Why when manually set to 1000M, these SFP+ ports are no longer reachable? 

- A setting I am missing ?

This was really wired to debug because the issue is only when reading from the SMB server, the errors show on the SG350 but the issue seems originating from the SG350X SFP+ ports config

[marce1000]

 - 1) Check port counters for involved-ports, look for errors if any. 2) Check the logs of the switch 3) Check the logs on the smb-server too (if it has the possibility to activate extended logging and or debugging  , then that may help too).

 M.

[Filomena]

Thank you,

But as I wrote, the ports statistics showed the mentioned errors count: Internal MAC Receive Errors + Packets with Errors

As I also said, the errors are reported on the GE07 port. It is not related to the SMB server as I explained, and I detail below:

Independently from the SMB server, when the SFP+ ports are set to 1 Gb, like I also said, the connection between the SG350X and SG350 is broken. The root cause is related to connecting the SG350X SFP+ ports directly as a trunk to the SG350. If I set them manually at 1Gb speed, the SG350X cannot be reached through the Trunk from the SG350.

Hopefully I can use the 1 Gb spare ports of my SG350X-24P which is logical (1Gb LAN port to 1Gb LAN port).

I am trying to understand if I am doing something wrong with the SFP+ -> 1Gb LAN ports connection ? I doubt it because setting the XG port to 1Gb breaks the connection. And no, there is nothing in the switch logs except the dropped packets count.

Is there a way to send these events to my syslog server ? They are not sent to the switch RAM or Flash log and only appear in Rmon and port statistics

[marce1000 · Accepted solution]

 - Could be a hardware problem, contact cisco at :
             https://www.cisco.com/c/en/us/support/web/tsd-cisco-small-business-support-center-contacts.html

 M.

[Filomena]

Thank you, sure, 

Do you think the issue is on the SG350 or SG350X side ? Because the problem happens on all the SG350 ports and no errors appear on the SG350X

A last note: I did not test the two SFP/LAN combo ports of the SG350: since the trunk is only 1Gb, I did not use the dedicated uplink ports on the switches. Can it be the cause ? Until now, any port could be used to trunk two switches in my experience.