SG500X Dropped packets

[Mike Horowitz]

I have been having issues for a long time with the SG500X units. I keep getting many "Drop Events", or dropped packets (as seen in the attachment). There are no real "errors" or issues that I can see otherwise. I have 2 stacked SG500X-48 port switches connected with 10GB cables. I have 13 different LAG groups that span the 2 switches (8 are down-links to SG200 switches for clients). I have 2 physical vmware boxes, and 1 SAN connected with 10GB cables. I have one main physical server that almost everyone accesses to connect to a foxpro database. The drops happen on the highest utilization ports, especially the physical server that is running foxpro. There are about 100 users total (connected to the SG200's).

Any ideas? I have gone through the configurations multiple times with Cisco, and with others. But no change. I have tried everything to correct.

[Mike Horowitz]

Still experiencing this issue. Does anyone have any ideas? I am lost after much time spent troubleshooting with various techs (from Cisco, vmware, Dell, others). Thank you in advance.

[Aleksandra Dargiel]

Hi Mike,

what is the boot code and firmware version you are running?

Aleksandra

[Mike Horowitz]

Hi Aleksandra,

Thank you for replying. Boot version is 1.3.7.01, and firmware is 1.3.7.18.

I know there is a newer version, but I have applied about 5 new firmware versions over the last 2 years to try to fix this. I will upgrade soon, but it is a bigger process now that SAN and vmware boxes are tied together over these switches.

EDIT: Of note is that the drops are on the highest utilization ports (vmware, SAN, and a very busy server hosting a foxpro database). We have been thinking it is a buffer issue onthe switch (whether a switch issue or us exceeding its capacity, though not sure how), but have not been able to confirm that.

Mike

[Aleksandra Dargiel]

Please check for any security settings such as DOS protection. Those are enabled by default with the later firmware.

[Mike Horowitz]

Storm Control and DOS Protection are Disabled. 

Block Syn-FIN Packets is Enabled.

[Aleksandra Dargiel]

Hi Mike,

Would it make any difference if you try to disable? 

Aleksandra

[Mike Horowitz]

Wow. Ironically, while I did not see this in the past, we had an issue where traffic was blocked for 60 seconds causing major issues. I disabled the TCP SYN blocking now. However, we still drop random packets at times even when not getting this message below. I just disabled the "Block SYN-FIN Packets" setting, so I will keep an eye on it for continued packet drops.

%SECURITYSUITE-I-SECSYNBLOCKED: 04:01:48 07-Jan-2015:  A TCP SYN Attack was identified on port Po11. TCP SYN traffic destined to the local system is automatically blocked for 60 second.

[Aleksandra Dargiel]

Hi Mike,

You may also try to to change threshold for identifying an attack. The default one for 1.2.9 was 20 TCP SYN PPS per ingress interface and from 1.3 was increased to 80 PPS. Maybe it is still to low for your network needs.

The default blocking time is 60 sec. Note that in case the SYN attack is still ongoing, the port will become blocked again. So that would explain your outage.

But since you are testing this disabled please let us know if there are any drop events.

Regards,

Aleksandra

[Mike Horowitz]

Hi Aleksandra,

Thanks for your help. After a lot of pain over a lot of time, I ordered replacement switches. I did everything possible, but it seems like the buffers cannot handle any type of traffic burst without dropping packets. The SG500X is similar to the 3750 line, and many users experienced similar patterns with those. It seems like these switches are not ready for vmware installs, or a foxpro database that sends out a lot of data.

Mike