Re: Cisco SG200-08 (SLM2008T-EU) IPv4 packet loss

gergelyimre · ‎05-12-2011

Hi

I recently bought a SG200-08 8-port gigabit switch. The switch firmware is the latest (1.0.0.16). Everything works fine, except that packet loss I have when flood-pinging on the LAN.

I have 3 machines connected to the switch, they have 1000 full duplex links, they all have Ubuntu Linux installed on them, no limiting on the switch (as far as I can tell), no limiting on the machines (nothing like htb, cbq), no QOS set up on the switch, the machines are fairly OK hardware-wise (dual-core, quad-core, six-core).

And still, it doesn't matter which way I ping, I get a consistent packet loss of 5-7% when flood pinging:

gimre@voy:~$ sudo ping -n -f 172.16.21.4 -c 10000 -q

PING 172.16.21.4 (172.16.21.4) 56(84) bytes of data.

--- 172.16.21.4 ping statistics ---

10000 packets transmitted, 9247 received, 7% packet loss, time 14944ms

rtt min/avg/max/mdev = 0.036/0.062/0.290/0.035 ms, ipg/ewma 1.494/0.051 ms

The really weird thing is that it doesn't happen on IPv6, and it's a LOT faster too:

gimre@voy:~$ sudo ping6 -f -n 2001:470:c952:0:224:1dff:fe7c:c65f -c 180000 -q

PING 2001:470:c952:0:224:1dff:fe7c:c65f(2001:470:c952:0:224:1dff:fe7c:c65f) 56 data bytes

--- 2001:470:c952:0:224:1dff:fe7c:c65f ping statistics ---

180000 packets transmitted, 180000 received, 0% packet loss, time 10017ms

rtt min/avg/max/mdev = 0.036/0.047/0.206/0.017 ms, ipg/ewma 0.055/0.056 ms

I've checked the interface on the other machine and the traffic is indeed getting through, I'm pinging the right IP address every time.

I've looked at the switch's every possible setting but I can't find the problem. It is not that big of a deal, because I rarely have to use over 500pps over the LAN but still, I would like it to not have packet loss.

There's no packet loss at around 500pps:

gimre@voy:~$ sudo ping -n 172.16.21.4 -i 0.002 -q -c 10000

PING 172.16.21.4 (172.16.21.4) 56(84) bytes of data.

--- 172.16.21.4 ping statistics ---

10000 packets transmitted, 10000 received, 0% packet loss, time 19997ms

rtt min/avg/max/mdev = 0.037/0.115/0.281/0.040 ms, ipg/ewma 1.999/0.109 ms

But after that (notice the -i parameter):

gimre@voy:~$ sudo ping -n 172.16.21.4 -i 0.0019 -q -c 10000

PING 172.16.21.4 (172.16.21.4) 56(84) bytes of data.

--- 172.16.21.4 ping statistics ---

10000 packets transmitted, 9401 received, 5% packet loss, time 15416ms

rtt min/avg/max/mdev = 0.036/0.110/0.289/0.039 ms, ipg/ewma 1.541/0.099 ms

Looks like some limitation is going on, but I can't find it. There's no other equipment between the two computers, only the SG200-08. Just for fun I did another test with IPv6 and large packets:

gimre@voy:~$ sudo ping6 -f -n 2001:470:c952:0:224:1dff:fe7c:c65f -c 180000 -q -s 1472

PING 2001:470:c952:0:224:1dff:fe7c:c65f(2001:470:c952:0:224:1dff:fe7c:c65f) 1472 data bytes

--- 2001:470:c952:0:224:1dff:fe7c:c65f ping statistics ---

180000 packets transmitted, 179999 received, 0% packet loss, time 33625ms

rtt min/avg/max/mdev = 0.133/0.167/0.448/0.023 ms, ipg/ewma 0.186/0.173 ms

That's 5300+ pps (~around 60mbps) without problems.

Any help is really appreciated.

gergelyimre · ‎05-12-2011

Seems like the empty lines got removed from my initial post, I hope it's still readable... sorry about that...

gergelyimre · ‎05-26-2011

Nobody? Not a clue, a hint?

rmanthey · ‎05-27-2011

I have seen issues with jumbo frames on IPv4 for this model of switch we are currently testing that issue with the switch. I will look into the packet loss issue. It looks like you are using the basic size for your ping of 32 bytes with frame over head of 24 or so. So the frames look to be about 56 bytes. I have seen the switch stop passing packets at 504 bytes. I will be doing further testing today. The is not a documented bug at this time. Let me do some testing today. I will post back to let you know my results.

thanks,

Cisco Small Business Support Center

Randy Manthey

CCNA, CCNA - Security

gergelyimre · ‎05-27-2011

Thanks a lot! Let me know if I can help with something. I'm away for the weekend but I'll be back Sunday afternoon. Thanks again.

rmanthey · ‎05-27-2011

I wasn't able to reproduce 100% but I used a SRW224G4-K9-NA and was able to do a flood test with no issues using packets at mtu 1472. The SLM2008T would not pass packets above 504 MTU period. The flood test I was using required me to use 1472 as the MTU size. The packet size issue has be escalated but I think it is also related to your issue, but I can't be sure with my testing tools. Please call into Cisco Small Business Support and open a case. Please reference my name and have the technician contact me. if I am not available when you call in they should email me the case number and I will take care of getting the case escalated.

Thanks,

Cisco Small Business Support Center

Randy Manthey

CCNA, CCNA - Security

1-866-606-1866

gergelyimre · ‎06-09-2011

To clarify things, I've opened a case and it turns out this is not a bug, it seems to be a feature. The switch is limiting ICMP packets, that's it. Otherwise I got no problem with it, the transfer speeds are quite nice (tested between two computers connected to the switch, 1000mbps links):

root@voy:~# iperf -c 172.16.21.6 -t 60

------------------------------------------------------------
Client connecting to 172.16.21.6, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 172.16.21.1 port 53582 connected with 172.16.21.6 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec  6.58 GBytes    942 Mbits/sec

It still would be nice to have the option to turn the ICMP limiting off, but it's not too big of a deal.

Thanks Randy for the help.

ribalda01 · ‎10-20-2011

Hello

How losing packages can this be a feature? If this is really a feature, how can we disable it :S

The test you are doing does not help you to check if the switch loses packages, because tcp will resend the missing packages, you have to do a udp test.

I am also having a bad time with this model. I dont know if it is a bad unit, or there has been a quality drop compared with the slm2008

Here are my results:

cheap 1000mb switch

newpili:/home/ricardo# ping -f 192.168.2.44 -c 10000

PING 192.168.2.44 (192.168.2.44) 56(84) bytes of data.

--- 192.168.2.44 ping statistics ---

10000 packets transmitted, 10000 received, 0% packet loss, time 3271ms

rtt min/avg/max/mdev = 0.053/0.239/1.880/0.094 ms, ipg/ewma 0.327/0.243 ms

newpili:/home/ricardo# iperf -c 192.168.2.44 -u -b 1000m

------------------------------------------------------------

Client connecting to 192.168.2.44, UDP port 5001

Sending 1470 byte datagrams

UDP buffer size: 1.91 MByte (default)

------------------------------------------------------------

[ 3] local 192.168.2.20 port 39779 connected with 192.168.2.44 port 5001

[ ID] Interval Transfer Bandwidth

[ 3] 0.0-10.0 sec 962 MBytes 807 Mbits/sec

[ 3] Sent 686497 datagrams

[ 3] Server Report:

[ 3] 0.0-10.0 sec 950 MBytes 797 Mbits/sec 0.015 ms 8612/686496 (1.3%)

[ 3] 0.0-10.0 sec 1 datagrams received out-of-order

root@ontario:~# iperf -s -u -i 1

------------------------------------------------------------

Server listening on UDP port 5001

Receiving 1470 byte datagrams

UDP buffer size: 114 KByte (default)

------------------------------------------------------------

[ 3] local 192.168.2.44 port 5001 connected with 192.168.2.20 port 39779

[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams

[ 3] 0.0- 1.0 sec 94.2 MBytes 790 Mbits/sec 0.015 ms 1305/68474 (1.9%)

[ 3] 1.0- 2.0 sec 95.4 MBytes 800 Mbits/sec 0.011 ms 733/68756 (1.1%)

[ 3] 2.0- 3.0 sec 95.1 MBytes 798 Mbits/sec 0.012 ms 817/68648 (1.2%)

[ 3] 3.0- 4.0 sec 95.2 MBytes 799 Mbits/sec 0.009 ms 897/68798 (1.3%)

[ 3] 4.0- 5.0 sec 95.0 MBytes 797 Mbits/sec 0.011 ms 834/68575 (1.2%)

[ 3] 5.0- 6.0 sec 95.0 MBytes 797 Mbits/sec 0.013 ms 791/68588 (1.2%)

[ 3] 6.0- 7.0 sec 95.1 MBytes 798 Mbits/sec 0.021 ms 844/68671 (1.2%)

[ 3] 7.0- 8.0 sec 95.1 MBytes 798 Mbits/sec 0.023 ms 828/68652 (1.2%)

[ 3] 8.0- 9.0 sec 95.3 MBytes 800 Mbits/sec 0.011 ms 762/68762 (1.1%)

[ 3] 0.0-10.0 sec 950 MBytes 797 Mbits/sec 0.016 ms 8612/686496 (1.3%)

[ 3] 0.0-10.0 sec 1 datagrams received out-of-order

newpili:/home/ricardo# iperf -c 192.168.2.44

------------------------------------------------------------

Client connecting to 192.168.2.44, TCP port 5001

TCP window size: 16.0 KByte (default)

------------------------------------------------------------

[ 3] local 192.168.2.20 port 47776 connected with 192.168.2.44 port 5001

[ ID] Interval Transfer Bandwidth

[ 3] 0.0-10.0 sec 1.09 GBytes 937 Mbits/sec

sg200-08 switch

newpili:/home/ricardo# ping -f 192.168.2.44 -c 10000

PING 192.168.2.44 (192.168.2.44) 56(84) bytes of data.

....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

--- 192.168.2.44 ping statistics ---

10000 packets transmitted, 9340 received, 6% packet loss, time 15076ms

rtt min/avg/max/mdev = 0.080/0.300/0.580/0.117 ms, ipg/ewma 1.507/0.273 ms

newpili:/home/ricardo# iperf -c 192.168.2.44 -u -b 1000m

------------------------------------------------------------

Client connecting to 192.168.2.44, UDP port 5001

Sending 1470 byte datagrams

UDP buffer size: 1.91 MByte (default)

------------------------------------------------------------

[ 3] local 192.168.2.20 port 58588 connected with 192.168.2.44 port 5001

[ ID] Interval Transfer Bandwidth

[ 3] 0.0-10.0 sec 965 MBytes 809 Mbits/sec

[ 3] Sent 688262 datagrams

[ 3] Server Report:

[ 3] 0.0-10.0 sec 951 MBytes 798 Mbits/sec 0.038 ms 9545/688261 (1.4%)

[ 3] 0.0-10.0 sec 1 datagrams received out-of-order

^Croot@ontario:~# iperf -s -u -i 1

------------------------------------------------------------

Server listening on UDP port 5001

Receiving 1470 byte datagrams

UDP buffer size: 114 KByte (default)

------------------------------------------------------------

[ 3] local 192.168.2.44 port 5001 connected with 192.168.2.20 port 58588

[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams

[ 3] 0.0- 1.0 sec 95.2 MBytes 799 Mbits/sec 0.013 ms 1014/68923 (1.5%)

[ 3] 1.0- 2.0 sec 95.1 MBytes 798 Mbits/sec 0.012 ms 966/68822 (1.4%)

[ 3] 2.0- 3.0 sec 95.1 MBytes 798 Mbits/sec 0.014 ms 875/68711 (1.3%)

[ 3] 3.0- 4.0 sec 95.1 MBytes 797 Mbits/sec 0.009 ms 1001/68812 (1.5%)

[ 3] 4.0- 5.0 sec 95.2 MBytes 799 Mbits/sec 0.012 ms 895/68804 (1.3%)

[ 3] 5.0- 6.0 sec 95.2 MBytes 799 Mbits/sec 0.009 ms 1039/68976 (1.5%)

[ 3] 6.0- 7.0 sec 95.1 MBytes 798 Mbits/sec 0.014 ms 851/68704 (1.2%)

[ 3] 7.0- 8.0 sec 95.2 MBytes 798 Mbits/sec 0.009 ms 937/68818 (1.4%)

[ 3] 8.0- 9.0 sec 95.2 MBytes 799 Mbits/sec 0.010 ms 945/68888 (1.4%)

[ 3] 9.0-10.0 sec 95.0 MBytes 797 Mbits/sec 0.012 ms 1023/68803 (1.5%)

[ 3] 0.0-10.0 sec 951 MBytes 798 Mbits/sec 0.038 ms 9545/688261 (1.4%)

[ 3] 0.0-10.0 sec 1 datagrams received out-of-order

newpili:/home/ricardo# iperf -c 192.168.2.44

------------------------------------------------------------

Client connecting to 192.168.2.44, TCP port 5001

TCP window size: 16.0 KByte (default)

------------------------------------------------------------

[ 3] local 192.168.2.20 port 47779 connected with 192.168.2.44 port 5001

[ ID] Interval Transfer Bandwidth

[ 3] 0.0-10.0 sec 1.09 GBytes 937 Mbits/sec

SLM2008

newpili:/home/ricardo# ping -f 192.168.2.44 -c 10000

PING 192.168.2.44 (192.168.2.44) 56(84) bytes of data.

--- 192.168.2.44 ping statistics ---

10000 packets transmitted, 10000 received, 0% packet loss, time 3311ms

rtt min/avg/max/mdev = 0.065/0.253/1.002/0.089 ms, ipg/ewma 0.331/0.254 ms

newpili:/home/ricardo# iperf -c 192.168.2.44 -u -b 1000m

------------------------------------------------------------

Client connecting to 192.168.2.44, UDP port 5001

Sending 1470 byte datagrams

UDP buffer size: 1.91 MByte (default)

------------------------------------------------------------

[ 3] local 192.168.2.20 port 38780 connected with 192.168.2.44 port 5001

[ ID] Interval Transfer Bandwidth

[ 3] 0.0-10.0 sec 963 MBytes 808 Mbits/sec

[ 3] Sent 686811 datagrams

[ 3] Server Report:

[ 3] 0.0-10.0 sec 951 MBytes 798 Mbits/sec 0.015 ms 8110/686810 (1.2%)

[ 3] 0.0-10.0 sec 1 datagrams received out-of-order

^Croot@ontario:~# iperf -s -u -i 1

------------------------------------------------------------

Server listening on UDP port 5001

Receiving 1470 byte datagrams

UDP buffer size: 114 KByte (default)

------------------------------------------------------------

[ 3] local 192.168.2.44 port 5001 connected with 192.168.2.20 port 38780

[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams

[ 3] 0.0- 1.0 sec 95.1 MBytes 797 Mbits/sec 0.015 ms 803/68607 (1.2%)

[ 3] 1.0- 2.0 sec 95.2 MBytes 799 Mbits/sec 0.010 ms 816/68754 (1.2%)

[ 3] 2.0- 3.0 sec 95.0 MBytes 797 Mbits/sec 0.010 ms 770/68545 (1.1%)

[ 3] 3.0- 4.0 sec 95.2 MBytes 799 Mbits/sec 0.012 ms 822/68730 (1.2%)

[ 3] 4.0- 5.0 sec 95.1 MBytes 798 Mbits/sec 0.013 ms 818/68672 (1.2%)

[ 3] 5.0- 6.0 sec 95.1 MBytes 798 Mbits/sec 0.014 ms 885/68745 (1.3%)

[ 3] 6.0- 7.0 sec 95.1 MBytes 797 Mbits/sec 0.011 ms 803/68616 (1.2%)

[ 3] 7.0- 8.0 sec 95.3 MBytes 799 Mbits/sec 0.020 ms 860/68844 (1.2%)

[ 3] 8.0- 9.0 sec 95.2 MBytes 799 Mbits/sec 0.012 ms 755/68674 (1.1%)

[ 3] 9.0-10.0 sec 95.1 MBytes 797 Mbits/sec 0.012 ms 779/68583 (1.1%)

[ 3] 0.0-10.0 sec 951 MBytes 798 Mbits/sec 0.015 ms 8110/686810 (1.2%)

[ 3] 0.0-10.0 sec 1 datagrams received out-of-order

newpili:/home/ricardo# iperf -c 192.168.2.44

------------------------------------------------------------

Client connecting to 192.168.2.44, TCP port 5001

TCP window size: 16.0 KByte (default)

------------------------------------------------------------

[ 3] local 192.168.2.20 port 47781 connected with 192.168.2.44 port 5001

[ ID] Interval Transfer Bandwidth

[ 3] 0.0-10.0 sec 1.09 GBytes 937 Mbits/sec

Summary

Router	Ping flood loss	UDP bandwidth	UDP Loss	TCP Bandwidth
Cheapo	0%	797 Mb	1.3%	937Mb
sg200-08	6%	798 Mb	1.4%	937Mb
slm2008	0%	798 Mb	1.2%	937Mb

mcr314519 · ‎06-18-2014

I have an SG200-26 switch for my home office.

4-5 Linux machines, some XEN servers. One machine is the household NFS server with everything from our photo and mp3 collection, to a master linux.git tree. Ideally a storage VLAN would hold all the NFS traffic, but the wifi machines aren't on that LAN.

My test case is TCP, both IPv4 and IPv6. A media web server spits out mp3 on port 80 (over IPsec) to my various devices/offices, getting the stuff over NFS. The music would skip. I started blaming bufferbloat on the uplink; then I realized it was happening at my desktop over gigabit ethernet. Rate limiting the media server to 400kb/s on the port-80 would significantly help. My other test case is just playing a video from the NFS server either over NFS or over port-80 to VLC. The video would skip. My HDHomeRun (ATSC receiver) can not play to my desktop through the SG200.

If I put an old 100Mb/s Belkin switch between the devices which I'm trying to stream media, then things work. So, for instance, in my lab, I unplug the cable from my desktop and the HDHomeRun from the SG200, and plug it into the 16-port Belkin, and plug the Belkin in to the SG200, then things work fine.

So the 10-year old 100Mb/s switch outperforms the SG200.

I'm running switch firmware: 1.3.7.18 now.

The music got perhaps slightly better, but the video still fails. I also upgraded the NIC in the NFS server from a Realtek to an e1000. I was then able to try turning the ethernet flow control on and off from both ends and see if that had an effect.

I also tried forcing the port on the NFS server to 100Mb/s both at the switch and machine side, and it had no effect: the SG200 can not keep up with 100Mb/s. I haven't tried 10Mb/s, but I could do that.

I've done tcpdump traces at both ends (I'm also mcr@tcpdump.org, btw.), and I clearly see TCP losing about every 6th packet and then retransmitting it.

On the advice of an IETF/Cisco colleague, I turned *off* all of the QoS mechanisms. WIth my original firmware, this had no effect. With 1.3.7.18, it seemed to improve things significantly.

This seems ridiculous.

mcr314519 · ‎06-18-2014

When I say "improve things significantly", I mean, the videos only step every 5 minutes, instead of every 1 minute.