SG 300-28 Switch - Jumbo Frames Problem.

[dicksondickson]

I just got the SG 300-28 28 port switch tonight and got it up and running however, i've encountered a problem regarding jumbo frames. In the documentation and product brochures, it states the SG 300 switches supports jumbo frames up to 10k. When I first setup the switch and enabled jumbo frames, i was getting very slow speeds in my network transfers (800kb/s !!!). All the workstations are running Intel PCIE nic cards with jumbo frames enabled at 9014 bytes. After some troubleshooting, i lowered the frame size to 4088 bytes and everything returned to normal with fast speeds.

I had a suspicion that it might be the switch that is causing the network slowdown with 9k frames; I went ahead and enabled the 9k jumbo frame settings on my NICs again and started to ping other workstations on the network using the "don't fragment" flag. It turns out, the largest packet that i can send out is 8972 bytes. This is a little far from 10k frames that is stated in the documentation and brochures. Please correct me if i'm wrong, but it seems that i've stumbled into a bug in switch.

Time for a firmware update?

[David Hornstein]

Hi Dickson C,

Interesting query.. TCP, UDP and ICMP packet overhead are fairly negligible according to the information below i would think about 94 bytes for ethernet plus tcp overhead.

The switch would internally label the ethernet frame to identify what VLAN the frame is in (even Vlan 1), so an extra 4 bytes would be used within the switch for that.

Ethernet frame format:

• 6 byte dest MAC  addr
• 6 byte src MAC  addr
• [4 byte optional 802.1q VLAN Tag]
• 2 byte length/type
• 46-9014 byte data (payload)
• 4 byte CRC

Ethernet overhead bytes:
12 byte intergap + 8 preamble + 14 header + 4 trailer = 18 bytes/packet w/o 802.1q
12 byte intergap + 8 preamble + 18 header + 4 trailer = 22 bytes/packet with 802.1q

TCP encapsulated in Ethernet:
 Assuming no header compression (e.g. not PPP)
 Add 20 IPv4 header or 40 IPv6 header (no options)
 Add 20 TCP header
 Add 12 bytes optional TCP timestamps

TCP overhead can be 52 bytes

Ethernet + TCP overhead around 52+22 bytes = 74 bytes

Your Intel ethernet NIC supports around 9500-byte, so the datasheets from intel suggest for a jumbo frame , but you have it enabled at 9014 bytes.

So your  NIC enabled at 9014 bytes - 74 bytes for Ethernet and TCP packet overhead= approximately 8940 bytes of data.

You say you are getting packet data throughput around 8972 bytes.

Check my maths, I have made a few assumptions.  What you reckon, worth a call to the Small Business Support center to double check, please open a case  and report back with the results. I really may be way off in some of my assumptions.

regards Dave

http://www.cisco.com/en/US/support/tsd_cisco_small_business_support_center_contacts.html

[dicksondickson]

Hey David,

Thanks for the info! I'll definitely be calling up SB Support. The weird thing is, the switch chokes on 9k frames. If any of the gear in the network doesn't support the specified frame size, wouldn't it just fragment the packet or just drop it altogether?

However, when connected to the Cisco switch, i get SLOW transfer rates (800kb/s) with 9k frames. (Please correct me if i'm wrong, my networking knowledge is very limited). When connected to a Netgear 8 port switch, theres no problems.

Also, in the Intel driver settings, it says it does not include header sizes in the settings so the actual frame size that the Intel NIC sends out is larger. Like you stated, the overhead negligible anyways. Even with all the overhead added in, it will still be below the maximum the Cisco switch supports (10k frames).

So why does the Cisco switch choke on 9k frames?

(Documentation says the switch supports 10k frames).

Is the Intel NICs not 'compatible' with the Cisco switch?

Anyone else here encountered this problem?

[dicksondickson]

So after some lengthy conversations on the phone and email exchanges with upper level techs at Cisco as well as extensive tests on my own. I figured out the low throughput I was experiencing were due to flow control being enabled on the switch.

When flow control is enabled on the switch with jumbo frames and in combination with using Intel NIC cards, the throughput slows to a crawl being to the point of being unusable. The same test setup with Realtek NICs (onboard) yield lower throughput compared to flow control disabled but not to the point of the Intel NICs. I don't know if this is due to the switch or the Intel NICs but before getting this switch, I was using a Netgear switch with jumbo frames + flow control enabled and didn't exhibit this problem. Might be something about the Intel NICs not playing nice with the Cisco SG 300 with flow control enabled.

Now everything seems to be working fine with speeds as they should be with flow control disabled.

Just a little comment on Cisco support: Techs were courteous and patient throughout. However, it took awhile to get through the level 1 techs and answering the usual tech support 101 questions (are you using the right cables...etc etc..) and get to the meat of things.

The higher level techs was cool, they analyzed my packet captures and even got a Windows 7 64 setup to try to replicate my problem but I think they only went as far as that since there wasn't any mention of changing settings on the switch or them testing with Intel NICs and didn't really provide a solution.

When asked if they will investigate further, the response was that engineering confirms the SG300 is functioning as designed and there is nothing further to investigate. The switch does come with flow control disabled by default and everything works fine when flow control is disabled so....is this a Intel problem or Cisco problem? Or is it operator error (me)?

So if anyone out there is using this switch and using Intel NICs. Disable flow control.

So when does one need to enable flow control on the switch? I've read conflicting material on the matter. Someone shed some light on this?

Thanks!

[David Hornstein]

Hi Dickson C

Most interested in your findings.

I can only guess you spent a lot of time looking at this.

Would you spend a little bit more time and  reply with  a Service Record (SR) Number that you were given my my Technical Assistance Center ?

I am interested to see the wireshark capture file when the errant behaviour was evident.

Flow control is not one of the settings that immediately comes to mind.

When I checked out a higher end switch such as a Cisco catalyst 2960 via command line,  I noticed the following default default settings;

So looks like I am assuming that by default the connected device does not support flow-Control or can handle pause frames.- Interesting

core#sh int fa 0/1

FastEthernet0/1 is up, line protocol is up (connected)

  Hardware is Fast Ethernet, address is d057.4cf5.3781 (bia d057.4cf5.3781)

  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 100Mb/s, media type is 10/100BaseTX

  input flow-control is off, output flow-control is unsupported

  ARP type: ARPA, ARP Timeout 04:00:00

I can be contacted at dhornste  at  cisco.com

regards Dave

[dicksondickson]

Hi David,  you got mail.

Yes I did spent a lot of time looking at this. I purchased the switch based on the features that it has and it fit the bill.

So when I encountered the slow transfer rate problem, I was bummed.

Now that I don't have that problem anymore, the switch is awesome. However, I do want to know, in what situations do I need to enable flow control on the switch?

The NICs have a flow control setting as well and enabling / disabling it in the OS drivers doesn't have any effect on throughput. It only happens when it is enabled on the switch.

Just a little more info, I did tests with Realtek (onboard) NICs, Intel PCIE workstation NICs (CT model), and Intel PT Server NIC. All the Intels exhibit the same slow transfer rate when flow control is enabled on the switch. Keep in mind it ONLY happens when using 9k frames. Anything lower than 9k frames, the Intel NICs perform as expected.

When flow control is turned on in the switch and jumbo frames     enabled, with the NIC set to 9k frames I get:

Intel NICs: 800kb/s
Realtek NICs: 45 megabyte /s

However, with flow control OFF on the switch, i get:

Intel NICs: 108 - 120 megabytes /s
Realtek: 68 megabytes / s

[David Carr]

Hey Dickson,

I have seent he flow control come into play when receiving traffic from a gig connection to a 10/100 connection. 

The flow control throttles down the transfer rate, so your not dropping packets if the incoming becomes too great.

Thats when I have implemented it.

[dicksondickson]

Hi David, thanks for the info. So basically in a gigabit network where all the attached devices are running gigabit, flow control is irrelevant and absolutely not needed?

[David Carr]

Exactly, not needed.

Only when there is a mis-match.

[dicksondickson]

Thanks David and David for the info and clarification.

Awesome i'm loving the switch and the support now. I'm recommending these to my friends.

[David Carr]

Your Welcome Dickson,

Please State that the issue is resolved on the community post that way this will help others with the same issue.