Cisco 3750 throughput problem

[prfraczek]

Hello,

I've just added new card, transceiver,  and the cable  to my 2 servers. All connected to 2 stacked 3750. Very simple network 2 10g servers and just PC- no routing etc

Unfortunately it is not working great. I am having problem with the speed.

10gbpsServer<----fiber---->10gbpsServer- speed over 500MB/s -Good between botf servers over fibre

1gbpsPC----RJ45 ----Cisco3750---fiber--->10gbpsServer- speed 110MB/s -OK 

1gbpsPC<----RJ45 ----Cisco3750---fiber---10gbpsServer- speed Just 40MB/s -BAD download to PC 

but multiple downloads everyone 40MB/s so 8PC x40MB/s

First stat

TenGigabitEthernet2/0/1 is up, line protocol is up (connected) 
  Hardware is Ten Gigabit Ethernet, address is 6416.8d98.9a9d (bia 6416.8d98.9a9d)
  MTU 1546 bytes, BW 10000000 Kbit, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full-duplex, 10Gb/s, link type is auto, media type is 10GBase-LR
  input flow-control is off, output flow-control is unsupported 
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:40, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 6
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 727000 bits/sec, 403 packets/sec
  5 minute output rate 1968000 bits/sec, 505 packets/sec
     27466628 packets input, 16318073143 bytes, 0 no buffer
     Received 1565 broadcasts (832 multicasts)
     0 runts, 3 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 832 multicast, 0 pause input
     0 input packets with dribble condition detected
     13981705 packets output, 9577820825 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

second stat

TenGigabitEthernet1/0/1 is up, line protocol is up (connected) 
  Hardware is Ten Gigabit Ethernet, address is 6400.f1cf.bf9d (bia 6400.f1cf.bf9d)
  MTU 1546 bytes, BW 10000000 Kbit, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full-duplex, 10Gb/s, link type is auto, media type is 10GBase-LR
  input flow-control is off, output flow-control is unsupported 
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:02, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 57000 bits/sec, 1 packets/sec
  5 minute output rate 67000 bits/sec, 12 packets/sec
     63681572 packets input, 81989423204 bytes, 0 no buffer
     Received 5197 broadcasts (3268 multicasts)
     0 runts, 49 giants, 0 throttles
     1 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 3268 multicast, 0 pause input
     0 input packets with dribble condition detected
     73067900 packets output, 64970199495 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

This is the config of the switch

transceivers stats are ok

[prfraczek]

wow. No one?

[Julio E. Moisa]

Hi

Have you verified if the PC's NIC is working as full duplex and speed 1G? The bandwidth for the PC will be around 1G, also the switchports are working in duplex full?

[paul driver]

Hello

The bottleneck is going from  1gb to 10gb and the return traffic and by the sounds of it, it seems your servers are windowing ( backing off on how much traffic to send)  as a result

res
Paul

[prfraczek]

any solution for this?

[Joseph W. Doherty]

To confirm, gig to 10g, okay, but 10g to gig not okay, correct?

If so, you seeing packet drops on gig interface?

The 3750 series don't provide a lot of RAM for interface buffering (2 MB per 24 edge ports and per the uplink ports). Additionally, a 3750's buffer management, at least if QoS is enabled, and especially if using defaults, reserves RAM per ports making it unavailable for ports that need more for burst traffic. (I've occasionally have had great success tuning buffer settings to reduce egress drops.)

[prfraczek]

Correct- gig to 10g, okay, but 10g to gig not okay. can be ram issue but no one is using network. I've tested last night still the same. gig-gig perfect, 10g-10g perfect just10g to 1gig crap. Must be some setting. All setup for full duplex and 1g.

[Joseph W. Doherty]

10g to gig is fast to slow, i.e. one of the "classical" congestion scenarios. Insufficient egress buffers will throttle traffic, especially during TCP slow start.

You need to either adjust receiving host's RWIN to be smaller than networks buffers or increase network buffers to be larger than receiving host's RWIN.

[prfraczek]

you're saying classical but I am googling this 4th day and could not find any solution. It is so 'basic' config and I cannot believe it just not working straight from 'the box'. How normally people doing this? What type of switches they use? 10g server and 1g lan clients... no difficulty, so simple setup. Can you recommend something?

[paul driver]

Hello 

No matter what you do you still will have a bottleneck 10gb into 1gb  will never get you the same as directly connected 10gb links.

I  would also look into your servers  also

what type of servers are they -what O/S and application are they running?

res

paul

[prfraczek]

2 servers W2008 R2 and 2016,mainly are share servers, but also running some watch folder compression software- just Windows compatible.

What if I turn qos off? tbh I don't need any vlans, etc. I can use tham just like 'un-manage' plain switches. Will this help?

[Joseph W. Doherty]

"What if I turn qos off?"

If often does help, again, especially if you're using QoS defaults.

However, I've found tuning the buffers offers the best performance.

What I've found often works well is to shift from a buffers reserved to ports model, to all buffers allocated to the common pool and ports allocate from it, as needed. (BTW, this, I believe, was the QoS architecture of the earlier 3500 switches.)

I've also found, you might also need to adjust queue buffer ratios, because the default won't allow any one port/queue to fully use all the common buffers. Doing this, though, further limits what other queues can allocate.

Do understand, what I'm suggesting doesn't solve all congestion issues, and in fact, can create some of its own. Much depends on the nature of how your traffic is trying to use your switch.

In one case, I had an iSCSI SAN device connected to a couple of 3750 gig ports that were showing multiple drops per second. Making change, as above, I reduced the drops, on the SAN ports, to a couple of drops per day.

[Joseph W. Doherty]

Yup, the problem is "classical", i.e. fast to slow. The other "classical" is multiple interfaces sending traffic to one interface. Either situation generally causes congestion on the egress interface.

Unfortunately, although the problem is "classical", there's no one stock solution, other than providing more egress bandwidth. The latter is why you don't see the issue gig to gig, 10g to 10g or gig to 10g.