Slow 10Gb link

nelsonk · ‎06-04-2012

I have a 10Gb link between two sites about a 1k miles apart. I have a Cat 6506 on one end and a Nex 7018 on the other. I have a laptop plugged in to each side. I run iperf tests with various settings changing the window sizes and find that it really doesn’t make a huge difference to the outcome of the tests.

To get a baseline I plugged another laptop in to a different port on the switches IE: PC1 = 3/4 , PC2= 2/7 and run iperf and get the following results- 6506 = 550 Mb average, 7018 = 939 Mb average.

When I try a iperf test across the 10Gb link I get 17-18Mb. If I change the test and run it with 10 parallel threads I only get 17-18 Mb per thread but can get 700-800 Mb total bandwidth. My limiting factor is my hosts are connected via 1Gb links I will not get 10 Gb.

I am trying to figure out why I am only getting 17-18 Mb per thread across this 10 Gb link.

Thanks for any ideas anyone can provide,

vmiller · ‎06-04-2012

Unless there are some layer 2 errors showing up on the link, Its doubtful that its the 10G.

http://cisconet.com/traffic-analysis/throughput/104-tcp-throughput-calculation-formula.html

Most times its TCP setup on the end stations that make things appear slow.

Leo Laohoo · ‎06-04-2012

I have a 10Gb link between two sites about a 1k miles apart.

Stupid question: Dark fibre or via an ISP?

Can you post the output to the command "sh interface ten X/Y/Z"?

nelsonk · ‎06-05-2012

Thanks for the reply.

ISP

Ethernet5/8 is up

Dedicated Interface

Hardware: 10000 Ethernet, address: 44b3.dc84.b22f (bia 44b3.dc84.b22f)

Description: 10Gb link

MTU 9216 bytes, BW 10000000 Kbit, DLY 10 usec

reliability 255/255, txload 1/255, rxload 3/255

Encapsulation ARPA

Port mode is trunk

full-duplex, 10 Gb/s, media type is 10G

Beacon is turned off

Auto-Negotiation is turned off

Input flow-control is off, output flow-control is off

Rate mode is dedicated

Switchport monitor is off

EtherType is 0x8100

Last link flapped 1week(s) 6day(s)

Last clearing of "show interface" counters never

23 interface resets

30 seconds input rate 167858264 bits/sec, 17340 packets/sec

30 seconds output rate 7295136 bits/sec, 4024 packets/sec

Load-Interval #2: 5 minute (300 seconds)

input rate 134.38 Mbps, 14.29 Kpps; output rate 5.54 Mbps, 3.16 Kpps

RX

77214448366 unicast packets 217239922 multicast packets 247003260 broadcast packets

77678699778 input packets 86652678395790 bytes

0 jumbo packets 0 storm suppression packets

0 runts 0 giants 1 CRC 0 no buffer

2 input error 1 short frame 0 overrun 0 underrun 0 ignored

0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop

0 input with dribble 0 input discard

0 Rx pause

TX

20043125899 unicast packets 301631055 multicast packets 470815729 broadcast packets

20815572683 output packets 11397424890924 bytes

0 jumbo packets

0 output error 0 collision 0 deferred 0 late collision

0 lost carrier 0 no carrier 0 babble 0 output discard

0 Tx pause

vmiller · ‎06-05-2012

Hmm. non zero counters for resets, input erros and crc.

I'd clear the counters at a specic time, then check 24 hours later.

Since the last clearing of show interfaces was "never" we don't know when those errors occured.

nelsonk · ‎06-06-2012

Here they are 29 hours later.

Ethernet5/8 is up

Dedicated Interface

Hardware: 10000 Ethernet, address: 44b3.dc84.b22f (bia 44b3.dc84.b22f)

Description: DMSP Beltline link

MTU 9216 bytes, BW 10000000 Kbit, DLY 10 usec

reliability 255/255, txload 1/255, rxload 2/255

Encapsulation ARPA

Port mode is trunk

full-duplex, 10 Gb/s, media type is 10G

Beacon is turned off

Auto-Negotiation is turned off

Input flow-control is off, output flow-control is off

Rate mode is dedicated

Switchport monitor is off

EtherType is 0x8100

Last link flapped 13:49:05

Last clearing of "show interface" counters 1d05h

1 interface resets

30 seconds input rate 105473208 bits/sec, 12087 packets/sec

30 seconds output rate 4216576 bits/sec, 3273 packets/sec

Load-Interval #2: 5 minute (300 seconds)

input rate 107.97 Mbps, 12.34 Kpps; output rate 7.11 Mbps, 3.72 Kpps

RX

1060231327 unicast packets 2602443 multicast packets 2611436 broadcast packets

1065445205 input packets 1090897061568 bytes

0 jumbo packets 0 storm suppression packets

0 runts 0 giants 0 CRC 0 no buffer

0 input error 0 short frame 0 overrun 0 underrun 0 ignored

0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop

0 input with dribble 0 input discard

0 Rx pause

TX

343355770 unicast packets 1503259 multicast packets 771391 broadcast packets

345630420 output packets 198922907428 bytes

0 jumbo packets

0 output error 0 collision 0 deferred 0 late collision

0 lost carrier 0 no carrier 0 babble 0 output discard

0 Tx pause

Leo Laohoo · ‎06-05-2012

Sorry, you got a 10 Gbps link from an ISP????

You sure about that???

vmiller · ‎06-06-2012

no low level link issues. I'd look to workstation tuning.