Trouble with Ethernet WAN

MylesselyM · ‎07-24-2013

Hi! This is my first post here. Thanks to Cisco for providing this valuable resource. I'm primarily a Unix system administrator, but I wear many hats do to the small size of the company I'm currently working for. In other words, we don't have a dedicated network engineer.

We recently turned up an AT&T Ethernet WAN circuit between our two data centers (one in Boston, the other in New Jersey). The circuit is supposed to provide 150 Mbit/s. Testing with iperf, I see the expected throughput running from New Jersey to Boston (>100 Mbit/s), but only about one fifth what is expected running from Boston to New Jersey (30 Mbit/s). There are no errors on the Cisco 3560 switches at either end. The odd thing is, if I run 5 iperf clients simultaneously in Boston, each will show just over 20 Mbit/s, which means just about 100 Mbit/s total.

Any ideas what might be wrong? Configuration snippets and iperf test results can be found at the end of the post. Thanks in advance for any suggestions.

Best wishes,

Myles

Boston Switch:

Cisco IOS Software, C3560 Software (C3560-IPBASEK9-M), Version 12.2(52)SE, RELEASE SOFTWARE (fc3)

cisco WS-C3560G-24TS (PowerPC405) processor (revision C0) with 131072K bytes of memory.

Processor board ID FOC1028Z3DD

#show interfaces gi0/27 status

Port Name Status Vlan Duplex Speed Type

Gi0/27 connected 26 full 1000 1000BaseLX SFP

#show interfaces gi0/27

GigabitEthernet0/27 is up, line protocol is up (connected)

Hardware is Gigabit Ethernet, address is 0018.b954.bd1b (bia 0018.b954.bd1b)

MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive not set

Full-duplex, 1000Mb/s, link type is force-up, media type is 1000BaseLX SFP

input flow-control is off, output flow-control is unsupported

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:34, output 00:00:01, output hang never

Last clearing of "show interface" counters never

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 0 bits/sec, 0 packets/sec

5 minute output rate 0 bits/sec, 0 packets/sec

119154244 packets input, 47574275052 bytes, 0 no buffer

Received 7261 broadcasts (7234 multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 7234 multicast, 0 pause input

0 input packets with dribble condition detected

189612796 packets output, 266293462185 bytes, 0 underruns

0 output errors, 0 collisions, 1 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 PAUSE output

0 output buffer failures, 0 output buffers swapped out

Boston Test Host:

# ifconfig eth3

eth3 Link encap:Ethernet HWaddr 00:24:81:7C:79:43

inet addr:192.168.26.4 Bcast:192.168.26.255 Mask:255.255.255.0

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:1128881484 errors:0 dropped:0 overruns:0 frame:0

TX packets:5901215833 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:126526667110 (117.8 GiB) TX bytes:8700597434340 (7.9 TiB)

Interrupt:185 Memory:fdea0000-fdec0000

# ethtool eth3

Settings for eth3:

Supported ports: [ TP ]

Supported link modes: 10baseT/Half 10baseT/Full

100baseT/Half 100baseT/Full

1000baseT/Full

Supports auto-negotiation: Yes

Advertised link modes: 10baseT/Half 10baseT/Full

100baseT/Half 100baseT/Full

1000baseT/Full

Advertised auto-negotiation: Yes

Speed: 1000Mb/s

Duplex: Full

Port: Twisted Pair

PHYAD: 1

Transceiver: internal

Auto-negotiation: on

Supports Wake-on: d

Wake-on: d

Current message level: 0x00000001 (1)

Link detected: yes

Single iperf client running in Boston:

# iperf -c 192.168.26.5 -n 1073741824 -m

------------------------------------------------------------

Client connecting to 192.168.26.5, TCP port 5001

TCP window size: 16.0 KByte (default)

------------------------------------------------------------

[ 3] local 192.168.26.4 port 36403 connected with 192.168.26.5 port 5001

[ ID] Interval Transfer Bandwidth

[ 3] 0.0-278.6 sec 1.00 GBytes 30.8 Mbits/sec

[ 3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)

Five iperf clients running simultaneously in Boston:

------------------------------------------------------------
Client connecting to 192.168.26.5, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.26.4 port 54037 connected with 192.168.26.5 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-81.5 sec   205 MBytes  21.1 Mbits/sec
[  3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)
------------------------------------------------------------
Client connecting to 192.168.26.5, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.26.4 port 54038 connected with 192.168.26.5 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-83.4 sec   205 MBytes  20.6 Mbits/sec
[  3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)
------------------------------------------------------------
Client connecting to 192.168.26.5, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.26.4 port 54039 connected with 192.168.26.5 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-85.7 sec   205 MBytes  20.0 Mbits/sec
[  3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)
------------------------------------------------------------
Client connecting to 192.168.26.5, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.26.4 port 54040 connected with 192.168.26.5 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-85.7 sec   205 MBytes  20.1 Mbits/sec
[  3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)
------------------------------------------------------------
Client connecting to 192.168.26.5, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.26.4 port 54041 connected with 192.168.26.5 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-83.0 sec   205 MBytes  20.7 Mbits/sec
[  3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)

New Jersey Switch:

Cisco IOS Software, C3560 Software (C3560-IPBASEK9-M), Version 12.2(52)SE, RELEASE SOFTWARE (fc3)

cisco WS-C3560G-48TS (PowerPC405) processor (revision D0) with 131072K bytes of memory.

Processor board ID FOC1047Z59G

#show interfaces gi0/52 status

Port Name Status Vlan Duplex Speed Type

Gi0/52 connected 26 full 1000 1000BaseLX SFP

#show interfaces gi0/52

GigabitEthernet0/52 is up, line protocol is up (connected)

Hardware is Gigabit Ethernet, address is 001a.6c4b.08b4 (bia 001a.6c4b.08b4)

MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive not set

Full-duplex, 1000Mb/s, link type is force-up, media type is 1000BaseLX SFP

input flow-control is off, output flow-control is unsupported

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:01, output 00:00:48, output hang never

Last clearing of "show interface" counters never

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 0 bits/sec, 0 packets/sec

5 minute output rate 0 bits/sec, 0 packets/sec

188270859 packets input, 264260280083 bytes, 0 no buffer

Received 224109 broadcasts (223448 multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 223448 multicast, 0 pause input

0 input packets with dribble condition detected

119392994 packets output, 47761060680 bytes, 0 underruns

0 output errors, 0 collisions, 2 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 PAUSE output

0 output buffer failures, 0 output buffers swapped out

New Jersey Test Host:

# ifconfig eth3

eth3 Link encap:Ethernet HWaddr 00:15:17:16:DF:3B

inet addr:192.168.26.5 Bcast:192.168.26.255 Mask:255.255.255.0

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:188140437 errors:0 dropped:0 overruns:0 frame:0

TX packets:119225785 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:263306036792 (245.2 GiB) TX bytes:46892828314 (43.6 GiB)

Interrupt:177 Memory:fdfa0000-fdfc0000

# ethtool eth3

Settings for eth3:

Supported ports: [ TP ]

Supported link modes: 10baseT/Half 10baseT/Full

100baseT/Half 100baseT/Full

1000baseT/Full

Supports auto-negotiation: Yes

Advertised link modes: 10baseT/Half 10baseT/Full

100baseT/Half 100baseT/Full

1000baseT/Full

Advertised auto-negotiation: Yes

Speed: 1000Mb/s

Duplex: Full

Port: Twisted Pair

PHYAD: 1

Transceiver: internal

Auto-negotiation: on

Supports Wake-on: d

Wake-on: d

Current message level: 0x00000001 (1)

Link detected: yes

Single iperf client running in New Jersey:

# iperf -c 192.168.26.4 -n 1073741824 -m

------------------------------------------------------------

Client connecting to 192.168.26.4, TCP port 5001

TCP window size: 16.0 KByte (default)

------------------------------------------------------------

[ 3] local 192.168.26.5 port 13424 connected with 192.168.26.4 port 5001

[ ID] Interval Transfer Bandwidth

[ 3] 0.0-77.7 sec 1.00 GBytes 111 Mbits/sec

[ 3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)

Joseph W. Doherty · ‎07-24-2013

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Might be lost packets in the MetroE providers cloud.

Might be one set of clients RWIN isn't large enough for th bandwidth delay product.

MylesselyM · ‎07-25-2013

I think frames are being dropped in the "cloud". I tried testing with iperf using UDP instead of TCP, setting a target bandwidth of roughly 140 Mbit/s. Quite a few datagrams were dropped.

# iperf -c 192.168.26.5 -u -b 140000000 -t 60 -i 6
------------------------------------------------------------
Client connecting to 192.168.26.5, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size:  126 KByte (default)
------------------------------------------------------------
[  3] local 192.168.26.4 port 34346 connected with 192.168.26.5 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 6.0 sec   100 MBytes   140 Mbits/sec
[  3]  6.0-12.0 sec   100 MBytes   140 Mbits/sec
[  3] 12.0-18.0 sec   100 MBytes   140 Mbits/sec
[  3] 18.0-24.0 sec   100 MBytes   140 Mbits/sec
[  3] 24.0-30.0 sec   100 MBytes   140 Mbits/sec
[  3] 30.0-36.0 sec   100 MBytes   140 Mbits/sec
[  3] 36.0-42.0 sec   100 MBytes   140 Mbits/sec
[  3] 42.0-48.0 sec   100 MBytes   140 Mbits/sec
[  3] 48.0-54.0 sec   100 MBytes   140 Mbits/sec
[  3] 54.0-60.0 sec   100 MBytes   140 Mbits/sec
[  3]  0.0-60.0 sec  1001 MBytes   140 Mbits/sec
[  3] Sent 714197 datagrams
[  3] Server Report:
[  3]  0.0-60.0 sec  1000 MBytes   140 Mbits/sec   0.168 ms  954/714196 (0.13%)
[  3]  0.0-60.0 sec  1 datagrams received out-of-order

Leo Laohoo · ‎07-24-2013

Who's doing packet shaping of your WAN traffic?

MylesselyM · ‎07-25-2013

Any shaping or QoS is being done by AT&T. To us, it's just Ethernet.

Thanks,

Myles

MylesselyM · ‎07-25-2013

AT&T shed some light on this for me, and it is now performing satisfactorily in both directions.

At each end, the 1 Gbit/s ethernet is multiplexed on 3 50 Mbit/s SONET STS channels (this is how 150 Mbit/s is achieved). The frames that wouldn't fit down the 3 STS channels were being discarded.

I added the following to the switch ports at both ends:

srr-queue bandwidth limit 14

(14% of 1 Gbit/s is roughly 140 Mbit/s)

Best wishes,

Myles