Slow WAN Data Transfer

[jjackson]

Hey All,

  Here is my setup:

Network A----->3750X----->2821(20mb P2P Fiber WAN Link)----->2801(25mb P2P Fiber WAN Link)------>3750------>Network B

The problem I am experiencing is that the fastest transfer rate I can obtain between the two networks is ~7mbs. I have no QOS or shaping of any kind enabled on any of the network equipment in the path and average 4ms latency, router to router. I have had the circuit provider check more than once on their config and they say all their equipment is configured correctly and that there is plenty of room in the pipe for us to achieve maximum transfer rates.

Where do I need to look next?

I am attempting to setup some off site storage to replicate our daily backups to. If I connect the NAS to the same switch as the backup server, we get blazing fast transfer speeds, as soon as I set the NAS at the remote location, speeds dropped to about a 10th of what I was getting locally and monitoring of the router to router link shows that the transfer rates are maxing out a ~7mbs.

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages wha2tsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

It can be lots of things, including your service provider is mistaken in the correctness of your network.

First, I would suggest a bandwidth test that confirms you can obtain, end-to-end, the bandwidth you're paying for, without drops.

If that looks good, you would want check your interfaces for drops and insure your receiving host can drive the link to full usage.

You mention you don't have any QoS, but assuming your hand-offs are Ethernet, but with much more physical bandwidth than you're contracted for, there might be drops within the provider's network.  (I also see the 2821 has 20 Mbps but the 2801 has 25 mbps?)

I would recommend you shape for your bandwidth caps, and then see if you get drops.

[Richard Burts]

Especially if the link is using an Ethernet handoff you should look for the possibility of duplex mismatch on the Ethernet interfaces. Perhaps posting the output of show interface from both routers would provide helpful information.

HTH

Rick

[jjackson]

Thanks Joseph and Richard,

  Everything seems to check out, connectivity wise. It is Ethernet hand off. Here is the sho interface command:

2821:

GigabitEthernet0/0 is up, line protocol is up
  Hardware is MV96340 Ethernet, address is 000f.352e.c8c1 (bia 000f.352e.c8c1)
  Description: Connected To Sunset Router
  Internet address is 172.16.255.1/24
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is T
  output flow-control is XON, input flow-control is XON
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:06, output 00:00:00, output hang never
  Last clearing of "show interface" counters 9w0d
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 6014000 bits/sec, 1009 packets/sec
  5 minute output rate 1320000 bits/sec, 882 packets/sec
     1949934781 packets input, 3673753093 bytes, 0 no buffer
     Received 232820 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     3003035796 packets output, 2048942741 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

2801:
FastEthernet0/1 is up, line protocol is up
  Hardware is Gt96k FE, address is 0012.da92.237f (bia 0012.da92.237f)
  Description: Connected To Sunset Metro-E
  Internet address is 172.16.255.2/24
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 22/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s, 100BaseTX/FX
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 14w2d
  Input queue: 0/75/3/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 597000 bits/sec, 542 packets/sec
  5 minute output rate 8632000 bits/sec, 872 packets/sec
     4209215518 packets input, 2501338960 bytes
     Received 318945 broadcasts, 0 runts, 0 giants, 0 throttles
     1 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog
     0 input packets with dribble condition detected
     2978309905 packets output, 3704167249 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out

I also checked the ports on the switches that these are connected to and all is clean.

[Richard Burts]

Thanks for the additional information. It may provide some insight into the possible problem.

From the 2821

Full-duplex, 1000Mb/s, media type is T

From the 2801

Full-duplex, 100Mb/s, 100BaseTX/FX

So one is saying 1000Mbps while the other says 100Mbps.

Further suggestion that there is a mismatch is in the reported txload and rxload:

 reliability 255/255, txload 1/255, rxload 1/255

reliability 255/255, txload 22/255, rxload 1/255

The 2801 says that it is 22/255 transmit load while the 2821 says it is only 1/255 receive load. I would expect that txload on one should be approximately equal to rxload on the other. But it is not.

HTH

Rick

[jjackson]

Thanks Rick,

  After using Iperf, I ran a test from Network A to network B and got the ~6mbs transfer rate. I then ran the test from Network B to Network A and was achieving ~20mbs transfer rate. I don't believe there is an issue with my equipment at this point. I finally got the circuit provider to agree to come on site and do their own analysis. Hopefully they can find the issue. 

[Richard Burts]

I am glad that the circuit provider has agreed to come on site and hope that they can find something. But I believe that the results of your iperf tests tend to confirm my suggestion that there is a mismatch between sites. When NetworkA transmits at 1000Mbps and NetworkB receives at 100Mbps there are packet drops because of the capacity mismatch and the throughput is low. But when NetworkB transmits at 100 Mbps and NetworkA receives at 1000Mbps there is no packet drop and your throughput is ~20.

HTH

Rick