Uploading Slows Download Speed (and vice versa) behind Cisco 2901 NAT

Ian Stephens · ‎07-19-2017

Hi there,

We have recently had a new leased line installed. It comes in over fiber with an NTE which converts to a 100Mb Ethernet connection (straight into the router).

We are guaranteed 100Mb up and 100Mb down from our provider.

We are running NAT on the router to provide Internet connectivity to clients private side.

We noticed that when downloading, we can achieve full throughput as this example shows:

The same is vice-versa; when we upload (without running a parallel download) we can obtain full throughput for the upload.

However, when we run both at the same time (upload and download) we notice that throughput is limited (for both).

See an example of an upload with a parallel download taking place:

... notice the decreased throughput.

We checked the CPU on the router and it's running at about 58% with both upload/download threads running at the same time. I don't think CPU is the issue here.

Perhaps it's simply the overheads of TCP/IP?

The inside interface (connected to our switch) is running at 1Gbps / full duplex.

The outside interface (connected to the NTE) is running at 100Mbps / full duplex.

Here is some of our configuration:

NAT/Routing

ip nat inside source list 100 interface GigabitEthernet0/1 overload
ip nat inside source static tcp 192.168.0.160 51413 interface GigabitEthernet0/1 51413
ip nat inside source static 192.168.0.210 x.x.x.x
ip nat inside source static 192.168.0.250 x.x.x.x
ip route 0.0.0.0 0.0.0.0 x.x.x.x

Interface

interface GigabitEthernet0/0
ip address 192.168.0.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
duplex auto
speed auto
no mop enabled

interface GigabitEthernet0/1
ip address x.x.x.x 255.255.255.248
ip nat outside
ip virtual-reassembly in
duplex auto
speed auto

We are also running CEF

ip cef

Thoughts around queueing

Perhaps running some kind of fair-queue on the interface will improve throughout but I was under the impressions that it was for slower links that have trouble like this. I am also under that impression that it uses lots of CPU.

Currently, both interfaces are running default FIFO.

Any thoughts, suggestions, explanations and help are greatly appreciated and I thank you in advance.

Georg Pauwen · ‎07-19-2017

Hello,

this could simply be an MTU issue. Try and configure:

ip mtu 1460

ip tcp adjust-mss 1420

on your interfaces.

Also, post the output of 'show interfaces GigabitEthernet0/1, if there are drops, we can look at implementing some sort of QoS.

Ian Stephens · ‎07-19-2017

Hi George,

Thank you for your reply.

If it was an MTU issue, surely it would be present when simply downloading (without uploading)? We get full throughput when only going one way.

Also, it's Ethernet to Ethernet we have the router placed in. We are able to use 1500 packet size without fragmentation or drops.

Here is the sh of GigabitEthernet0/1

Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1297400
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 142000 bits/sec, 11 packets/sec
  5 minute output rate 15000 bits/sec, 12 packets/sec
     2516901849 packets input, 2142836399 bytes, 1297115 no buffer
     Received 45832 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles 
     2790 input errors, 0 CRC, 0 frame, 2790 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     1283937838 packets output, 3980297965 bytes, 0 underruns
     0 output errors, 0 collisions, 4 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     1 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

Georg Pauwen · ‎07-19-2017

Hello,

you have a lot of output drops. Try and implement the below:

policy-map SHAPE_100
class class-default
shape average 100000000

interface GigabitEthernet0/1
service-policy output SHAPE_100

MTU can be tricky, but in your case you are probably right, and MTU might not be an issue. You can test the max MTU size by sending a ping with different packet sizes to the IP address where you try to download from, check at which size you get a reply:

C:\windows\system32>ping -f -l 1500 8.8.8.8

Pinging 8.8.8.8 with 1500 bytes of data:
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.

Ping statistics for 8.8.8.8:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

C:\windows\system32>ping -f -l 1472 8.8.8.8

Pinging 8.8.8.8 with 1472 bytes of data:
Reply from 8.8.8.8: bytes=64 (sent 1472) time=17ms TTL=45
Reply from 8.8.8.8: bytes=64 (sent 1472) time=16ms TTL=45
Reply from 8.8.8.8: bytes=64 (sent 1472) time=22ms TTL=45
Reply from 8.8.8.8: bytes=64 (sent 1472) time=15ms TTL=45

Ping statistics for 8.8.8.8:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 15ms, Maximum = 22ms, Average = 17ms

Ian Stephens · ‎07-19-2017

Hi again Georg,

We are getting responses to pings with packet size of 1472 and lower - so of course, with the 28 byte overhead, this equates to an MTU of 1500 - so that seems fine.

ryan:~ ryan$ ping -D -s 1472 8.8.4.4
PING 8.8.4.4 (8.8.4.4): 1472 data bytes
1480 bytes from 8.8.4.4: icmp_seq=0 ttl=60 time=15.523 ms
1480 bytes from 8.8.4.4: icmp_seq=1 ttl=60 time=11.091 ms
1480 bytes from 8.8.4.4: icmp_seq=2 ttl=60 time=17.449 ms

We have applied the shaping configuration you supplied but this, however, has made throughput even worse. Here is a screenshot of up/down threads:

Notice that we are only achieving 7.1MB/s for upload and only 10.1MB/s for download when running up/down together - we can usually achieve ~11.2 for both when running separately.

Georg Pauwen · ‎07-19-2017

Hello,

what is the URL where you download this from ? Do you get slow downloads just from that one site ?

That said, can you post the full config of your router ?

Ian Stephens · ‎07-19-2017

We are downloading from an external server in the Netherlands that has a 10GbE connection to the world - this thing can keep up. We are also uploading to the same server. No, download performance is impacted from all sources when we are also running upload.

Of course, config below:

sh config
Using 2304 out of 262136 bytes
!
! Last configuration change at 12:56:56 UTC Wed Jul 19 2017 by ryan
!
version 15.5
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname xxx.net
!
boot-start-marker
boot system flash:c2900-universalk9-mz.SPA.155-3.M5.bin
boot-end-marker
!
!
enable secret 5 xxx
enable password xxx
!
no aaa new-model
ethernet lmi ce
!
!
!
!
!
!
!
!
!
ip dhcp excluded-address 192.168.0.1
ip dhcp excluded-address 192.168.0.80 192.168.0.255
!
ip dhcp pool NET-POOL
network 192.168.0.0 255.255.255.0
default-router 192.168.0.1 
dns-server 8.8.8.8 8.8.4.4 
!
! 
! 
no ip domain lookup
ip domain name xxx
ip name-server 8.8.8.8
ip name-server 8.8.4.4
ip cef 
no ipv6 cef
multilink bundle-name authenticated
! 
! 
! 
! 
license udi pid CISCO2901/K9 sn xxx
! 
! 
username xxx privilege 15 secret 5 xxx
! 
redundancy
! 
! 
! 
! 
! 
! 
interface Embedded-Service-Engine0/0
no ip address
shutdown 
! 
interface GigabitEthernet0/0
ip address 192.168.0.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
duplex auto
speed auto
no mop enabled
! 
interface GigabitEthernet0/1
ip address xxx.xxx.xxx.xxx 255.255.255.248
ip nat outside
ip virtual-reassembly in
duplex auto
speed auto
! 
ip default-gateway xxx.xxx.xxx.xxx
ip forward-protocol nd
! 
no ip http server
no ip http secure-server
! 
ip nat inside source list 100 interface GigabitEthernet0/1 overload
ip nat inside source static tcp 192.168.0.160 51413 interface GigabitEthernet0/1 51413
ip nat inside source static 192.168.0.210 xxx.xxx.xxx.xxx
ip nat inside source static 192.168.0.250 xxx.xxx.xxx.xxx
ip route 0.0.0.0 0.0.0.0 xxx.xxx.xxx.xxx
! 
! 
! 
snmp-server community public RO 12
snmp-server location xxx, United Kingdom
snmp-server contact Ryan Barclay, xxx, xxx
access-list 12 permit 192.168.0.0 0.0.0.255
access-list 100 permit ip 192.168.0.0 0.0.0.255 any
! 
control-plane
! 
! 
! 
line con 0
line aux 0
line 2 
no activation-character
no exec 
transport preferred none
transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
stopbits 1
line vty 0 4
access-class 12 in
password xxx
login local
transport input ssh
! 
scheduler allocate 20000 1000
! 
end

I have used xxx to remove sensitive data.

Thanks, Georg.

Georg Pauwen · ‎07-19-2017

Hello,

on a side note, 100MB down is guaranteed by the provider...it is probably worth checking if they have actually implemented this (correctly). Since downloading from all sites is affected, I wonder if you really get 100MB.

Ian Stephens · ‎07-19-2017

Yes, we get 100Mb down when downloading one-way (tested). We get 100Mb up when uploading one-way (tested). But we get throughput issues when we do both at the same time.

Georg Pauwen · ‎07-19-2017

Hello,

sounds almost like a speed/duplex problem. Try and set GigabitEthernet to 'speed 100' and 'duplex full' manually.

Ian Stephens · ‎07-19-2017

Interesting, we set "speed 100" and "duplex full" and the NTE won't provide a link to the router even though that's the agreed settings when it's set to auto:

However, we can set the duplex to full and it still provides a link.

I've tested that with "duplex full" and performance does indeed seem better but not perfect. We are getting perhaps around 9.5 MB/s for up and down.

Georg Pauwen · ‎07-19-2017

Ryan,

9.5MB is still miserable. Is the Ethernet cable you are using Cat5e or better ?

lpassmore · ‎07-19-2017

Are you running both of these downloads on the same computer? What happens if you try it on two separate computers simultaneously?

Ian Stephens · ‎07-20-2017

Just tested running the upload on one machine and the download on the other. Same results.

When running an upload/download on separate machines, I can get ~11MB/s upload and the download stays 5-7MB/s.

If I start the upload first, the download starts very slow and works its way up to around 5-7MB/s very slowly (over 1 minute) but it fluctuates between 5-7MB/s over the course of the test - constantly up and down. The upload stays consistent at 11MB/s. I don't know if this indicates anything (FIFO).

As soon as I cancel the upload, the download works its way up to ~11MB/s pretty quickly.

Georg Pauwen · ‎07-20-2017

Hello,

sorry for the confusion: so when your inside interface (GigabitEthernet0/0) is set to auto speed/auto duplex, this is the result ? Problem solved ? What do you have connected to GigabitEthernet0//0 ?