Slow throughput over MPLS between windows

[Tibor M]

Hi guys,

I found a lot of forums regarding my problem, but no one solve it.

Scenario: Windows 10 doing transfer to another Windows 10 over FTP on 300mbps L2 tunnel/MPLS. One way is working on full speed, other way it's working just 20mbps.

Architecture:

Win10 A (FileZilla FTP client) -> ASA 5516 -> L2 tunnel/MPLS from ISP -> ASA5508 -> Win10 B (FileZilla FTP Server)

- no QoS

- no ACLs

- no limitations

- no FTP inspection on ASA

- pure routing

Problem:

When Win10 A upload anything over FTP to Win 10 B it's going just 20mbps

When Win10 A download anything over FTP from Win 10 B it's going almost full speed 300mbps

When Win10 B download anything from CentOS Linux HTTP (from location where Win10 A is) it's going 300mbps

So i found that any Win to Win transfer is extremly slow. Doesn't matter which Windows, I have tried from more machines to do same scenario, it's same. But Win B download from Linux HTTP server is fast. Also If I open FTP connection from linux server to Win10 B and PUT file there its also fast...

My ISP is telling me that all is OK, but it's very strange behavior why Win-Linux trasnfers are fast and why Win-Win FTP transfers (which should be fast all the time, it's not CIFS and latency is just 12ms) is slow.

I have tried to disable RSS and Chimey on windows, but it didn't helped.

Did anybody faced similar problem? If yes, can you point me what I should tell my ISP to look on?

[Dennis Mink]

I would run iperf between the two hosts, 

https://iperf.fr/

and see what throughput you get in each direction.  Take FTP out of the equation see if you can swamp the link. or at least test if you get similar results in each direction. 

[Tibor M]

so I found that problem is directly in ASA5508-X.

So what we've tested?

- collegue disconnected ASA from MPLS port in ISP switch and connected his laptop directly to it and I have run iperf from my windows to his windows and i've got 291mbps throughput.

- then collegue connected ASA 5508 back to MPLS port and connected his laptop back to LAN and I run iperf again from my windows to his windows and i've got 21mbps

- If I do same test from my site but from Linux host to his windows i'm getting 291mbps again, so wtf?

iperf on his windows is running as "iperf3.exe -s"

iperf on my windows and in linux box as "iperf3.exe -c hostname.domain.com -p 5201 -t 20 -P 3"

so there must be something wrong with ASA but I do not have any QoS or anything else. Basically ASA is in it's default settings from factory, I have just 3 inspection rules (icmp, icmp error, dns) in global policy + adjusted TCP class to decrease TTL and idle connection timeout 1:00:00. routing is static. MPLS connected interface has security level 100, same as LAN. ACL is permit ip any any.

used firmware is 9.9.1(4)

and ASA is really harming just Windows to Windows traffic. all other traffics are good.

[Florin Barhala]

Can you review "show interface Gigabit x/y " output on ASA.
Maybe you have some Layer1 cable errors.

Also when running the tests through ASA run "show perfmon".

[Tibor M]

hi,

no drops, no errors :( doing transfer from gig1/6 to gig1/5.

Interface GigabitEthernet1/5 "vlan455", is up, line protocol is up
  Hardware is Accelerator rev01, BW 1000 Mbps, DLY 10 usec
        Full-Duplex(Full-duplex), 1000 Mbps(1000 Mbps)
        Input flow control is unsupported, output flow control is off
        Description: vlan455
        MAC address 4c77.6ddb.1613, MTU 1500
        IP address 10.80.255.1, subnet mask 255.255.255.0
        442891 packets input, 245629036 bytes, 0 no buffer
        Received 5 broadcasts, 0 runts, 0 giants
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
        0 pause input, 0 resume input
        0 L2 decode drops
        395498 packets output, 500171725 bytes, 0 underruns
        0 pause output, 0 resume output
        0 output errors, 0 collisions, 0 interface resets
        0 late collisions, 0 deferred
        0 input reset drops, 0 output reset drops
        input queue (blocks free curr/low): hardware (1958/1903)
        output queue (blocks free curr/low): hardware (2047/2013)
  Traffic Statistics for "vlan455":
        442887 packets input, 237181633 bytes
        395498 packets output, 492828259 bytes
        71 packets dropped
      1 minute input rate 533 pkts/sec,  27963 bytes/sec
      1 minute output rate 685 pkts/sec,  970232 bytes/sec
      1 minute drop rate, 0 pkts/sec
      5 minute input rate 5 pkts/sec,  1148 bytes/sec
      5 minute output rate 5 pkts/sec,  2633 bytes/sec
      5 minute drop rate, 0 pkts/sec
vpnsk# show int gig1/6
Interface GigabitEthernet1/6 "route-prg-ke", is up, line protocol is up
  Hardware is Accelerator rev01, BW 1000 Mbps, DLY 10 usec
        Full-Duplex(Full-duplex), 1000 Mbps(1000 Mbps)
        Input flow control is unsupported, output flow control is off
        Description: route-prg-ke
        MAC address 4c77.6ddb.1614, MTU 1500
        IP address 172.16.201.2, subnet mask 255.255.255.0
        400303 packets input, 497376997 bytes, 0 no buffer
        Received 0 broadcasts, 0 runts, 0 giants
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
        0 pause input, 0 resume input
        0 L2 decode drops
        448232 packets output, 247093536 bytes, 0 underruns
        0 pause output, 0 resume output
        0 output errors, 0 collisions, 0 interface resets
        0 late collisions, 0 deferred
        0 input reset drops, 0 output reset drops
        input queue (blocks free curr/low): hardware (2012/1904)
        output queue (blocks free curr/low): hardware (2047/1936)
  Traffic Statistics for "route-prg-ke":
        399886 packets input, 489948223 bytes
        448232 packets output, 238554788 bytes
        547 packets dropped
      1 minute input rate 685 pkts/sec,  970612 bytes/sec
      1 minute output rate 533 pkts/sec,  28006 bytes/sec
      1 minute drop rate, 0 pkts/sec
      5 minute input rate 3 pkts/sec,  322 bytes/sec
      5 minute output rate 3 pkts/sec,  847 bytes/sec
      5 minute drop rate, 0 pkts/sec

PERFMON STATS:                     Current      Average
Xlates                                0/s          0/s
Connections                           0/s          1/s
TCP Conns                             0/s          0/s
UDP Conns                             0/s          0/s
URL Access                            0/s          0/s
URL Server Req                        0/s          0/s
TCP Fixup                             0/s          0/s
TCP Intercept Established Conns       0/s          0/s
TCP Intercept Attempts                0/s          0/s
TCP Embryonic Conns Timeout           0/s          0/s
FTP Fixup                             0/s          0/s
AAA Authen                            0/s          0/s
AAA Author                            0/s          0/s
AAA Account                           0/s          0/s
HTTP Fixup                            0/s          0/s

VALID CONNS RATE in TCP INTERCEPT:    Current      Average
                                       N/A         100.00%

[Tibor M]

I have found that I have a lot of TCP retransmissions and out of order packets on both sides when monitoring with Wireshark... but not sure which device it's causing or if it caused by MPLS

[Florin Barhala]

Are you sure FTP inspection on ASA is off ?

[Tibor M]

hi, 1 000 000% sure... was on call with Cisco TAC and it looks like problem of MPLS, but my ISP was at my office and remote location and did some RFC tests with certified device and they are telling that all is ok on their side :( could somehost 2 ASA boxes inline harm it?

strange is that out-of-order frames and re-transmissions were captured on input interface on remote ASA. packets captured on local ASA on output interface were same as computer sent. all is just on one way from local ASA to remote location. if I capture packets on way from remote to local, there are no errors :(

[Florin Barhala]

I have to subscribe to Dennis's reply: iperf can give you the final answer altough I feel this is not ISP related but Application Level stuff.

If you're stuck on Windows I would look for SCP or TFTP or just pure SMB traffic and test each one at a time.

[Tibor M · Accepted solution]

problem was on ISP MPLS, ASA boses were fine.

[cody.reed]

We are seeing the exact same problem. Did the ISP state the problem that needed fixed on the MPLS to resolve the issue?

[Lumbee]

did someone find the resolution to this? it states to click on the link for the solution but it doesnt take me anywhere?

[mohamed.alhossadi]

hi

what are the solution to that