I found a lot of forums regarding my problem, but no one solve it.
Scenario: Windows 10 doing transfer to another Windows 10 over FTP on 300mbps L2 tunnel/MPLS. One way is working on full speed, other way it's working just 20mbps.
Win10 A (FileZilla FTP client) -> ASA 5516 -> L2 tunnel/MPLS from ISP -> ASA5508 -> Win10 B (FileZilla FTP Server)
- no QoS
- no ACLs
- no limitations
- no FTP inspection on ASA
- pure routing
When Win10 A upload anything over FTP to Win 10 B it's going just 20mbps
When Win10 A download anything over FTP from Win 10 B it's going almost full speed 300mbps
When Win10 B download anything from CentOS Linux HTTP (from location where Win10 A is) it's going 300mbps
So i found that any Win to Win transfer is extremly slow. Doesn't matter which Windows, I have tried from more machines to do same scenario, it's same. But Win B download from Linux HTTP server is fast. Also If I open FTP connection from linux server to Win10 B and PUT file there its also fast...
My ISP is telling me that all is OK, but it's very strange behavior why Win-Linux trasnfers are fast and why Win-Win FTP transfers (which should be fast all the time, it's not CIFS and latency is just 12ms) is slow.
I have tried to disable RSS and Chimey on windows, but it didn't helped.
Did anybody faced similar problem? If yes, can you point me what I should tell my ISP to look on?
Solved! Go to Solution.
I would run iperf between the two hosts,
and see what throughput you get in each direction. Take FTP out of the equation see if you can swamp the link. or at least test if you get similar results in each direction.
so I found that problem is directly in ASA5508-X.
So what we've tested?
- collegue disconnected ASA from MPLS port in ISP switch and connected his laptop directly to it and I have run iperf from my windows to his windows and i've got 291mbps throughput.
- then collegue connected ASA 5508 back to MPLS port and connected his laptop back to LAN and I run iperf again from my windows to his windows and i've got 21mbps
- If I do same test from my site but from Linux host to his windows i'm getting 291mbps again, so wtf?
iperf on his windows is running as "iperf3.exe -s"
iperf on my windows and in linux box as "iperf3.exe -c hostname.domain.com -p 5201 -t 20 -P 3"
so there must be something wrong with ASA but I do not have any QoS or anything else. Basically ASA is in it's default settings from factory, I have just 3 inspection rules (icmp, icmp error, dns) in global policy + adjusted TCP class to decrease TTL and idle connection timeout 1:00:00. routing is static. MPLS connected interface has security level 100, same as LAN. ACL is permit ip any any.
used firmware is 9.9.1(4)
and ASA is really harming just Windows to Windows traffic. all other traffics are good.
no drops, no errors :( doing transfer from gig1/6 to gig1/5.
Interface GigabitEthernet1/5 "vlan455", is up, line protocol is up Hardware is Accelerator rev01, BW 1000 Mbps, DLY 10 usec Full-Duplex(Full-duplex), 1000 Mbps(1000 Mbps) Input flow control is unsupported, output flow control is off Description: vlan455 MAC address 4c77.6ddb.1613, MTU 1500 IP address 10.80.255.1, subnet mask 255.255.255.0 442891 packets input, 245629036 bytes, 0 no buffer Received 5 broadcasts, 0 runts, 0 giants 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 0 pause input, 0 resume input 0 L2 decode drops 395498 packets output, 500171725 bytes, 0 underruns 0 pause output, 0 resume output 0 output errors, 0 collisions, 0 interface resets 0 late collisions, 0 deferred 0 input reset drops, 0 output reset drops input queue (blocks free curr/low): hardware (1958/1903) output queue (blocks free curr/low): hardware (2047/2013) Traffic Statistics for "vlan455": 442887 packets input, 237181633 bytes 395498 packets output, 492828259 bytes 71 packets dropped 1 minute input rate 533 pkts/sec, 27963 bytes/sec 1 minute output rate 685 pkts/sec, 970232 bytes/sec 1 minute drop rate, 0 pkts/sec 5 minute input rate 5 pkts/sec, 1148 bytes/sec 5 minute output rate 5 pkts/sec, 2633 bytes/sec 5 minute drop rate, 0 pkts/sec vpnsk# show int gig1/6 Interface GigabitEthernet1/6 "route-prg-ke", is up, line protocol is up Hardware is Accelerator rev01, BW 1000 Mbps, DLY 10 usec Full-Duplex(Full-duplex), 1000 Mbps(1000 Mbps) Input flow control is unsupported, output flow control is off Description: route-prg-ke MAC address 4c77.6ddb.1614, MTU 1500 IP address 172.16.201.2, subnet mask 255.255.255.0 400303 packets input, 497376997 bytes, 0 no buffer Received 0 broadcasts, 0 runts, 0 giants 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 0 pause input, 0 resume input 0 L2 decode drops 448232 packets output, 247093536 bytes, 0 underruns 0 pause output, 0 resume output 0 output errors, 0 collisions, 0 interface resets 0 late collisions, 0 deferred 0 input reset drops, 0 output reset drops input queue (blocks free curr/low): hardware (2012/1904) output queue (blocks free curr/low): hardware (2047/1936) Traffic Statistics for "route-prg-ke": 399886 packets input, 489948223 bytes 448232 packets output, 238554788 bytes 547 packets dropped 1 minute input rate 685 pkts/sec, 970612 bytes/sec 1 minute output rate 533 pkts/sec, 28006 bytes/sec 1 minute drop rate, 0 pkts/sec 5 minute input rate 3 pkts/sec, 322 bytes/sec 5 minute output rate 3 pkts/sec, 847 bytes/sec 5 minute drop rate, 0 pkts/sec
PERFMON STATS: Current Average Xlates 0/s 0/s Connections 0/s 1/s TCP Conns 0/s 0/s UDP Conns 0/s 0/s URL Access 0/s 0/s URL Server Req 0/s 0/s TCP Fixup 0/s 0/s TCP Intercept Established Conns 0/s 0/s TCP Intercept Attempts 0/s 0/s TCP Embryonic Conns Timeout 0/s 0/s FTP Fixup 0/s 0/s AAA Authen 0/s 0/s AAA Author 0/s 0/s AAA Account 0/s 0/s HTTP Fixup 0/s 0/s VALID CONNS RATE in TCP INTERCEPT: Current Average N/A 100.00%
I have found that I have a lot of TCP retransmissions and out of order packets on both sides when monitoring with Wireshark... but not sure which device it's causing or if it caused by MPLS
hi, 1 000 000% sure... was on call with Cisco TAC and it looks like problem of MPLS, but my ISP was at my office and remote location and did some RFC tests with certified device and they are telling that all is ok on their side :( could somehost 2 ASA boxes inline harm it?
strange is that out-of-order frames and re-transmissions were captured on input interface on remote ASA. packets captured on local ASA on output interface were same as computer sent. all is just on one way from local ASA to remote location. if I capture packets on way from remote to local, there are no errors :(