Cisco CSR 1000V 16.5 speed throttled - Azure Cloud

[gangadhar.akula]

1.Download from AZURE CISCO CSR on East US location to local ubuntu box.

root@kernels:~# curl -O http://username:password@52.168.161.119/bootflash/test10M.pdf
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 10.0M 100 10.0M 0 0 115k 0 0:01:28 0:01:28 --:--:-- 116k

2.Download from Apache server on Azure, East US location.

root@kernels:~# curl -O http://52.224.179.28/test10M.pdf
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 10.0M 100 10.0M 0 0 2253k 0 0:00:04 0:00:04 --:--:-- 2486k 

3.Download from Apache server on AWS.
root@kernels:~# curl -O http://54.201.202.34/test10M.pdf
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 10.0M 100 10.0M 0 0 2032k 0 0:00:05 0:00:05 --:--:-- 2118k

It took 90 seconds to download 10MB file from Cisco CSR whereas the same file got downloaded from Apache servers in 5 seconds.

Apache server and Cisco CSR are located in East US. 

I have checked duplex mismatch, both boxes are operating at 1000Mbps and Full duplex mode.

Anyone has any ideas about what cloud be the possible reasons?

[Mark Malone]

found your thread from earlier we spoke about must have been just the hyperlink not working
is the Apache a physical server or cloud based too ?
when you run a trace route to each device is there any extra hops or discrepancies in ttl speed from same source ?
what i mean is are they in same building room etc , it could be going though a different set of devices hence why the speeds are different or taking a completely different path back

[gangadhar.akula]

Thanks for the response Mark.

Yes. Apache server is also cloud-based.

This I'm running from a ubuntu box on AWS. RTT is approximately same.

ubuntu@ip-172-31-31-79:~$ nslookup gangadharcsr.eastus.cloudapp.azure.com
Address: 52.168.161.119

ubuntu@ip-172-31-31-79:~$ nslookup gangadharubuntu.eastus.cloudapp.azure.com
Address: 52.224.179.28

ubuntu@ip-172-31-31-79:~$ ping gangadharcsr.eastus.cloudapp.azure.com -c 4
PING gangadharcsr.eastus.cloudapp.azure.com (52.168.161.119) 56(84) bytes of data.
64 bytes from 52.168.161.119: icmp_seq=1 ttl=223 time=88.4 ms
64 bytes from 52.168.161.119: icmp_seq=2 ttl=223 time=85.8 ms
64 bytes from 52.168.161.119: icmp_seq=3 ttl=223 time=85.8 ms
64 bytes from 52.168.161.119: icmp_seq=4 ttl=223 time=85.7 ms

--- gangadharcsr.eastus.cloudapp.azure.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3002ms
rtt min/avg/max/mdev = 85.724/86.469/88.436/1.174 ms
ubuntu@ip-172-31-31-79:~$ ping gangadharubuntu.eastus.cloudapp.azure.com -c 4
PING gangadharubuntu.eastus.cloudapp.azure.com (52.224.179.28) 56(84) bytes of data.
64 bytes from 52.224.179.28: icmp_seq=1 ttl=29 time=81.0 ms
64 bytes from 52.224.179.28: icmp_seq=2 ttl=29 time=81.0 ms
64 bytes from 52.224.179.28: icmp_seq=3 ttl=29 time=81.9 ms
64 bytes from 52.224.179.28: icmp_seq=4 ttl=29 time=80.9 ms

--- gangadharubuntu.eastus.cloudapp.azure.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3003ms
rtt min/avg/max/mdev = 80.975/81.261/81.947/0.398 ms

I have captured trace routes for both the boxes, they seem similar, but not the same.

Have you found any incorrect behavior in traces?

ubuntu@ip-172-31-31-79:~$ traceroute gangadharubuntu.eastus.cloudapp.azure.com
traceroute to gangadharubuntu.eastus.cloudapp.azure.com (52.224.179.28), 30 hops max, 60 byte packets
1 ec2-50-112-0-48.us-west-2.compute.amazonaws.com (50.112.0.48) 17.643 ms ec2-50-112-0-52.us-west-2.compute.amazonaws.com (50.112.0.52) 13.072 ms 13.048 ms
2 100.66.8.176 (100.66.8.176) 12.268 ms 100.66.8.120 (100.66.8.120) 12.643 ms 100.66.8.28 (100.66.8.28) 18.204 ms
3 100.66.11.42 (100.66.11.42) 22.183 ms 100.66.11.102 (100.66.11.102) 16.095 ms 100.66.10.36 (100.66.10.36) 14.488 ms
4 100.66.6.231 (100.66.6.231) 22.057 ms 100.66.7.165 (100.66.7.165) 26.350 ms 100.66.6.235 (100.66.6.235) 22.031 ms
5 100.66.4.31 (100.66.4.31) 16.455 ms 100.66.4.135 (100.66.4.135) 15.633 ms 100.66.4.1 (100.66.4.1) 16.435 ms
6 100.65.8.1 (100.65.8.1) 0.537 ms 100.65.9.1 (100.65.9.1) 1.459 ms 100.65.8.97 (100.65.8.97) 0.471 ms
7 52.93.13.64 (52.93.13.64) 1.732 ms 52.93.15.208 (52.93.15.208) 0.501 ms 52.93.15.212 (52.93.15.212) 0.483 ms
8 52.93.12.56 (52.93.12.56) 44.198 ms 52.93.12.242 (52.93.12.242) 6.492 ms 52.93.13.52 (52.93.13.52) 1.840 ms
9 52.93.13.15 (52.93.13.15) 0.663 ms 52.93.12.55 (52.93.12.55) 0.717 ms 52.93.12.113 (52.93.12.113) 0.654 ms
10 * * *
11 52.95.53.10 (52.95.53.10) 11.614 ms 52.95.53.148 (52.95.53.148) 10.358 ms 52.95.53.130 (52.95.53.130) 12.091 ms
12 52.95.52.119 (52.95.52.119) 6.247 ms 52.95.52.121 (52.95.52.121) 8.196 ms 52.95.52.53 (52.95.52.53) 7.345 ms
13 52.95.216.139 (52.95.216.139) 7.975 ms 7.053 ms 52.95.217.159 (52.95.217.159) 7.069 ms
14 be-71-0.ibr02.stb.ntwk.msn.net (104.44.8.68) 80.364 ms 85.741 ms 85.591 ms
15 be-71-0.ibr02.stb.ntwk.msn.net (104.44.8.68) 85.709 ms be-72-0.ibr02.stb.ntwk.msn.net (104.44.8.70) 79.713 ms be-6-0.ibr01.cnr03.mwh01.ntwk.msn.net (104.44.4.102) 86.641 ms
16 be-7-0.ibr01.cnr04.cys04.ntwk.msn.net (104.44.4.123) 86.500 ms 80.120 ms 52.95.217.159 (52.95.217.159) 8.291 ms
17 be-2-0.ibr01.cnr01.cys04.ntwk.msn.net (104.44.4.187) 85.602 ms 84.895 ms be-7-0.ibr01.cnr04.cys04.ntwk.msn.net (104.44.4.123) 85.839 ms
18 be-6-0.ibr01.den07.ntwk.msn.net (104.44.4.113) 86.193 ms be-7-0.ibr01.cnr04.cys04.ntwk.msn.net (104.44.4.123) 85.507 ms be-6-0.ibr01.den07.ntwk.msn.net (104.44.4.113) 81.529 ms
19 be-1-0.ibr02.den07.ntwk.msn.net (104.44.4.59) 85.130 ms 86.338 ms 84.771 ms
20 be-5-0.ibr01.cnr03.dsm05.ntwk.msn.net (104.44.4.96) 81.641 ms 86.155 ms 85.836 ms
21 be-3-0.ibr01.cnr02.dsm05.ntwk.msn.net (104.44.4.175) 82.967 ms 87.192 ms 85.471 ms
22 be-4-0.ibr02.ch1.ntwk.msn.net (104.44.4.77) 80.461 ms be-3-0.ibr01.cnr02.dsm05.ntwk.msn.net (104.44.4.175) 87.018 ms be-4-0.ibr02.ch1.ntwk.msn.net (104.44.4.77) 80.116 ms
23 be-2-0.ibr03.ch1.ntwk.msn.net (104.44.4.57) 86.851 ms be-4-0.ibr02.ch1.ntwk.msn.net (104.44.4.77) 81.054 ms be-2-0.ibr03.ch1.ntwk.msn.net (104.44.4.57) 85.473 ms
24 be-4-0.ibr01.was02.ntwk.msn.net (104.44.4.32) 81.316 ms 81.001 ms be-2-0.ibr03.ch1.ntwk.msn.net (104.44.4.57) 80.537 ms
25 be-6-0.ibr01.bl7.ntwk.msn.net (104.44.5.85) 85.979 ms 86.294 ms 86.295 ms
26 ae103-0.icr02.bl20.ntwk.msn.net (104.44.10.121) 84.601 ms 84.235 ms be-6-0.ibr01.bl7.ntwk.msn.net (104.44.5.85) 84.453 ms
27 * * *
28 * ae101-0.icr01.bl20.ntwk.msn.net (104.44.10.119) 80.972 ms *
29 * * *
30 * * *
ubuntu@ip-172-31-31-79:~$

ubuntu@ip-172-31-31-79:~$ traceroute gangadharcsr.eastus.cloudapp.azure.com
traceroute to gangadharcsr.eastus.cloudapp.azure.com (52.168.161.119), 30 hops max, 60 byte packets
1 ec2-50-112-0-50.us-west-2.compute.amazonaws.com (50.112.0.50) 16.501 ms ec2-50-112-0-48.us-west-2.compute.amazonaws.com (50.112.0.48) 15.987 ms ec2-50-112-0-54.us-west-2.compute.amazonaws.com (50.112.0.54) 18.511 ms
2 100.66.8.254 (100.66.8.254) 17.146 ms 100.66.8.82 (100.66.8.82) 17.218 ms 100.66.8.226 (100.66.8.226) 17.209 ms
3 100.66.11.110 (100.66.11.110) 15.816 ms 100.66.10.108 (100.66.10.108) 21.550 ms 100.66.11.40 (100.66.11.40) 19.310 ms
4 100.66.6.15 (100.66.6.15) 29.890 ms 100.66.7.109 (100.66.7.109) 18.769 ms 100.66.7.139 (100.66.7.139) 14.678 ms
5 100.66.4.85 (100.66.4.85) 28.114 ms 100.66.4.97 (100.66.4.97) 18.296 ms 100.66.4.71 (100.66.4.71) 28.104 ms
6 100.65.9.161 (100.65.9.161) 0.620 ms 100.65.11.129 (100.65.11.129) 6.662 ms 100.65.8.1 (100.65.8.1) 0.324 ms
7 52.93.13.64 (52.93.13.64) 0.562 ms 0.591 ms 52.93.15.214 (52.93.15.214) 0.631 ms
8 52.93.13.52 (52.93.13.52) 26.283 ms 52.93.12.62 (52.93.12.62) 17.342 ms 52.93.13.58 (52.93.13.58) 20.335 ms
9 52.93.12.47 (52.93.12.47) 1.370 ms 52.93.13.7 (52.93.13.7) 0.912 ms 52.93.12.117 (52.93.12.117) 0.908 ms
10 * * *
11 52.95.53.146 (52.95.53.146) 10.196 ms 52.95.53.132 (52.95.53.132) 17.199 ms 54.239.42.255 (54.239.42.255) 6.257 ms
12 52.95.52.53 (52.95.52.53) 7.241 ms 52.95.52.153 (52.95.52.153) 9.365 ms 52.95.52.87 (52.95.52.87) 8.220 ms
13 52.95.217.159 (52.95.217.159) 6.180 ms 52.95.216.139 (52.95.216.139) 8.036 ms 52.95.217.159 (52.95.217.159) 6.175 ms
14 be-71-0.ibr02.stb.ntwk.msn.net (104.44.8.68) 86.169 ms 52.95.52.255 (52.95.52.255) 7.248 ms 52.95.216.139 (52.95.216.139) 7.228 ms
15 52.95.217.159 (52.95.217.159) 7.043 ms be-72-0.ibr02.stb.ntwk.msn.net (104.44.8.70) 87.099 ms be-6-0.ibr01.cnr03.mwh01.ntwk.msn.net (104.44.4.102) 81.785 ms
16 be-7-0.ibr01.cnr04.cys04.ntwk.msn.net (104.44.4.123) 87.435 ms 80.685 ms 80.169 ms
17 be-2-0.ibr01.cnr01.cys04.ntwk.msn.net (104.44.4.187) 86.140 ms 81.612 ms 84.858 ms
18 be-6-0.ibr01.den07.ntwk.msn.net (104.44.4.113) 81.913 ms 85.874 ms 85.861 ms
19 be-1-0.ibr02.den07.ntwk.msn.net (104.44.4.59) 81.485 ms 84.768 ms 85.540 ms
20 be-5-0.ibr01.cnr03.dsm05.ntwk.msn.net (104.44.4.96) 85.402 ms 82.960 ms 85.285 ms
21 be-3-0.ibr01.cnr02.dsm05.ntwk.msn.net (104.44.4.175) 81.104 ms 86.635 ms 81.979 ms
22 be-4-0.ibr02.ch1.ntwk.msn.net (104.44.4.77) 80.847 ms 80.436 ms 79.135 ms
23 be-4-0.ibr02.ch1.ntwk.msn.net (104.44.4.77) 79.978 ms be-2-0.ibr03.ch1.ntwk.msn.net (104.44.4.57) 81.091 ms 85.691 ms
24 be-4-0.ibr01.was02.ntwk.msn.net (104.44.4.32) 80.759 ms 81.782 ms be-2-0.ibr03.ch1.ntwk.msn.net (104.44.4.57) 85.711 ms
25 be-2-0.ibr03.ch1.ntwk.msn.net (104.44.4.57) 85.572 ms be-6-0.ibr01.bl7.ntwk.msn.net (104.44.5.85) 85.517 ms 86.226 ms
26 ae102-0.icr02.bl7.ntwk.msn.net (104.44.9.217) 79.759 ms ae100-0.icr01.bl7.ntwk.msn.net (104.44.9.199) 84.335 ms ae102-0.icr02.bl7.ntwk.msn.net (104.44.9.217) 79.021 ms
27 * * *
28 * * ae100-0.icr01.bl7.ntwk.msn.net (104.44.9.199) 84.980 ms
29 * * *
30 * * *

[Mark Malone]

This is a difficult one to identify because there doesn't look to be an issue in the traces so you would have to try and wireshark the traffic as its downloading and see if there are any discrepancies between the transfers like re transmits that causing it to take so long compared to the Apache or again there could be a limiter on either end of these cloud setups that you cant see that could be delaying the transfer slightly. i would be more concerned if there were actual routing delays , uploading an image to a router , its not even prioritized in copp so it could also depend on the load the router is under while its being transferred , like when you ping a virtual interface you get fluctuations of time even local because ping is not prioritized at all in copp unless specifically setup too

[gangadhar.akula]

Thanks for the response, Mark.
Regarding wireshark capture,
I have captured traffic on my local box while downloading from apache and CSR boxes and compared both. I have continuously observed re-transmission of tcp packets on CSR capture whereas in apache capture I found only once. You can check it by using display filter "tcp.analysis.retransmission" but I did not understand why retransmission is happening from starting to end, is that an indication that there is packet loss or limiter?

local IP: 192.168.1.141 

CSR IP Address: 52.168.161.119
Apache IP Address: 52.224.179.28

Regarding Limiter,
Yes, there might be a limiter on Azure Cloud, I have not yet figure out this case.

Regarding Load,
Cisco CSR is just set up, no configuration changed and no traffic flows through it, You can consider that is no load.
PS: I found this problem with Azure d2 standard and d3 standard boxes, d2 have 2 CPU d3 have 4 CPU and more resources.

[Mark Malone]

Im not a wireshark expert by any means but it looks like packet loss is occurring at some level , there can be legitimate 3 in row fast re transmits but that doesn't look to be the case in that capture

good explanation of what i see in your output dup acks followed by retransmits

https://networkengineering.stackexchange.com/questions/38471/what-does-tcp-dup-ack-mean

There can be several things going on - the most common would be the use of TCP Fast Retransmission which is a mechanism by which a receiver can indicate that it has seen a gap in the received sequence numbers that implies the loss of one or more packets in transit. The repeated acknowledgements at the last known value before the gap signal which packets the sender should retransmit. This can occur without waiting for the acknowledgement timeout for the lost packet to hit on the transmitter - which, as the name implies, means recovering a lot faster.

It's also possible that the same symptom of gaps in sequence numbers might be seen in a situation where packets are being delivered out of order. As above, if the receiver sees (for example) a segment with sequence #5 followed by another with #7 before seeing sequence #6 then it might try to begin to trigger a fast retransmit. Upon seeing #6 arrive, though, it would stop sending the duplicate acknowledgements.

A less common cause would be certain media problems where certain packets might end up being seen more than once. If this is the case, however, you're likely to see other problems on the link (...including other packets showing as dupes in Wireshark).

So - if you're seeing a few random duplicate ACK's but no (or few) actual retransmissions then it's likely packets arriving out of order. If you're seeing a lot more duplicate ACK's followed by actual retransmission then some amount of packet loss is taking place. Both situations are, unfortunately, entirely possible on the global Internet. If you're seeing other kinds of duplicate packets as CRC issues and generally slow performance then it might make sense to look at link issues on your own network.

[gangadhar.akula]

Thanks for your help Mark and for above information, managed to fix the issue.

I have created a couple of CSR instances on AWS cloud to check is this something to do with Azure Cloud.

CISCO CSR 1000V - BYOL instance C4.large

Got same speed as I got on Azure. up to 100KBPS = 800Kbps

CISCO CSR 1000V - BYOL instance C4.xlarge

Got same speed as I got on Azure. up to 100KBPS = 800Kbps

CISCO CSR 1000V - AX instance C4.large

Getting better speeds. up to 5MBps = 40Mbps

Based on my experiments 16.x, precisely 16.4,16.5,16.6 and 16.7 versions of CSR has some speed throttling configured or some software related issues or trial license of BYOL is has something to do with the speed issue, these are all my speculations based on experience.

What do you think?