Cisco CSR SCP stalled issue deployed on Azure

[gangadhar.akula]

I have deployed Cisco CSR 1000v - XE 16.5 2 NICs on Microsoft Azure.

When I try to download a file to any other box from Cisco CSR using scp, It always says stalled at the end of the file download.

While downloading a 1MB file the process of download stalled exactly at 93% which means some 70 KB left.

While downloading a 10MB file the process of download stalled exactly at 99% which means some 100 KB left.

Based on this behavior, the chunk of bytes missing in each download. After waiting for some time in the stalled state, the connection closed.

Any areas that I can check to avoid this behavior.  

PS: I have observed similar issue with 16.6 and 16.4 versions.

[Mark Malone]

Hi
i had an issue like this an it was software i got around it by slowing down the transfer , i had to go right down to 1mb speed though but it didn't stall at that speed
i transfer from a Linux jump-box to the routers globally , i did it tyhis way below

scp -l 1024 cat3k_caa-universalk9.SPA.03.06.06.E.152-2.E6.bin username@routername:flash:/cat3k_caa-universalk9.SPA.03.06.06.E.152-2.E6.bin

[gangadhar.akula]

Thanks Mark for the response.

Reducing transfer speed is not working for me. 

ubuntu@ip-172-31-31-79:~$ scp -l 1024 gangadhar@52.168.161.119:/bootflash:test1M.pdf .
Password:
test1M.pdf 93% 960KB 7.4KB/s - stalled -^C

ubuntu@ip-172-31-31-79:~$ scp -l 1024 gangadhar@52.168.161.119:bootflash:/test1M.pdf .
Password:
test1M.pdf 93% 960KB 26.2KB/s - stalled -^C

ubuntu@ip-172-31-31-79:~$ scp -l 1024 gangadhar@52.168.161.119:flash:/test1M.pdf .
Password:
test1M.pdf 93% 960KB 0.0KB/s - stalled -stest1M.pdf 93% 960KB 0.0KB/s - stalled -

Connection to 52.168.161.119 closed by remote host.
lost connection
ubuntu@ip-172-31-31-79:~$ scp -l 50 gangadhar@52.168.161.119:flash:/test1M.pdf .
Password:
test1M.pdf 93% 960KB 0.2KB/s - stalled -

Any more ideas?

[Mark Malone]

is there a firewall between you and the CSR , stalled usually occurs when the scp transfer is trying to use as much BW as possible and when any type of delay is seen it stalls

there is space yes on the bootflash your uploading to yes
does the same thing happen with ftp ?

[gangadhar.akula]

Thanks Mark for the response.

I think there is no firewall in between my boxes and CISCO CSR. I have tried to download files from CISCO CSR to ubuntu server on AWS and locally deployed  Ubuntu box on a Virtual box.

I can upload files to CISCO CSR from my boxes but the download of files has a problem. 

File system

Yes. There is a lot of free space in flash.
gangadharcsr#show file systems
File Systems:

Size(b) Free(b) Type Flags Prefixes
- - opaque rw system:
- - opaque rw tmpsys:
* 7835201536 6211211264 disk rw bootflash: flash:
3606913024 3498029056 disk ro webui:
- - opaque rw null:
- - opaque ro tar:
- - network rw tftp:
33554432 33540594 nvram rw nvram:
- - opaque wo syslog:
- - network rw rcp:
- - network rw pram:
- - network rw http:
- - network rw ftp:
- - network rw scp:
- - network rw https:
- - opaque ro cns:

gangadharcsr#

Upload does not have any problem.
Any box --> Cisco CSR :

ubuntu@ip-172-31-31-79:~$ scp /var/www/html/test1M.pdf gangadhar@52.168.161.119:/bootflash:test1M_upload.pdf
Password:
test1M.pdf 100% 1024KB 1.0MB/s 00:01
Connection to 52.168.161.119 closed by remote host.
ubuntu@ip-172-31-31-79:~$ scp /var/www/html/test10M.pdf gangadhar@52.168.161.119:/bootflash:test10M_upload.pdf
Password:
test10M.pdf 100% 10MB 1.1MB/s 00:09
Connection to 52.168.161.119 closed by remote host.
root@kernels:~# scp /var/www/html/test1M.pdf gangadhar@52.168.161.119:/bootflash:test1M_upload.pdf
Password:
test1M.pdf 100% 1024KB 146.3KB/s 00:07
Connection to 52.168.161.119 closed by remote host.

Download from CISCO CSR to any box has a problem.
CISCO CSR --> AWS ubuntu server
ubuntu@ip-172-31-31-79:~$ scp gangadhar@52.168.161.119:/bootflash:test1M.pdf .
Password:
test1M.pdf 93% 960KB 28.5KB/s - stalled -^C

ubuntu@ip-172-31-31-79:~$ scp -l 1024 gangadhar@52.168.161.119:/bootflash:test1M.pdf .
Password:
test1M.pdf 93% 960KB 12.5KB/s - stalled -

CISCO CSR --> Local Ubuntu Server on Virtual box

root@kernels:~# scp gangadhar@52.168.161.119:/bootflash:test1M.pdf .
Password:
test1M.pdf 93% 960KB 15.9KB/s - stalled -^C

root@kernels:~# scp -l 1024 gangadhar@52.168.161.119:/bootflash:test1M.pdf .
Password:
test1M.pdf 93% 960KB 20.6KB/s - stalled -^

To try with FTP do I need to setup FTP server on CISCO CSR?

[Mark Malone]

Hmm thats odd its working one way , for FTP on the CSR side you only need to set an ftp password and user for it to work , its just to rule out SCP protocol as the issue itself , if FTP works then their maybe something limiting SCP transfers on download on upstream to the device

ip ftp username xxxxx
ip ftp password xxxxx

[gangadhar.akula]

Thanks for the response.

I have configured FTP on the router side using the commands that you have given.

configure terminal

ip ftp username cisco
ip ftp password cisco123

exit

show ip sockets on csr side does not show ftp port as listening socket.

Any other way to check whether FTP service is listening or not? something equivalent to netstat -lntop

From the client, I'm trying to connect to the server using telnet but it is not happening.

ubuntu@ip-172-31-31-79:~$ telnet 52.168.161.119 21
Trying 52.168.161.119...
telnet: Unable to connect to remote host: Connection refused
ubuntu@ip-172-31-31-79:~$

PS:I have configured azure security group to allow ftp traffic.

[Mark Malone]

By default on cisco devices all ports are open unless you specifically block them

you could check sometimes there is hidden commands , you could see this with sh run all | i ftp

does this only happen with this device , is there other devices in the cloud you could test over same connection to rule out the cloud part incase its do

[gangadhar.akula]

Thanks for the response, Mark.

Instead of using FTP I tried to use HTTP. I have enabled HTTP server on the box and tried to download files using curl. Using Curl I can download files with HTTP protocol. 

ubuntu@ip-172-31-31-79:~$ curl -O http://username:password@52.168.161.119/bootflash/test1M.pdf
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1024k 100 1024k 0 0 114k 0 0:00:08 0:00:08 --:--:-- 115k
ubuntu@ip-172-31-31-79:~$ curl -O http://username:password@52.168.161.119/bootflash/test10M.pdf
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 10.0M 100 10.0M 0 0 115k 0 0:01:28 0:01:28 --:--:-- 116k
ubuntu@ip-172-31-31-79:~$

With SCP:

ubuntu@ip-172-31-31-79:~$ scp gangadhar@52.168.161.119:/bootflash:test1M.pdf .
Password:
test1M.pdf 93% 960KB 23.2KB/s - stalled -

Based on above investigation the problem seems to be with SCP protocol implementation only. 

Do you think of any possible solutions?

Result of the command that you have given

gangadharcsr#sh run all | i ftp
voice statistics push ftp max-file-size 4294967295
ivr record jitter tftp: 32000
ip ftp passive
ip ftp username cisco
ip ftp password cisco123
no ip tftp claim-netascii
ip tftp blocksize 512

This is happening with other CISCO CSR devices that I have setup on Azure cloud, I do not have AWS cloud account, otherwise, I would have given it a shot.

[Mark Malone]

honestly if your not in control of the full path between source and destination you probably will need to engage with the cloud owners, i dont think theres anything specific on CSR IOS that would restrict or block SCP like that as there on by default to all protocols , i would think there is something set on cloud side thats limiting it especially that its effecting all CSRs, if there the same version IOS you could upgrade one to make sure its not some weird but but by ruling out http is working would suggest its not the individual unit or the CSR but more whats happening inpath as its transfers

[gangadhar.akula]

Thanks for all your responses, Mark. 

I have understood that this might be a problem due to traffic routing through Azure cloud network, not sure though.  Do you suggest me to contact Azure cloud team?

Can you have look at this post, which also gives you an insight into another issue?

That will give you an insight into Azure cloud network, they have a load balancer network and that distributes traffic to CSR, this is what I'm thinking of seeing the configuration of security groups.

[Mark Malone]

when i click that it says topic not found , my cloud experience is limited too though as we ourselves are only breaking into it , mostly AWS too , i would definitely contact your local Azure support and provide the information seen and explain whats happening with SCP they may be able to check the perimeter devices your passing through to see if there is some policy in place that may be restricting it

[gangadhar.akula]

Thanks for the response Mark. Seems you have corrected the link.

SCP stall issue fixed by using ssh and tcp window-size 50000 based on the response received from Arshad(arshadm@cisco.com). Earlier today, I fired the same post to the ask-csr-azure-pm@cisco.com. I have received following response in the evening.

SCP stall happens if you use ssh window size greater than 60800. Can you reduce the sip windows size and try again. If you reduce the ip ssh windows size to < 60750 you will be fine. Let me know if it works for you. This behavior is documented in one of Closed bug CSCvf89353.

ubuntu@ip-172-31-31-79:~$ ssh gangadhar@52.168.161.119
Password:
gangadharcsr#config t
Enter configuration commands, one per line. End with CNTL/Z.
gangadharcsr(config)#ip ssh window-size 50000

%% Warning: This cli may have impact on CPU. So,use only for SCP
Please configure ip tcp window-size<> with same value,for this CLI to work
gangadharcsr(config)#ip tcp window-size 50000
gangadharcsr(config)#exit
gangadharcsr#exit
Connection to 52.168.161.119 closed by remote host.
Connection to 52.168.161.119 closed.
ubuntu@ip-172-31-31-79:~$ scp gangadhar@52.168.161.119:/bootflash:test1M.pdf .
Password:
test1M.pdf 100% 1024KB 102.4KB/s 00:10
ubuntu@ip-172-31-31-79:~$ scp gangadhar@52.168.161.119:/bootflash:test10M.pdf .
Password:
test10M.pdf 100% 10MB 110.1KB/s 01:33
Connection to 52.168.161.119 closed by remote host.

However, SCP is working now but the speed is still a concern, is this solution recommended?

Reducing window size may further reduce transfer speed?

It took 93 seconds to download 10MB file from CSR and took 2 seconds to download the same file from Apache server which is next to CSR.

ubuntu@ip-172-31-31-79:~$ scp gangadhar@52.168.161.119:/bootflash:test10M.pdf .
Password:
test10M.pdf 100% 10MB 110.1KB/s 01:33
Connection to 52.168.161.119 closed by remote host.
ubuntu@ip-172-31-31-79:~$ scp gangadhar@52.224.179.28:/var/www/html/test10M.pdf .
gangadhar@52.224.179.28's password:
test10M.pdf 100% 10MB 5.0MB/s 00:02
ubuntu@ip-172-31-31-79:~$

PS: when I use 60000 window-size downloads stopped at 98%.

[Mark Malone]

However, SCP is working now but the speed is still a concern, is this solution recommended?

I dont think you have a choice as they have listed a bug you hit and provided a workaround , the other solution would be ask them is it fixed in any newer releases yet then you could upgrade an prob go full speed again in SCP, you could check but i just looked at that bug id in the toolkit and its locked to internal use by Cisco only so we cant see what releases are effected or fixed for this issue we dont have the privilege so not sure why the TAC engineer provided it ,he may not know its locked down though to outside users

[gangadhar.akula]

Thanks, Mark, for the information and explaining about bug system.

I have seen the same SCP stall issue with AWS cloud instances.

AX, BYOL instances have the same issue on versions CSR 1000v XE -16.4, 16.5,16,6 and 16.7(The latest version). Are there any fixes implemented for this issue?  @arshadm, apart from the workaround of reducing the window size.