Slow loading of files for ISR routers

[sigcerder]

Hello everyone.
I have GW:

1. isr 4431

2. isr 4331

3. isr 1000

I would like to upgrade all my routers via ansible, but when I started playbook, my firmware uploaded with wight 700 MB very slowly. For example, I can wait 50 minutes for each router, at the time of my speed download and upload 1gbs.
I'm not sure but I think I need to change my control plane, can someone give me advice about it?

[Joseph W. Doherty]

Cannot speak about Ansible.  I have no idea how it does its transfers.

In general though, whenever the two devices don't have LAN like RTTs, a transfer using UDP can be much slower than one using TCP.  Even when using TCP, there are TCP options, which might need to be enabled, on one or both sides, to increase TCP transfer rates.  Lastly, although you might not be bumping into this, writing files to flash is often slow compared to what many modern links support.

Again, cannot speak on Ansible, but having done many, many IOS downloads to remote devices, the forgoing can sometimes make huge transfer rate differences.

[MHM Cisco World]

I semd you message check it

[M02@rt37]

Hello @sigcerder,

You try by

tftp

?

If yes, try to adjust the

tftp

blocksize (default value is 512). Increase this value.

(config)#ip tftp blocksize ?

[sigcerder]

Hello everyone. I apologize, I forgot to specify that file transfer is configured via

scp

[M02@rt37 · Accepted solution]

Hello @sigcerder,

Try to adjust

ip ssh window-size

command.

--You must mandatorily use the

ip tcp window-size bytes

command (with same value as

ssh window-size

) to improve performance, as SSH works on TCP protocol.

https://content.cisco.com/chapter.sjs?uri=/searchable/chapter/content/en/us/td/docs/ios-xml/ios/sec_usr_ssh/configuration/xe-17/sec-usr-ssh-xe-17-book/sec-usr-ssh-xe-16-book_chapter_0110.html.xml#GUID-CDFC299B-CE2A-4901-BB83-F51A73046770

[Joseph W. Doherty]

What I had in mind, for one of the TCP parameters settings, as also referenced in your reference, was

ip tcp window-size

An important setting on the receiver.

BTW, and important setting on the sender, is PMTUD being active.

Now that we know it's using SCP, then it appears we also want to set

ip ssh window-size

to match.  (NB: an option, of which I was unaware.)

What may not be mentioned, in that reference, would be what to set

window-size

to.  The ideal setting would be for the BDP (bandwidth delay product).

BTW, that reference has an interesting note, i.e.: 

Note	Although the maximum supported value is 131072, it is recommended to use a maximum value of 65536, as low-end devices may have more impact on the CPU.

That's unexpected.

First time, I recall, ever seeing that mentioned, that CPU might be impacted, but maybe it does on some Cisco "low-end" devices.

What does happen, if you exceed a RWIN in excess of 64KB, the TCP implementation, on both ends, must support large windows.  Also, TCP's later SACK is useful for large TCP receive windows.

[sigcerder]

Yes, there has been a noticeable improvement. Following the execution of the following commands:

ip tcp window-size 131072
ip ssh window-size 131072

However, the transfer speed is still suboptimal, as I'm only able to transfer 300 MB within a 10-minute period. Furthermore, the CPU load is consistently at 96%.

What would be the most effective approach to address these issues?

[Joseph W. Doherty]

Did you take note of M02@rt37's reference, I highlighted, about potential impact to CPU, using more than 64KB?

Did you calculate optimal BDP?

"What would be the most effective approach to address these issues?"

Depends on the cause of this high CPU usage.

While CPU at 96%, might be helpful to see CPU's top 5 busiest processes.

You might also try a manual file transfer using FTP, for comparison.

[sigcerder]

Yes, I read the topic, I understand the basic idea, I will try to calculate BDP and experiment.
As far as I understood this problem is actual for many people and as a basic solution not to use

scp for firmware file

transfer in production environment.

[Joseph W. Doherty]

Unsure SCP, alone, is the problem, because those routers have crypto hardware, I believe.  So, would still try FTP for a comparison.

[sigcerder]

I was able to transfer a 680 megabyte scale file in 27 minutes

At the same time, the bandwidth of one channel is 1 Gbs and the second is 100 mbps. Perhaps there is something else I can do?