topic Re: Slow loading of files for ISR routers in Routing and SD-WAN

Slow loading of files for ISR routers

sigcerder — Tue, 29 Aug 2023 06:59:31 GMT

Hello everyone.
I have GW:

```
isr 4431
```
```
isr 4331
```
```
isr 1000
```

I would like to upgrade all my routers via ansible, but when I started playbook, my firmware uploaded with wight 700 MB very slowly. For example, I can wait 50 minutes for each router, at the time of my speed download and upload 1gbs.
I'm not sure but I think I need to change my control plane, can someone give me advice about it?

Re: Slow loading of files for ISR routers

Joseph W. Doherty — Sat, 19 Aug 2023 15:51:34 GMT

Cannot speak about Ansible. I have no idea how it does its transfers.

In general though, whenever the two devices don't have LAN like RTTs, a transfer using UDP can be much slower than one using TCP. Even when using TCP, there are TCP options, which might need to be enabled, on one or both sides, to increase TCP transfer rates. Lastly, although you might not be bumping into this, writing files to flash is often slow compared to what many modern links support.

Again, cannot speak on Ansible, but having done many, many IOS downloads to remote devices, the forgoing can sometimes make huge transfer rate differences.

Re: Slow loading of files for ISR routers

MHM Cisco World — Sat, 19 Aug 2023 15:56:59 GMT

I semd you message check it

Re: Slow loading of files for ISR routers

M02@rt37 — Tue, 29 Aug 2023 07:01:44 GMT

Hello @sigcerder,

You try by

tftp

If yes, try to adjust the

tftp

blocksize (default value is 512). Increase this value.

(config)#ip tftp blocksize ?

Re: Slow loading of files for ISR routers

sigcerder — Tue, 29 Aug 2023 07:02:36 GMT

Hello everyone. I apologize, I forgot to specify that file transfer is configured via

scp

Re: Slow loading of files for ISR routers

M02@rt37 — Tue, 29 Aug 2023 07:14:15 GMT

Hello @sigcerder,

Try to adjust

ip ssh window-size

command.

--You must mandatorily use the

ip tcp window-size bytes

command (with same value as

ssh window-size

) to improve performance, as SSH works on TCP protocol.

https://content.cisco.com/chapter.sjs?uri=/searchable/chapter/content/en/us/td/docs/ios-xml/ios/sec_usr_ssh/configuration/xe-17/sec-usr-ssh-xe-17-book/sec-usr-ssh-xe-16-book_chapter_0110.html.xml#GUID-CDFC299B-CE2A-4901-BB83-F51A73046770

Re: Slow loading of files for ISR routers

Joseph W. Doherty — Tue, 29 Aug 2023 07:26:03 GMT

What I had in mind, for one of the TCP parameters settings, as also referenced in your reference, was

ip tcp window-size

An important setting on the receiver.

BTW, and important setting on the sender, is PMTUD being active.

Now that we know it's using SCP, then it appears we also want to set

ip ssh window-size

to match. (NB: an option, of which I was unaware.)

What may not be mentioned, in that reference, would be what to set

window-size

to. The ideal setting would be for the BDP (bandwidth delay product).

BTW, that reference has an interesting note, i.e.:

Note

Although the maximum supported value is 131072, it is recommended to use a maximum value of 65536, as low-end devices may have more impact on the CPU.

That's unexpected.

First time, I recall, ever seeing that mentioned, that CPU might be impacted, but maybe it does on some Cisco "low-end" devices.

What does happen, if you exceed a RWIN in excess of 64KB, the TCP implementation, on both ends, must support large windows. Also, TCP's later SACK is useful for large TCP receive windows.

Re: Slow loading of files for ISR routers

sigcerder — Tue, 29 Aug 2023 07:26:57 GMT

Yes, there has been a noticeable improvement. Following the execution of the following commands:

ip tcp window-size 131072
ip ssh window-size 131072

However, the transfer speed is still suboptimal, as I'm only able to transfer 300 MB within a 10-minute period. Furthermore, the CPU load is consistently at 96%.

What would be the most effective approach to address these issues?

Re: Slow loading of files for ISR routers

sigcerder — Mon, 21 Aug 2023 10:56:52 GMT

I was able to transfer a 680 megabyte scale file in 27 minutes

At the same time, the bandwidth of one channel is 1 Gbs and the second is 100 mbps. Perhaps there is something else I can do?

Re: Slow loading of files for ISR routers

Joseph W. Doherty — Mon, 21 Aug 2023 15:47:39 GMT

Did you take note of M02@rt37's reference, I highlighted, about potential impact to CPU, using more than 64KB?

Did you calculate optimal BDP?

"What would be the most effective approach to address these issues?"

Depends on the cause of this high CPU usage.

While CPU at 96%, might be helpful to see CPU's top 5 busiest processes.

You might also try a manual file transfer using FTP, for comparison.

Re: Slow loading of files for ISR routers

sigcerder — Tue, 29 Aug 2023 07:30:27 GMT

Yes, I read the topic, I understand the basic idea, I will try to calculate BDP and experiment.
As far as I understood this problem is actual for many people and as a basic solution not to use

scp for firmware file

transfer in production environment.

Re: Slow loading of files for ISR routers

Joseph W. Doherty — Wed, 23 Aug 2023 15:07:29 GMT

Unsure SCP, alone, is the problem, because those routers have crypto hardware, I believe. So, would still try FTP for a comparison.