cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
47080
Views
112
Helpful
22
Replies

3850 Flash - Slow File Transfer rates

During a recent 3850 switch upgrade  I noticed I got very slow data rate (~200 KB/sec & took around 24 min) when copying new image to 3850 Flash.

cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin (213MB)        

Since it was from a Network TFTP server, I tried it again with local TFTP from a PC directly connected to 3850 and still got same low file transfer rates.

Did anyone else notice this ? What could be the cause for this slow file transfer rates ?

Rasika

1 Accepted Solution

Accepted Solutions

Richard Primm
Cisco Employee
Cisco Employee

Hello Rasika,

By default, the Catalyst 3850 uses a tftp block size value of 512, which is the lowest possible value.  The reasoning behind this default value is to ensure interoperability with legacy tftp servers. 

 

3850(config)#ip tftp blocksize ?
  <512-8192>  blocksize value

 

For IOS-XE version 3.3.2 and below, you will have to manually change the block size in the global configuration to speed up the transfer process.  The example below is a transfer comparison when using the default block size of 512K versus a transfer using the maximum block size value of 8192K. 

 

3850#copy tftp flash:
Address [172.16.108.26]?
Source filename [IOS/3850/cat3k_caa-universalk9.SPA.03.03.02.SE.150-1.EZ2.bin
Destination filename [cat3k_caa-universalk9.SPA.03.03.02.SE.150-1.EZ2.bin]?
Accessing tftp://*****@172.16.108.26/cat3k_caa-universalk9.SPA.03.03.02.SE.150-1.EZ2.bin...
Loading cat3k_caa-universalk9.SPA.03.03.02.SE.150-1.EZ2.bin from 172.16.108.26 (via Vlan1): !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[OK - 257243236 bytes]
 
257243236 bytes copied in 1416.360 secs (181623 bytes/sec)

 

The results show that the 245mb .bin file was transferred in 1416.360 seconds, which equals almost 24 minutes.  I will now adjust the block size value to the maximum allowed value of 8192.

 

3850#config t
3850(config)#ip tftp blocksize 8192
3850(config)#end

 

The same transfer is now run again.

 

3850#copy tftp flash:
Address [172.16.108.26]?
Source filename [IOS/3850/cat3k_caa-universalk9.SPA.03.03.02.SE.150-1.EZ2.bin
Destination filename [cat3k_caa-universalk9.SPA.03.03.02.SE.150-1.EZ2.bin]?
Accessing tftp://*****@172.16.108.26/cat3k_caa-universalk9.SPA.03.03.02.SE.150-1.EZ2.bin...
Loading cat3k_caa-universalk9.SPA.03.03.02.SE.150-1.EZ2.bin from 172.16.108.26 (via Vlan1):
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[OK - 257243236 bytes]
 
257243236 bytes copied in 124.100 secs (2072871 bytes/sec)

 

The results are clearly better.   The same file was transferred in just over two minutes with the new block size value.

 

Hope that helps!!

 

Luke

 

 

 

View solution in original post

22 Replies 22

Jonn cos
Level 4
Level 4

What tftp server you are using ? have you checked duplex negotiation issues ?

Tried with multiple tftp servers like tftpd32 on windows 7, in builtin tftp on Macbook OSX & linux tftp server on network. I am sure ports are full duplex & 1G speeds

Also checked with SCP & speed was better (double the speed aroun 400KB/sec), still took around 10mins to copy the image.Still too slow from my perspective.

Rasika

Try sticking the IOS into a flash drive and compare the outcomes.

I will do this tomorrow & let you know Leo... already off for the day

No problem.  Enjoy your day off. 

Just curious, after uploading did it do CRC check ?

Hi Jonn,

Not too sure, no visible indication.

But the time observed is just copy the file to flash.

Regards

Rasika

Hi Leo,

Here is the result of copying from USB to 3850 Flash, file transfer completed within 60s & this is what I expected via TFTP as well.

3850-1#copy usbflash0:cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin flash:

Destination filename [cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin]?

Copy in progress...CCCCC.....CCCCC

223734376 bytes copied in 60.180 secs (3717753 bytes/sec)

Here is the result I am getting with TFTP via directly connected PC. It is more than 20 minutes

3850-1#copy tftp://10.15.5.253/cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin flash:

Destination filename [cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin]?

Accessing tftp://10.15.5.253/cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin...

Loading cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin from 10.15.5.253 (via Vlan999): !!!!!!!!    !!!!!

[OK - 223734376 bytes]

223734376 bytes copied in 1212.560 secs (184514 bytes/sec)

There is no duplex/collision issues as I can see laptop directly connected & negotiated to Full duplex/1000Mbps.

3850-1#sh int g1/0/1

GigabitEthernet1/0/1 is up, line protocol is up (connected)

  Hardware is Gigabit Ethernet, address is 58bf.eab6.5681 (bia 58bf.eab6.5681)

  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX

Copy the same file to a 6500 flash & it took around 5 min, much faster than 3850/3750X

Why that much of drastic (>20 times) file transfer rate difference between locally attached device (USB) compare and Network server ? Cannot be writing speed to the 3850 flash, may be there is an efficient way of writing into disk when copy from locally attached USB ?

From the practical point of view copy an image from a network server would be the common use case & wonder if there is any way make this faster.

Thanks

Rasika

(184514 bytes/sec)

Hmmmm ... that's pretty slow for over the network.

Something is slowing things down.

Do you have anti-virus enabled from the TFTP server? 

Let's do the basics first before you consider checking routing.

Is there drastic differences the way wrting to flash via TFTP

Compare the speed when you run HTTP.

Here is further more test result. MacBook Air is directly connected to switch g1/0/1  with no any other connectivity to this switch. Laptop IP 10.15.5.254/23 & Switch SVI-10.15.4.255/23

Copy the same file via TFTP & SCP (renamed the file to "telst.file" for simplicity)

3850-1#copy tftp://10.15.5.254/test.file flash:

Destination filename [test.file]?

Accessing tftp://10.15.5.254/test.file...

Loading test.file from 10.15.5.254 (via Vlan999): !!!!!!  !!!!!

[OK - 223734376 bytes]

223734376 bytes copied in 1195.530 secs (187142 bytes/sec)

3850-1#copy scp://10.15.5.254/test.file flash:

Address or name of remote host [10.15.5.254]?

Source username [scp]?

Source filename [test.file]?

Destination filename [test.file]?

%Warning:There is a file already existing with this name

Do you want to over write? [confirm]

Password:

Sending file modes: C0777 223734376 test.file

!!!!!!!!!!!!!!!   !!!!!!!!!

223734376 bytes copied in 544.870 secs (410620 bytes/sec)

Regards

Rasika

So directly TFTP is slow when network connected????

This smells like a bug, Rasika.  I'd recommend you open a TAC Case.

Yes, I am directly TFTP, SCP.

Did the same with FTP & had much improved results, it took only 145s

3850-1#copy ftp: flash:

Address or name of remote host [10.15.5.254]?

Source filename [test.file]?

Destination filename [test.file]?

%Warning:There is a file already existing with this name

Do you want to over write? [confirm]

Accessing ftp://10.15.5.254/test.file...

Loading test.file !!!!!!!!!  !!!!!!!!

[OK - 223734376/4096 bytes]

223734376 bytes copied in 144.150 secs (1552094 bytes/sec)

It's looks like issue is related to TFTP most, I will see if TAC can provide some reasonable justification for this.

Thanks for the help

Rasika

Hmmmmm ... You could potentially try this ...

1.  Telnet into the switch.  DO NOT use SSH (higher CPU and memory).

2.  Run the TFTP copy again.

3.  Telnet into the switch again (using a different session) and check the CPU and memory utilization.

I am interested to know if the slow copy is due to memory or CPU utilization.

By the way, what is the CURRENT IOS version?

Did you see the same response with the NEWER IOS?

Hi Leo,

Here is the outcome of the testing I did following your steps.  It's looks like FTP is more CPU intensive, still it transfer file much faster than the TFTP. This result based on windown7 PC as TFTP/FTP server sitting on my network. (not directly connected). Still result is similar to yesterday.

Switch Ports Model              SW Version        SW Image              Mode  

------ ----- -----              ----------        ----------            ----  

     1 56    WS-C3850-48P       03.02.03.SE       cat3k_caa-universalk9 INSTALL

     2 56    WS-C3850-48P       03.02.03.SE       cat3k_caa-universalk9 INSTALL

This is the CPU & Memory utilization while image is copying via TFTP

copy tftp://x.x.13.2/cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin flash:


Loading cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin from x.x.13.2 (via Vlan999): !!!   !!!!!!

[OK - 223734376 bytes]

223734376 bytes copied in 1220.860 secs (183260 bytes/sec)

3850-1#sh processes cpu sorted    

Core 0: CPU utilization for five seconds: 22%; one minute: 25%;  five minutes: 18%

Core 1: CPU utilization for five seconds: 22%; one minute: 23%;  five minutes: 15%

Core 2: CPU utilization for five seconds: 16%; one minute: 16%;  five minutes: 10%

Core 3: CPU utilization for five seconds: 9%; one minute: 11%;  five minutes: 7%

PID    Runtime(ms) Invoked  uSecs  5Sec     1Min     5Min     TTY   Process

10636  1897970     13788712 138    9.11     10.03    6.57     0     iosd              

5575   776750      22015186 81     6.33     6.47     4.55     1088  fed               

6228   1592940     34820227 45     1.29     1.24     0.78     0     pdsd              

5577   1105820     30047779 36     0.19     0.11     0.11     0     stack-mgr         

5576   677960      58761469 11     0.14     0.12     0.10     0     platform_mgr      

6240   1684280     1759977  956    0.14     0.12     0.10     0     cpumemd      

3850-1#sh processes memory sorted

System memory  : 4004464K total, 1515164K used, 2489300K free, 150332K kernel reserved

Lowest(b)      : 1341049596

PID     Text         Data      Stack     Heap      RSS       Total     Process

10636   56944        35544     92        3468      194712    333240    iosd              

5575    15040        313580    92        10728     172176    639704    fed               

6068    4            94196     116       88484     95508     113168    idope.py          

5577    836          163552    88        4364      74840     371792    stack-mgr         

11639   4            188868    236       20888     73576     294588    wnweb_paster.py   

Here is the stats while file copying via FTP

3850-1#copy ftp: flash:

Accessing ftp://x.x.13.2/cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin...

Loading cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin !!!  !!!!!!!!!!

[OK - 223734376/4096 bytes]

223734376 bytes copied in 231.410 secs (966831 bytes/sec)

3850-1#sh processes cpu sorted     

Core 0: CPU utilization for five seconds: 51%; one minute: 45%;  five minutes: 31%

Core 1: CPU utilization for five seconds: 45%; one minute: 43%;  five minutes: 28%

Core 2: CPU utilization for five seconds: 43%; one minute: 38%;  five minutes: 24%

Core 3: CPU utilization for five seconds: 27%; one minute: 26%;  five minutes: 17%

PID    Runtime(ms) Invoked  uSecs  5Sec     1Min     5Min     TTY   Process

10636  2437970     13916799 140    23.22    20.87    13.43    0     iosd              

5575   1180460     22333651 82     17.54    15.15    9.70     1088  fed               

6228   1673940     36108966 46     0.60     0.83     0.68     0     pdsd              

5576   681010      59011248 11     0.14     0.13     0.11     0     platform_mgr   

Regards

Rasika

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Innovations in Cisco Full Stack Observability - A new webinar from Cisco