09-29-2013 11:33 PM - edited 03-07-2019 03:44 PM
During a recent 3850 switch upgrade I noticed I got very slow data rate (~200 KB/sec & took around 24 min) when copying new image to 3850 Flash.
cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin (213MB)
Since it was from a Network TFTP server, I tried it again with local TFTP from a PC directly connected to 3850 and still got same low file transfer rates.
Did anyone else notice this ? What could be the cause for this slow file transfer rates ?
Rasika
Solved! Go to Solution.
03-29-2014 07:22 PM
Hello Rasika,
By default, the Catalyst 3850 uses a tftp block size value of 512, which is the lowest possible value. The reasoning behind this default value is to ensure interoperability with legacy tftp servers.
3850(config)#ip tftp blocksize ? <512-8192> blocksize value
For IOS-XE version 3.3.2 and below, you will have to manually change the block size in the global configuration to speed up the transfer process. The example below is a transfer comparison when using the default block size of 512K versus a transfer using the maximum block size value of 8192K.
3850#copy tftp flash: Address [172.16.108.26]? Source filename [IOS/3850/cat3k_caa-universalk9.SPA.03.03.02.SE.150-1.EZ2.bin Destination filename [cat3k_caa-universalk9.SPA.03.03.02.SE.150-1.EZ2.b in]? Accessing tftp://*****@172.16.108.26/cat3k_caa-universalk9.S PA.03.03.02.SE.150-1.EZ2.bin... Loading cat3k_caa-universalk9.SPA.03.03.02.SE.150-1.EZ2.bi n from 172.16.108.26 (via Vlan1): !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!! [OK - 257243236 bytes] 257243236 bytes copied in 1416.360 secs (181623 bytes/sec)
The results show that the 245mb .bin file was transferred in 1416.360 seconds, which equals almost 24 minutes. I will now adjust the block size value to the maximum allowed value of 8192.
3850#config t 3850(config)#ip tftp blocksize 8192 3850(config)#end
The same transfer is now run again.
3850#copy tftp flash: Address [172.16.108.26]? Source filename [IOS/3850/cat3k_caa-universalk9.SPA.03.03.02.SE.150-1.EZ2.bin Destination filename [cat3k_caa-universalk9.SPA.03.03.02.SE.150-1.EZ2.b in]? Accessing tftp://*****@172.16.108.26/cat3k_caa-universalk9.S PA.03.03.02.SE.150-1.EZ2.bin... Loading cat3k_caa-universalk9.SPA.03.03.02.SE.150-1.EZ2.bi n from 172.16.108.26 (via Vlan1): !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!! [OK - 257243236 bytes] 257243236 bytes copied in 124.100 secs (2072871 bytes/sec)
The results are clearly better. The same file was transferred in just over two minutes with the new block size value.
Hope that helps!!
Luke
09-29-2013 11:55 PM
What tftp server you are using ? have you checked duplex negotiation issues ?
09-30-2013 12:46 AM
Tried with multiple tftp servers like tftpd32 on windows 7, in builtin tftp on Macbook OSX & linux tftp server on network. I am sure ports are full duplex & 1G speeds
Also checked with SCP & speed was better (double the speed aroun 400KB/sec), still took around 10mins to copy the image.Still too slow from my perspective.
Rasika
09-30-2013 12:54 AM
Try sticking the IOS into a flash drive and compare the outcomes.
09-30-2013 01:04 AM
I will do this tomorrow & let you know Leo... already off for the day
09-30-2013 01:05 AM
No problem. Enjoy your day off.
09-30-2013 01:11 AM
Just curious, after uploading did it do CRC check ?
09-30-2013 04:59 PM
Hi Jonn,
Not too sure, no visible indication.
But the time observed is just copy the file to flash.
Regards
Rasika
09-30-2013 04:51 PM
Hi Leo,
Here is the result of copying from USB to 3850 Flash, file transfer completed within 60s & this is what I expected via TFTP as well.
3850-1#copy usbflash0:cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin flash:
Destination filename [cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin]?
Copy in progress...CCCCC.....CCCCC
223734376 bytes copied in 60.180 secs (3717753 bytes/sec)
Here is the result I am getting with TFTP via directly connected PC. It is more than 20 minutes
3850-1#copy tftp://10.15.5.253/cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin flash:
Destination filename [cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin]?
Accessing tftp://10.15.5.253/cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin...
Loading cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin from 10.15.5.253 (via Vlan999): !!!!!!!! !!!!!
[OK - 223734376 bytes]
223734376 bytes copied in 1212.560 secs (184514 bytes/sec)
There is no duplex/collision issues as I can see laptop directly connected & negotiated to Full duplex/1000Mbps.
3850-1#sh int g1/0/1
GigabitEthernet1/0/1 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 58bf.eab6.5681 (bia 58bf.eab6.5681)
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
Copy the same file to a 6500 flash & it took around 5 min, much faster than 3850/3750X
Why that much of drastic (>20 times) file transfer rate difference between locally attached device (USB) compare and Network server ? Cannot be writing speed to the 3850 flash, may be there is an efficient way of writing into disk when copy from locally attached USB ?
From the practical point of view copy an image from a network server would be the common use case & wonder if there is any way make this faster.
Thanks
Rasika
09-30-2013 05:17 PM
(184514 bytes/sec)
Hmmmm ... that's pretty slow for over the network.
Something is slowing things down.
Do you have anti-virus enabled from the TFTP server?
Let's do the basics first before you consider checking routing.
Is there drastic differences the way wrting to flash via TFTP
Compare the speed when you run HTTP.
09-30-2013 10:04 PM
Here is further more test result. MacBook Air is directly connected to switch g1/0/1 with no any other connectivity to this switch. Laptop IP 10.15.5.254/23 & Switch SVI-10.15.4.255/23
Copy the same file via TFTP & SCP (renamed the file to "telst.file" for simplicity)
3850-1#copy tftp://10.15.5.254/test.file flash:
Destination filename [test.file]?
Accessing tftp://10.15.5.254/test.file...
Loading test.file from 10.15.5.254 (via Vlan999): !!!!!! !!!!!
[OK - 223734376 bytes]
223734376 bytes copied in 1195.530 secs (187142 bytes/sec)
3850-1#copy scp://10.15.5.254/test.file flash:
Address or name of remote host [10.15.5.254]?
Source username [scp]?
Source filename [test.file]?
Destination filename [test.file]?
%Warning:There is a file already existing with this name
Do you want to over write? [confirm]
Password:
Sending file modes: C0777 223734376 test.file
!!!!!!!!!!!!!!! !!!!!!!!!
223734376 bytes copied in 544.870 secs (410620 bytes/sec)
Regards
Rasika
09-30-2013 10:13 PM
So directly TFTP is slow when network connected????
This smells like a bug, Rasika. I'd recommend you open a TAC Case.
09-30-2013 11:06 PM
Yes, I am directly TFTP, SCP.
Did the same with FTP & had much improved results, it took only 145s
3850-1#copy ftp: flash:
Address or name of remote host [10.15.5.254]?
Source filename [test.file]?
Destination filename [test.file]?
%Warning:There is a file already existing with this name
Do you want to over write? [confirm]
Accessing ftp://10.15.5.254/test.file...
Loading test.file !!!!!!!!! !!!!!!!!
[OK - 223734376/4096 bytes]
223734376 bytes copied in 144.150 secs (1552094 bytes/sec)
It's looks like issue is related to TFTP most, I will see if TAC can provide some reasonable justification for this.
Thanks for the help
Rasika
10-01-2013 02:58 AM
Hmmmmm ... You could potentially try this ...
1. Telnet into the switch. DO NOT use SSH (higher CPU and memory).
2. Run the TFTP copy again.
3. Telnet into the switch again (using a different session) and check the CPU and memory utilization.
I am interested to know if the slow copy is due to memory or CPU utilization.
By the way, what is the CURRENT IOS version?
Did you see the same response with the NEWER IOS?
10-01-2013 09:20 PM
Hi Leo,
Here is the outcome of the testing I did following your steps. It's looks like FTP is more CPU intensive, still it transfer file much faster than the TFTP. This result based on windown7 PC as TFTP/FTP server sitting on my network. (not directly connected). Still result is similar to yesterday.
Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
1 56 WS-C3850-48P 03.02.03.SE cat3k_caa-universalk9 INSTALL
2 56 WS-C3850-48P 03.02.03.SE cat3k_caa-universalk9 INSTALL
This is the CPU & Memory utilization while image is copying via TFTP
copy tftp://x.x.13.2/cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin flash:
Loading cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin from x.x.13.2 (via Vlan999): !!! !!!!!!
[OK - 223734376 bytes]
223734376 bytes copied in 1220.860 secs (183260 bytes/sec)
3850-1#sh processes cpu sorted
Core 0: CPU utilization for five seconds: 22%; one minute: 25%; five minutes: 18%
Core 1: CPU utilization for five seconds: 22%; one minute: 23%; five minutes: 15%
Core 2: CPU utilization for five seconds: 16%; one minute: 16%; five minutes: 10%
Core 3: CPU utilization for five seconds: 9%; one minute: 11%; five minutes: 7%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
10636 1897970 13788712 138 9.11 10.03 6.57 0 iosd
5575 776750 22015186 81 6.33 6.47 4.55 1088 fed
6228 1592940 34820227 45 1.29 1.24 0.78 0 pdsd
5577 1105820 30047779 36 0.19 0.11 0.11 0 stack-mgr
5576 677960 58761469 11 0.14 0.12 0.10 0 platform_mgr
6240 1684280 1759977 956 0.14 0.12 0.10 0 cpumemd
3850-1#sh processes memory sorted
System memory : 4004464K total, 1515164K used, 2489300K free, 150332K kernel reserved
Lowest(b) : 1341049596
PID Text Data Stack Heap RSS Total Process
10636 56944 35544 92 3468 194712 333240 iosd
5575 15040 313580 92 10728 172176 639704 fed
6068 4 94196 116 88484 95508 113168 idope.py
5577 836 163552 88 4364 74840 371792 stack-mgr
11639 4 188868 236 20888 73576 294588 wnweb_paster.py
Here is the stats while file copying via FTP
3850-1#copy ftp: flash:
Accessing ftp://x.x.13.2/cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin...
Loading cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin !!! !!!!!!!!!!
[OK - 223734376/4096 bytes]
223734376 bytes copied in 231.410 secs (966831 bytes/sec)
3850-1#sh processes cpu sorted
Core 0: CPU utilization for five seconds: 51%; one minute: 45%; five minutes: 31%
Core 1: CPU utilization for five seconds: 45%; one minute: 43%; five minutes: 28%
Core 2: CPU utilization for five seconds: 43%; one minute: 38%; five minutes: 24%
Core 3: CPU utilization for five seconds: 27%; one minute: 26%; five minutes: 17%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
10636 2437970 13916799 140 23.22 20.87 13.43 0 iosd
5575 1180460 22333651 82 17.54 15.15 9.70 1088 fed
6228 1673940 36108966 46 0.60 0.83 0.68 0 pdsd
5576 681010 59011248 11 0.14 0.13 0.11 0 platform_mgr
Regards
Rasika
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide