TFTPing IOS to cat3650 Flash - 10+ hours - !!OOO!OOOOO!!!!!OOOOO!!

[Dean Romanelli]

Hi All,

I kicked off a TFTP transfer last night before I went to bed. It has been running for about 10+ hours. I am trying to upgrade the IOS on a catalyst 3650 switch in Colombia, South America across a VPN WAN link from northeast USA. I came in this morning to check the progress and I have this:

Switch#copy tftp flash
Address or name of remote host [192.168.0.30]?
Source filename [cat3k_caa-universalk9.SPA.03.06.06.E.152-2.E6.bin]?
Destination filename [cat3k_caa-universalk9.SPA.03.06.06.E.152-2.E6.bin]?
Accessing tftp://192.168.0.30/cat3k_caa-universalk9.SPA.03.06.06.E.152-2.E6.bin...
Loading cat3k_caa-universalk9.SPA.03.06.06.E.152-2.E6.bin from 192.168.0.30 (via Vlan1): !!!!!!!!!!!!!!!OOOOOOOOOO!OOOOO
!!!OOOOOOOO!OO!OOO!OOOOOOOOOOOOOOOOOOOOOOO.O!OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO!OOOOOOOOOOOOOOOOOOOOOOOOOO
!!!!!!!!OOOOOOOOOOOOOOOOOOOOOO!OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO!OOOOOOOOOOOOOOOOOOOOOOOOOOOOOO!OOOOOOOOOOOOOOOOOOOOOOOOOO
OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO!OOOOOOOOOOOOOOOOOOOOOOOOOOO!OOOOOOOOOOOOOOOOOOOOOOOOO!OOOOOOOOOOOOOOOOOOOOOOO!O
OOOOOOOOOOO!OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO!OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO.O!OOOOOOOOOOOOOOOOOOOOOOOOO
OOOOOOOOOOOOOOOOOOOOOOOOOOO!OOOOOOOOO!OOOOOO!OOO!O!OOO!O!!!O!!O!!OOO!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!O!!!!!!O!!
!O!!!!!.O!OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO
O!O!O!!!!!!!O!O!!!O!OO!!O!.O!OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO
OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO
OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO
OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO
OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO!!!!O!!O!!!O!!!!!!!O!!!O!!!!!!!!!!!O!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!O!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO
OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO!OOOOO!O!!OO!O!!O!!O!!!!!!!

The file is 289MB (303,772,864 bytes).  dir and show flash suggest I have enough space to accommodate this image:

Switch#dir | inc bytes
1621966848 bytes total (1175330816 bytes free)

Switch#show flash | inc bytes
1092874240 bytes available (446701568 bytes used)

When I do a dir, it shows that over the past 10 hours, 184418304 bytes out of 303772864 bytes have completed. That is roughly 60% complete.  That is ridiculous for 10+ hours.

I've ran the image through HashCalc as well. The MD5 hash on Cisco's download site for this image is 7c2abd4f5856b80afbcc283f6f12797a and the hash of that file downloaded to my PC is 7c2abd4f5856b80afbcc283f6f12797a per hash calc, so it would appear this is not a corrupt file issue.

When I turn debugging on and run wireshark, I see data still flowing:

Switch#debug tftp packets
TFTP Packet debugging is on
Switch#debug tftp events
TFTP Event debugging is on
Switch#term mon
Switch#
*Jul  6 11:09:28.224: TFTP: Sending ACK for block 38065, socket_id 0x3BEDB6A0
*Jul  6 11:09:28.305: TFTP: Received block 38066, socket_id 0x3BEDB6A0
*Jul  6 11:09:28.305: TFTP: Sending ACK for block 38066, socket_id 0x3BEDB6A0
*Jul  6 11:09:28.387: TFTP: Received block 38067, socket_id 0x3BEDB6A0
*Jul  6 11:09:28.387: TFTP: Sending ACK for block 38067, socket_id 0x3BEDB6A0
*Jul  6 11:09:28.469: TFTP: Received block 38068, socket_id 0x3BEDB6A0
*Jul  6 11:09:28.469: TFTP: Sending ACK for block 38068, socket_id 0x3BEDB6A0
*Jul  6 11:09:28.551: TFTP: Received block 38069, socket_id 0x3BEDB6A0
*Jul  6 11:09:28.551: TFTP: Sending ACK for block 38069, socket_id 0x3BEDB6A0
*Jul  6 11:09:28.632: TFTP: Received block 38070, socket_id 0x3BEDB6A0
*Jul  6 11:09:28.632: TFTP: Sending ACK for block 38070, socket_id 0x3BEDB6A0
*Jul  6 11:09:28.714: TFTP: Received block 38071, socket_id 0x3BEDB6A0
*Jul  6 11:09:28.714: TFTP: Sending ACK for block 38071, socket_id 0x3BEDB6A0
*Jul  6 11:09:28.796: TFTP: Received block 38072, socket_id 0x3BEDB6A0
*Jul  6 11:09:28.796: TFTP: Sending ACK for block 38072, socket_id 0x3BEDB6A0
*Jul  6 11:09:28.878: TFTP: Received block 38073, socket_id 0x3BEDB6A0
*Jul  6 11:09:28.878: TFTP: Sending ACK for block 38073, socket_id 0x3BEDB6A0
*Jul  6 11:09:28.960: TFTP: Received block 38074, socket_id 0x3BEDB6A0
*Jul  6 11:09:28.960: TFTP: Sending ACK for block 38074, socket_id 0x3BEDB6A0
*Jul  6 11:09:29.043: TFTP: Received block 38075, socket_id 0x3BEDB6A0
*Jul  6 11:09:29.043: TFTP: Sending ACK for block 38075, socket_id 0x3BEDB6A0
*Jul  6 11:09:29.125: TFTP: Received block 38076, socket_id 0x3BEDB6A0
*Jul  6 11:09:29.125: TFTP: Sending ACK for block 38076, socket_id 0x3BEDB6A0
*Jul  6 11:09:29.207: TFTP: Received block 38077, socket_id 0x3BEDB6A0
*Jul  6 11:09:29.207: TFTP: Sending ACK for block 38077, socket_id 0x3BEDB6A0
*Jul  6 11:09:29.289: TFTP: Received block 38078, socket_id 0x3BEDB6A0
*Jul  6 11:09:29.289: TFTP: Sending ACK for block 38078, socket_id 0x3BEDB6A0
*Jul  6 11:09:29.371: TFTP: Received block 38079, socket_id 0x3BEDB6A0
*Jul  6 11:09:29.371: TFTP: Sending ACK for block 38079, socket_id 0x3BEDB6A0

I am using Solarwinds TFTP Server, which allegedly has a 4GB file size transfer limit, which is far greater than what I am trying to push.

Windows firewall on my PC pushing the image is off.

That leaves WAN loss as the only other culprit, but my pings tests are clean, with reasonable latency considering the distance:

FWCoreUSDC1-VPN5520# ping inside 192.168.142.15 repeat 1000
Type escape sequence to abort.
Sending 1000, 100-byte ICMP Echos to 192.168.142.15, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (1000/1000), round-trip min/avg/max = 80/81/110 ms

What am I missing here?

[Julio E. Moisa · Accepted solution]

Hi

My suggestion is upgrade the switch through wire, using remote PC, and no wireless or VPN either. It could affect the transferred packets.

It should not take 10hrs to upgrade unless you are using Xmodem. 

Steps to verify before to upgrade:

- Enough available space on the switch
- Disable window firewall
- Verify communication between both devices PC-Network device

!!!!!!!!!!!!!!! <-- good
OOOOOOOO <--- not good, it could represent any problem during the transfer.

[Dean Romanelli]

Hi guys,

Thanks for replying.  I am waiting for on the ground support in South America to get me onto a PC so I can try it locally.  In the meantime, I noticed something.  My firewall on site at the location where the switch is has "inspect tftp" on it. The transfer has to pass through this firewall.  Could this be the problem?

[Julio E. Moisa]

Hi Dean,

TFTP inspection is enabled by default, it will inspect RRQ, Write requests and Error notifications, to be honest I would not remove that, I prefer connect the remote PC to the switch and upgrade it. 

[Dean Romanelli]

Thanks guys.

Unfortunately it is proving difficult today to get field hands to get me onto a remote PC to do it locally, so for the time being I am trying another remote transfer, this time using tftpd64 with 1440 blocksize. So far no O's so that is an improvement, but still super slow.  If someone becomes available I will break it and redo locally.

[Julio E. Moisa]

Hi Dean,

Please keep us posted.

:-)

[Dean Romanelli]

We are 10% complete so far after 33 minutes of transfer.

The results are much better, but still not perfect. Though I don't see any drops, so in theory any O's should arrive by the time it hits 100% I hope. And again, if I get some field support, I will do it locally instead.

Switch#copy tftp: flash:
Address or name of remote host []? 192.168.0.30
Source filename []? cat3k_caa-universalk9.SPA.03.06.06.E.152-2.E6.bin
Destination filename [cat3k_caa-universalk9.SPA.03.06.06.E.152-2.E6.bin]?
Accessing tftp://192.168.0.30/cat3k_caa-universalk9.SPA.03.06.06.E.152-2.E6.bin...
Loading cat3k_caa-universalk9.SPA.03.06.06.E.152-2.E6.bin from 192.168.0.30 (via Vlan1): !!!!!!!!!!O!O!!!!!!!!!!!!O!O!!O!!!!!!!!!!!!!!OO!!O!!!O!!!

[lewislampkin]

In this case, some workarounds were recommended by others, trying to work within the constraints of how TFTP is designed:

From RFC 1350:

Each data packet contains one block of
   data, and must be acknowledged by an acknowledgment packet before the
   next packet can be sent.

Keep in mind that the round trip time between when the next ack is received will become your biggest limiting factor, making TFTP the type of protocol that will  scale "worse" as the round trip time increases, and it's no surprise it's taking so long to transfer the file from Northeast US to Colombia, South America.

The next time you have to do something like this across the WAN, you could try an alternate transfer protocol, such as FTP, for example, and you should get a faster result, as the implementation should allow more outstanding traffic before an ACK is required.

Hope this helps.

[Dean Romanelli]

Thank You Lewis.

So the up side of that is as long as I am getting !'s and O's and no periods, TCP will make sure all of those O's are addressed by the time the transfer is complete, and the exchange should still complete, right?

[Dean Romanelli]

Finally got some on the ground support and am currently running this from a PC on site with no O's.

Lessen learned --> no international TFTP'ing.

Thanks All.

[Julio E. Moisa]

Great! thank you for the update,

Have a great day 

:-)

[Leo Laohoo]

I am with Julio. 

Use Remote Desktop to find a PC that you can "store" the IOS into.  Then install TFTPd32/TFTPd64.  Point the switch to this PC to pull the IOS from.  

And don't forget to compare the MD5 hash value of this file against the MD5 hash value found in the Cisco website.  Do this after the IOS is copied into the switch and before rebooting or "installing".

[Karsten Iwen]

In addition to what Julio and Leo said: I also would increase the TFTP blocksize which will speed up the transfer:

switch(config)#ip tftp blocksize 1400

[Dean Romanelli]

Thanks Karsten. Will try.

No point in doing 8192 due to fragmentation right?

[Network713]

Used my desktop Win10, tftpd64. I deleted the current IOS on 3560, then did a tftp and got !OO!OO!OO!OO!OO!OO!OOO!OO!... [timed out].

I tried changing the tftpd64 settings, timeout 6 & retransmit 10 (like others suggested, it didn't help).

3560 doesn't have the command "ip tftp block-size xxxx"

The solution for me was to use another tftp server and program, I used my Solarwinds server with Solarwinds tftp and it was all good (but it took a long time). I know it is best to do it on a local pc, but I don't have that option, I'm doing the tftp over a WAN link.

-----------------------------------------------------------------------------------------

WS-C3560V2-48PS#delete /recursive /force flash:c3560-ipbasek9-mz.122-55.SE12.bin

WS-C3560V2-48PS#dir
Directory of flash:/

2 -rwx 796 Aug 30 2017 15:17:17 -05:00 vlan.dat
3 -rwx 5511 Sep 21 2020 17:32:19 -05:00 private-config.text
5 -rwx 4120 Sep 21 2020 17:32:19 -05:00 multiple-fs
6 drwx 512 May 1 1993 13:39:31 -05:00 crashinfo_ext
8 -rwx 64627 Sep 21 2020 17:32:18 -05:00 config.text

WS-C3560V2-48PS#copy tftp://10.10.10.247/c3560-ipbasek9-mz.150-2.SE11.bin flash:c3560-ipbasek9-mz.150-2.SE11.bin
Destination filename [c3560-ipbasek9-mz.150-2.SE11.bin]?
Accessing tftp://10.10.15.247/c3560-ipbasek9-mz.150-2.SE11.bin...
Loading c3560-ipbasek9-mz.150-2.SE11.bin from 10.10.15.247 (via Vlan10): !OO!OO!OO!OO!OO!OO!OOO!OO!... [timed out]

WS-C3560V2-48PS#copy tftp://10.10.10.100/c3560-ipbasek9-mz.150-2.SE11.bin flash:c3560-ipbasek9-mz.150-2.SE11.bin
Destination filename [c3560-ipbasek9-mz.150-2.SE11.bin]?
Accessing tftp://10.10.18.100/c3560-ipbasek9-mz.150-2.SE11.bin...
Loading c3560-ipbasek9-mz.150-2.SE11.bin from 10.10.18.100 (via Vlan10): !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!O!!!!!!!!!!
[OK - 16664448 bytes]
16664448 bytes copied in 1200.217 secs (13885 bytes/sec)

WS-C3560V2-48PS#verify /md5 flash:c3560-ipbasek9-mz.150-2.SE11.bin
.........................................................................................................................................................................Done!
verify /md5 (flash:c3560-ipbasek9-mz.150-2.SE11.bin) = b81d9bfd47be75d966d122af60003fef