TFTP timeout - tried many troubleshooting steps/Forum suggestions

[ITguy84]

Hello, I recently setup a new distro switch and VMware cluster (3 hosts).

I've got a Windows 2019 box running TFTPD64 and the other switches on my network can connect to it just fine and up load copies of configs. Perfect. 

This one switch (Catalyst 9300X) is giving me grief. I can see it connect to the server, it creates a file but it's 0 bytes, and empty. Then on the switch the command returns an error;

UMSW1(config)#do copy run tftp://10.xxxxxxxx
Address or name of remote host [10.xxxxxxxx]?
Destination filename [umsw1-confg]?
!.....
%Error writing tftp://xxxxxxxxxxx/umsw1-confg (Timed out).....

I can ping the TFTP server (it initially makes a connection that I can see in TFTPD64) but fails to write. 

I changed the default directory to something with no security, and even added the "everyone" group with full permissions to the security tab. 

Firewall is off for testing (other switches are able to write)

I tried to set the source interface on the switch, but it didn't help. There are technically two interfaces for the ESXi host, I tried both with no luck. I also tried the vlan interface for both the switch and the VM. (E.g.;

#ip tftp source-interface vlan 133

#ip tftp source-interface vlan 134

#ip tftp source-interface twe1/0/3

#ip tftp source-interface twe2/0/3

(I know it should be a port-channel it's on the docket, just too many other fires to fight first)

On the TFTP server I see this when I run the command to copy run tftp://10.xxxxxxxxx

The file writes to the DIR, but it's blank and 0 bytes. 

Feel like I've tried every tutorial and forum post out there to no avail. Help is greatly appreciated!

[marce1000]

  - Some suggestions : -  check the switch logs too when the tftp is tried
                                     - try another tftp server brand such as the one on a linux box

 M.

[Richard Burts]

As one step in investigating this issue I suggest on the switch running debug for tftp, attempt the transfer, and post the debug output.

[ITguy84]

Ok, so i ran a TFTP packet debug;

UMSW1#debug tftp packets
TFTP Packet debugging is on
UMSW1#copy run tftp://10.XXXXXXX
Address or name of remote host [10.XXXXXXX]?
Destination filename [umsw1-confg]?
!
*Apr 4 11:48:28.477: TFTP: Sending write request (retry 0), len 33, optlen 13, socket_id 0x7F2427A66C30 len 33
7F241EF84D30: 0002756D 7377312D 636F6E66 ..umsw1-conf
7F241EF84D40: 67006F63 74657400 626C6B73 697A6500 g.octet.blksize.
7F241EF84D50: 38313932 0000 8192..
*Apr 4 11:48:28.480: TFTP: Received OACK, socket_id 0x7F2427A66C30
*Apr 4 11:48:28.480: TFTP: Processing OACK, socket_id 0x7F2427A66C30
*Apr 4 11:48:28.480: TFTP: parsed option name blksize, value 8192
*Apr 4 11:48:28.480: TFTP: Sending block 1 (retry 0), len 8192, socket_id 0x7F2427A66C30.
*Apr 4 11:48:32.480: TFTP: Sending block 1 (retry 1), len 8192, socket_id 0x7F2427A66C30.
*Apr 4 11:48:36.478: TFTP: Received ACK for block 0, socket_id 0x7F2427A66C30
*Apr 4 11:48:37.481: TFTP: Sending block 1 (retry 2), len 8192, socket_id 0x7F2427A66C30.
*Apr 4 11:48:43.482: TFTP: Sending block 1 (retry 3), len 8192, socket_id 0x7F2427A66C30
*Apr 4 11:48:47.274: %SFF8472-3-THRESHOLD_VIOLATION: Twe2/1/7: Rx power high warning; Operating value: -1.4 dBm, Threshold value: -3.0 dBm..
*Apr 4 11:48:50.481: TFTP: Sending block 1 (retry 4), len 8192, socket_id 0x7F2427A66C30.
%Error writing tftp://10.133.134.29/umsw1-confg (Timed out)
*Apr 4 11:48:58.481: TFTP: Sending block 1 (retry 0), len 0, socket_id 0x7F2427A66C30.
*Apr 4 11:49:02.482: TFTP: Sending block 1 (retry 1), len 0, socket_id 0x7F2427A66C30.
*Apr 4 11:49:07.483: TFTP: Sending block 1 (retry 2), len 0, socket_id 0x7F2427A66C30.
*Apr 4 11:49:13.483: TFTP: Sending block 1 (retry 3), len 0, socket_id 0x7F2427A66C30.
*Apr 4 11:49:20.483: TFTP: Sending block 1 (retry 4), len 0, socket_id 0x7F2427A66C30.

Switch logs aren't very helpful;

*Apr 4 10:50:03.405: %SMART_LIC-3-COMM_FAILED: Communications failure with the Cisco Smart License Utility (CSLU) : Unable to resolve server hostname/domain name
*Apr 4 10:56:29.761: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: XXXXXX ] [Source: 10.133.4.51] [localport: 22] at 10:56:29 UTC Thu Apr 4 2024
*Apr 4 10:57:53.047: %SW_MATM-4-MACFLAP_NOTIF: Host dad3.a1d5.dc73 in vlan 76 is flapping between port Twe1/1/7 and port Twe1/0/15
*Apr 4 10:58:44.157: %SFF8472-3-THRESHOLD_VIOLATION: Twe2/1/7: Rx power high warning; Operating value: -1.4 dBm, Threshold value: -3.0 dBm.
*Apr 4 10:59:23.519: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.XXXXXX port 0 CLI Request Triggered
*Apr 4 10:59:24.518: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.XXXXXX port 514 started - CLI initiated
*Apr 4 11:01:21.502: %SYS-5-CONFIG_I: Configured from console by XXXXXX on vty0 (10.133.4.51)
*Apr 4 11:03:18.437: %SYS-5-CONFIG_I: Configured from console by XXXXXX on vty0 (10.133.4.51)
*Apr 4 11:08:44.766: %SFF8472-3-THRESHOLD_VIOLATION: Twe2/1/7: Rx power high warning; Operating value: -1.4 dBm, Threshold value: -3.0 dBm.
*Apr 4 11:15:08.426: %SYS-6-TTY_EXPIRE_TIMER: (exec timer expired, tty 1 (10.133.4.51)), user XXXXXX
*Apr 4 11:15:08.426: %SYS-6-LOGOUT: User XXXXXX has exited tty session 1(10.133.4.51)
*Apr 4 11:18:45.376: %SFF8472-3-THRESHOLD_VIOLATION: Twe2/1/7: Rx power high warning; Operating value: -1.4 dBm, Threshold value: -3.0 dBm.
*Apr 4 11:28:46.007: %SFF8472-3-THRESHOLD_VIOLATION: Twe2/1/7: Rx power high warning; Operating value: -1.4 dBm, Threshold value: -3.0 dBm.
*Apr 4 11:29:30.185: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: XXXXXX] [Source: 10.133.4.51] [localport: 22] at 11:29:30 UTC Thu Apr 4 2024
*Apr 4 11:34:20.534: %SYS-6-CLOCKUPDATE: System clock has been updated from 11:34:20 UTC Thu Apr 4 2024 to 07:34:20 Atlanti Thu Apr 4 2024, configured from console by XXXXXX  on vty0 (10.133.4.51).
*Apr 4 11:34:26.851: %SYS-6-CLOCKUPDATE: System clock has been updated from 07:34:26 Atlanti Thu Apr 4 2024 to 07:34:26 Atl Thu Apr 4 2024, configured from console by XXXXXX on vty0 (10.133.4.51).
*Apr 4 11:34:32.649: %SYS-5-CONFIG_I: Configured from console by XXXXXX on vty0 (10.133.4.51)
*Apr 4 11:37:02.318: %SYS-6-CLOCKUPDATE: System clock has been updated from 07:37:02 Atl Thu Apr 4 2024 to 08:37:02 Atl Thu Apr 4 2024, configured from console by XXXXXX on vty0 (10.XXXXXX).
*Apr 4 11:38:46.622: %SFF8472-3-THRESHOLD_VIOLATION: Twe2/1/7: Rx power high warning; Operating value: -1.4 dBm, Threshold value: -3.0 dBm.

Lines in bold reference the IP address of the TFTP server (also the syslog server)

I tried using SolarWinds, but it won't install properly. Spent about an hour playing with it trying various suggestions to get it to install, but the service will not start. 

[ITguy84 · Accepted solution]

Found the answer in another one of @Richard Burts forum posts; 

The trick was this;

"ip tftp blocksize 512"

[Richard Burts]

Thanks for the update. Glad to know that you have found a solution (and glad that one of my posts pointed you in the right direction.).