DNAC swim update via https failed

[dijix1990]

Recently I started to get error when I try to distribute firmware to my switches

Task Status
Failure (NCSW32001: Distribution failed using protocol: HTTPS. 
Accessing https://172.18.55.55/api/v1/file/temporary/8447249a-e518-4069-80c9-a7f21c94872f...
%Error opening https://172.18.55.55/api/v1/file/temporary/8447249a-e518-4069-80c9-a7f21c94872f (I/O error)
)

I tried to reinstal certificate and check telnet connection it's always good

telnet 172.18.55.55 443
Trying 172.18.55.55, 443 ... Open
^CHTTP/1.1 400 Bad Request
Date: Sat, 17 May 2025 10:44:30 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Connection: close
ETag: "67a53540-0"
Server: kong/0.14.1
Cache-Control: no-store
Pragma: no-cache
Content-Security-Policy: default-src 'self' 'unsafe-inline' 'unsafe-eval' blob: data:; frame-ancestors 'self'
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Frame-Options: SAMEORIGIN

apparently, some kind of problem is that the switch does not always work on the first attempt to connect to DNA to download the firmware over HTTPS, while there are no losses, there is connectivity

01 - FAIL sw-24_3#copy https://172.18.55.55/core/img/cisco-bridge.png flash:
Destination filename [cisco-bridge.png]?
%Warning:There is a file already existing with this name
Do you want to over write? [confirm]
Accessing https://172.18.55.55/core/img/cisco-bridge.png...
%Error opening https://172.18.55.55/core/img/cisco-bridge.png (I/O error)

02 - FAIL sw-24_3#copy https://172.18.55.55/core/img/cisco-bridge.png flash:
Destination filename [cisco-bridge.png]?
%Warning:There is a file already existing with this name
Do you want to over write? [confirm]
Accessing https://172.18.55.55/core/img/cisco-bridge.png...
%Error opening https://172.18.55.55/core/img/cisco-bridge.png (I/O error)

03 - SUCCESS sw-24_3#copy https://172.18.55.55/core/img/cisco-bridge.png flash:
Destination filename [cisco-bridge.png]?
%Warning:There is a file already existing with this name
Do you want to over write? [confirm]
Accessing https://172.18.55.55/core/img/cisco-bridge.png...
Loading https://172.18.55.55/core/img/cisco-bridge.png
4144 bytes copied in 0.116 secs (35724 bytes/sec)

CiscoCatalyst Center Version 2.3.7.9-70301

[Andrii Oliinyk]

It looks like you're encountering intermittent HTTPS download failures from your DNA Center (at IP address 172.18.55.55) to your Cisco switches during firmware distribution. The error message "Error opening https://..." with "(I/O error)" suggests a problem with the data transfer itself, even though basic connectivity (like the telnet to port 443) seems to be working.

Here's a breakdown of potential causes and troubleshooting steps you can take, keeping in mind the intermittent nature of the issue:

Potential Causes:

1. Intermittent Network Issues: While basic connectivity might be fine, there could be subtle, transient network problems between the switch and DNA Center that affect larger data transfers over HTTPS. This could include:

   • Packet loss or corruption: Even small amounts of loss can disrupt HTTPS transfers.
   • MTU (Maximum Transmission Unit) mismatch: If the MTU is not consistent along the path, fragmentation and reassembly issues could occur, leading to errors.
   • Bandwidth saturation or congestion: Temporary spikes in network traffic could be affecting the transfer.
   • Issues with intermediary devices: Firewalls, load balancers (if present), or other network devices in the path could be intermittently causing problems.

2. DNA Center Issues: Although less likely given the successful third attempt in your example, there could be intermittent issues on the DNA Center side:

   • Resource contention: DNA Center might be temporarily overloaded, leading to slow or failed file serving.
   • Temporary service instability: Some internal services within DNA Center responsible for file serving might be experiencing transient issues.

3. Switch Issues: Problems on the switch itself could also be a factor:

   • Limited resources (CPU, memory): During the download, the switch might be under heavy load, leading to I/O errors.
   • Buffer overflows or other temporary software glitches: Transient software issues on the switch could be affecting HTTPS downloads.

4. HTTPS Negotiation Issues (Less Likely After Certificate Reinstallation): While you reinstalled certificates, there could still be subtle issues with the TLS/SSL handshake or cipher suite negotiation that manifest intermittently.

Troubleshooting Steps:

1. Further Network Investigation:

   • Ping with larger packet sizes and "do not fragment" bit: From the switch to the DNA Center IP, try pinging with larger packet sizes (e.g., ping 172.18.55.55 size 1472 df-bit) to check for potential MTU issues.
   • Path MTU Discovery (PMTUd): Ensure PMTUd is working correctly on both the switch and the network path to DNA Center.
   • Monitor network statistics: Use tools like SNMP monitoring on the switch and along the network path to look for packet loss, errors, or congestion during the times when the firmware distribution fails.
   • Traceroute during failures: If possible, run a traceroute from the switch to DNA Center when a failure occurs to see if the path changes or if there are any unusual hops.

2. DNA Center Health Check:

   • Monitor DNA Center performance: Check the CPU utilization, memory usage, and disk I/O on your DNA Center appliance.
   • Review DNA Center logs: Examine the DNA Center logs for any errors related to file serving or HTTPS.

3. Switch Resource Monitoring:

   • Monitor switch CPU and memory: Use commands like show processes cpu sorted and show memory summary on the switch during firmware distribution to see if resource utilization is high.
   • Check interface statistics: Look for errors, drops, or high utilization on the switch's interfaces connected to the network.

4. HTTPS Specific Checks:

   • Verify DNA Center HTTPS configuration: Double-check the HTTPS configuration on DNA Center, ensuring the correct certificates are in place and the service is running correctly.
   • Test HTTPS from other devices: Try accessing other HTTPS resources from the affected switches to see if the issue is isolated to communication with DNA Center.

5. Consider Using a Different Protocol (If Feasible for Initial Troubleshooting):

   • While HTTPS is the recommended and secure method, temporarily try using TFTP or SCP (if configured and allowed in your environment) for firmware distribution to see if the issue persists with a different protocol. This could help isolate whether the problem is specifically with HTTPS.

6. Increase Switch HTTP Client Timeout (Potentially a Workaround):

   • You could try increasing the HTTP client timeout on the switch, although this might just delay the failure if the underlying issue is network-related. The command might look something like ip http client timeout <seconds> (check the specific IOS version syntax).

7. Software Version Considerations:

   • Ensure both your DNA Center version (2.3.7.9-70301) and the target switch IOS version are compatible according to Cisco's documentation. Sometimes, issues can arise due to software incompatibilities.

Based on your observation that the connection works on the third attempt, it strongly suggests an intermittent issue rather than a persistent configuration problem. Focus your troubleshooting on identifying these transient network or resource-related bottlenecks.

Start by closely monitoring the network path and the resource utilization on both the switch and DNA Center during firmware distribution attempts. The intermittent success indicates that the fundamental connectivity and configuration are likely correct, but something is causing temporary failures in the data transfer process.

packet capturing would be good approach as 1st step

[dijix1990]

Thanks for your troubleshooting guide! I noticed that it started after changing system certificate on my dna center. After replacing my certificate I updated telemetry setting via dna for my devices. 

I noticed that problems only with switches (9200L/9300L/9500/9600), for routers everything is good. I checked channels (we use zabbix for snmp and icmp test) and channels without problems, and firewalls between dna and devices...

I will try to increase timeouts 

[Andrii Oliinyk]

is the DNAC new system certificate issued by the same CA&subCA as it was before?

[dijix1990]

Yes. BTW I encreased timeout connection to 60 seconds and attempts to 5. I haven't ended my test but after it some of my switches started to get firmware.

[Andrii Oliinyk]

sounds like session timers issue since this point. is DNAC behind FW & maybe anything changed on it? 

[dijix1990]

yeah dnac behind fpr4225, but there wasn't anything changes about half year