Issue with upgrading C695 appliance

[Siebe]

We are in the process of installing a new appliance in a datacenter. The appliance has been licensed and all firewall ports have been opened. We are getting updates from Anti-Spam, Anti-Virus and all other services.

At this moment we have an issue updating the AsyncOS version. When we click on Upgrade (after the healthcheck) we either receive a time-out error: see attachment or No route to Host.

On a rare occasion it did work an we were able to select the AsyncOS update and started with de DownloadInstall process. In the GUI/CLI we sometimes reach 22%  and get a Time-Out error or this error message pointing to:
http://updates.ironport.com/asyncos/phoebe-12-5-1-037/hints/default/1
When performing a Packet capture on a working appliance and this appliance we can see a difference in behaviour. The device shows in Network monitor. We receive an error from the Gateway IP: ICMP: Destination Unreachable Message, Host Unreachable 208.90.58.115 and 5 other IronPort.com update sites.

When using Telnet (CLI) we are able to contact the update sites from IronPort.com on port 80 and 443. But not all of the time. At first it does not work and 1 minute later the connection is available (using the same Networkinterface).

According to our networkteam it is an issue with the Appliance. But i am unable to find any errors.

Does someone know how we can troubleshoot this issue.

[Siebe]

Edited the Update Server list with both static websites: http://updates-static.ironport.com/ and http://downloads-static.ironport.com.

Created a new network capture. The list of IP-address which are unreachable by the Gateway are limited to 4 now.

The downloads starts every time. But fails on 20% with a time-out error.

[dmccabej]

Hello,

It sounds like you've already performed most of the troubleshooting steps that would be recommended. Given the intermittent failures, network errors, and items mentioned, this is more than likely an issue with the network where the impacted ESA is located. 

I would probably start with narrowing down the differences in network/location/ISPs/etc between the working and non-working ESAs. Perhaps there is some sort of bandwidth restriction in place? or proxy/firewall which could potentially be killing the connection after a certain amount of time?

It may be best to open up a Cisco TAC case and we can assist with digging deeper.

Thanks!

-Dennis M.

[Siebe]

Hello Dennis,

We checked our logging and we see a lot of TCP Retransmission from the appliance to the ironport.com website: see attachment. One of our network engineers did see that the networkinterface on Data1 and Data 2 we not always online. The interface was going up and down according to the logging. They think it has something to do with the Appliance. But there is not diagnostic/logging on the Appliance where i can find the status of the Networkinterfaces. The only thing i can check is the current status: CLI ->Etherconfig -> Media. But these are all on auto-detect and are up.

Regards,

Siebe

[dmccabej]

Hello,

TCP Retransmissions will usually mean that we're not seeing responses to certain packets being sent out. Though, I'm sure your network engineer is already aware of that. There are also multiple interfaces on the C695 as well as the possibility of fiber, so, those interfaces may or may not be important depending on your configuration and which interface the ESA is using to establish the connection. 

We do have an article that goes over some customer-facing methods for network troubleshooting, but it sounds like you may have exhausted them. https://www.cisco.com/c/en/us/support/docs/security/email-security-appliance/117797-technote-esa-00.html. 

My recommendation would be to open up a Cisco TAC case and we can help review from the back-end where we have access to additional troubleshooting commands. This would also allow us to take a closer look at your configuration and to make sure everything on the ESA itself looks good. 

Thanks!

-Dennis M.

[Siebe · Accepted solution]

It took  a long time. But we have convinced the Firewall team it was an issue with the Firewall. They performed a Firmware update and now everything is working.

[dmccabej]

Great to hear that you finally got it sorted. Thanks for letting us know how it was resolved!

Thanks

-Dennis M.