Cisco needs a better way to clear failed FTD (FDM/FMC/CDO) tasks from and FTD device once it's failed, will not succeed after multiple attempts, and won't "Clear All".
Scenario: Upgrading a set of 1010 HA pair devices fails. Scheduled weekly Threat, URL, etc. updates fail.
I recently tried remotely upgrading an HA Active/Failover pair via FDM. I've been successful with this automatic procedure in the past, so I was confident all steps were followed properly. However, in this instance, after the upgrade was started the Primary device eventually rebooted and never came back online. The Secondary unit was not active. After 12 hours passed, hoping to prove it was just a slower-than-expected upgrade, I had an office worker there manually swap the cables on the devices, so the Secondary device would become active. I had the worker also try rebooting the Primary device. No luck. It was a brick (or so I thought).
After two weeks of delays, the worker finally packed up and shipped the failed unit back to me for re-imaging, the only choice I was left with. it did appear to be a brick.
To tangent just a bit: When using open-source code, it's not enough to get it working. You have to try doubly hard to rule out possible failures of code, because open-source coders aren't developing just for Cisco, likely. Breaking a new code version is absolutely just as important as building it! It was my job in the past. I was branded with 'Murphree's Law of Oppessticism' (optimist/pessimist). Therein lies Rule #1, "If anybody can break it, I can!" That said, I realize the complexity of Cisco's engineering solutions. The problem is way more complicated than can be fully and holistically supported, a well-known industry risk.
After rebuilding the failed firewall, it was now on the desired 6.5.0-115 code version. However, the Secondary device still running in the office there was on 6.4 still. Apparently, it never was triggered to upgrade during the automatic upgrade procedure, but perhaps downloaded updates were passed on to it after the upgrade failed.
And here's the problem: Scheduled deployments configured on the Primary device were scheduled for both devices, but now, the updates downloaded to both devices apparently weren't for the same version of the FTD code. Hence, the automatic updates would fail on the Secondary unit stuck on old FTD code. Because of this, the Secondary FDM shows the deployment of these updates having failed, and there's simply no way to remove the staged updates from within FDM. TAC has looked at this already, in two cases I've provided. The only conclusion we came to was to perform a manual FTD upgrade on the Secondary unit to 6.5.0-115 and then deploy the scheduled updates. This was revealed looking at pigtail deploy. We even tried disabling or orphaning Snort in some ways to see if it would perhaps get past this problem and then move on.
Resolution: There seems to be no clear, easy way to manually clear a failed deployment like this. How did this happen? After some seriously careful concluding I have three guesses:
- It's not possible with the OS or tools used in the FTD image,
- Requires significant database querying and deleting, or,
- Hasn't been fully investigated.
What I'd like to see is an effort to fix this problem, since it renders the device non-configurable until resolved. The device continues to operate, but no changes or further updates can be made. This borders on Smartnet hardware failure. In the interest of fast-paced security needs, this one is a bit tricky.
I consider this a monumental issue in need of an urgent solution, and further training needs to be provided to TAC, since current repairs of this condition are quite expensive, risky, or potentially very time-consuming.
Perhaps a small seek and destroy script would suffice for the short-term, executable in the expert mode sudo. Thanks in advance!
RFC 1925
