Clear out failed deploy (VRT/VDB) - Surely there's a way

[bmurphree@myemma.com]

I'm often bitten by this issue:

Scenario: When FTD firewalls are set to automatically download and install Vulnerability DataBase (VDB) updates and Vulnerability Research Team (VRT) updates on the firewall, sometimes they will fail to deploy, especially in HA environments. You can't deploy any other changes to the device until that's sorted, but it's not easily sorted.

Is there a way to clear out these staged updates/changes to apply these updates so we can recover the ability to make changes to the firewall? While you can Clear All, the changes in FDM with no error, they're not cleared from staging.

This is really a debilitating issue when you're in the middle of an upgrade.

NOTE We don't use FMC.

Thanks in advance!

[Marvin Rhoads]

They need to be removed from the device's internal database which usually requires TAC assistance.

The issue of why they don't deploy should be resolvable. I've done a couple of FDM-managed HA deployments and haven't encountered that issue yet. What release are you running?

[bmurphree@myemma.com]

Hello. Multiple releases. 6.6.0-90, 6.6.5.1-15 (with the .2 hotfix), and 7.1.0-90, although I've not run into VDB and VRT hangups like this on 7.1.0-90 yet. Typically, they should be resolvable. Sometimes it takes 5-6 deploy attempts before it will actually work, but that's no guarantee. The issue is being looked at by TAC, but what's difficult to work with is the fact that any staged changes can be cleaned out/removed through FDM except vulnerability updates. It's like there's no intelligence built into that, so, a failed VDB or VRT deploy hangs up a change-management process and maintenance windows. This has proven to make automatic updates frail and difficult to trust.

Anyway, I'd love to know the process to query and remove the staged VRT/VDB updates (or the process to ensure success every time) so I can avoid TAC tickets and punted maintenance windows.

Thanks!

[bmurphree@myemma.com]

Whew! This has been a very painful process. Because there's still no way to clear a failing VRT/VDB staged deploy, we're having to go through some much more complicated and lengthy procedure to break HA and manually work on these devices individually. TAC didn't show up to the call, so we didn't have their help. There has to be a better way to do this.

[ida71]

Did you finally get this resolved ? I have a similar issue, where a global update introduced policy changes whilst VDB deploy was pending. Now can't deploy to one HA pair from FMC, TAC have been looking at it for over a month & tried a few things but nothing seems to work & FMC/FTD logs are pretty poor at revealing the reason for the failure. They didn't think this thing through very well from an admin perspective

[jeff.hawkes]

We are also experiencing this issue at multiple sites. It needs to be more stable. We have an urgent update to an IPSec tunnel. We can't update the tunnel as the vdb update is stuck/corrupt. I have tried uploading it manually, to the old or newer versions. This still fails. Now I have to contact TAC again and get them to clear it out. Bring back the stability of the ASA.

[Marvin Rhoads]

@jeff.hawkes what version of FMC/FTD are you running? The issues mentioned in this 2 year old thread are generally resolved in modern (7.2.5.1 or 7.4.1) releases.

[Ed Melendez]

Is this the same issue? Because it's still present.

We can't deploy any changes to the device.

[jeff.hawkes]

yes, that is exactly the same error message I have. Does anyone know if there is really a version that this does not happen to? I had heard that 7.4 has a rollback feature.

[jeff.hawkes]

Hi All,

Cisco TAC gave me a great workaround.

SSH into the firepower.

Then run 

through CLI go to the FMC :

>

>expert

# sudo su –

# touch /ngfw/var/cisco/deploy/pkg/var/cisco/packages/vdb-.tgz

# touch /ngfw/var/cisco/deploy/pkg/var/cisco/packages/vdb-379.tgz

Depending on the version of the file that is corrupt. Check the error message, mine was vdb-389.tgz

Then run the cloud update of the VDB again. Then it allows you to deploy changes.

[Shazz]

@jeff.hawkes Did you try the workaround?

[jeff.hawkes]

Yes, the work around worked. It appears to wipe the file. then it allows the cloud update to update it again. 

From memory, the touch command just empties a file. Not sure why that allows it to be overwritten, but it worked.

[Kamado]

Followed these steps while connected to the device over a VPN connection. After running the cloud update, I tried to deploy changes then lost connection to the VPN. Now I am getting a certificate error when I try to login to the VPN. The certificate error is preventing DUO authentication so I am not able to login to the VPN.

[Shazz]

The latest fix for this defect:

CSCwi92848.
The defect is not customer visible yet.
As Workaround, please perform the following steps:-
1. Correct the permissions of the file: /ngfw/var/sf/appid/odp/appid.conf:

1a. Login into the Device's CLI
1b. Go to expert mode with 'expert'
1c. The user must raise themselves to root access with 'sudo su'
1d. Use the same password used to login, if the user can't raise to root, try using the admin user/password.
1e. Fix the permissions with 'chmod 644 /ngfw/var/sf/appid/odp/appid.conf'

2. Reinstall VDB-379:

2a. Download from software.cisco.com<https://software.cisco.com>
2b. Manually upload and install from the GUI

3. Try to deploy once more, should be successful.