Deployment failed due to internal errors. If problem persists.....

[Herald Sison]

Hi All,

Anyone experienced this error upon deploying some changes on your FTD using FMC.

"Deployment failed due to internal errors. If problem persists after retrying, contact Cisco TAC." My initial fix is i just rebooted my FMC and so far i have not seen the error again. hopefully that would fix it permanently.

This is what i found about this bug but no resolutions were given.

https://bst.cisco.com/quickview/bug/CSCvz07712

I am using FMCv version 7.3.0 ang FTD 7.0.4 running snort 3 on ASA5508X device. after upgrading my FMC to 7.3.0 i experienced a lot of problems already and also after upgrading to snort 3.

my other concern also is every time i deploy any changes my PBR flexconfiguration got messed up and i need to remove it and deploy then add it again then deploy again to fix it. 

anyone here experienced this kind of problem?

[Marius Gunnerud]

I have seen a similar issue where deployment fails a couple times.  One time i tried deploying to more than 3 FTDs from FMCv at the same time, this failed.  I boiled this down to there not being allocated enough resources for the virtual machine.  The second time was because of a SRU mismatch between the SRU installed on the FTD and the one installed on FMC.  Had to manually update the SRU on the FTD to solve this issue.

With regard to the PBR flexconfig, I am assuming you have it set to Deployment: Once?  Change this to Deployment: Everytime.  Also I would suggest changing Type to be Append and not Prepend...if it is not already set to Append.

To solve the deployment you can either try to trick the FMC into thinking the deployment succeeded or delete the failed deployment and try again and hopefully it will be successful.

in expert mode

sudo su -
OmniQuery.pl -db sdb -e "select device_id, device_name,device_status,job_id from jobs_device_status order by device_status_time;"
<snip>
| 292057826315 | 93162b1c-2328-11ea-af62-df076f2fb9bc | FAILED | 433792192049 |
<snip>

OmniQuery.pl -db sdb -e "update jobs_device_status set device_status='SUCCEEDED' where job_id='433792192049';"

OmniQuery.pl -db sdb -e "select device_id, device_name,device_status,job_id from jobs_device_status order by device_status_time;"
<snip>
| 292057826315 | 93162b1c-2328-11ea-af62-df076f2fb9bc | SUCCEEDED | 433792192049 |
<snip>

or
(status 7 indicates a hung deployment)
OmniQuery.pl -db mdb -e "select status,category,hex(uuid),body from notification where status=7;
<snip>
| 7 | deployment:category | FB80DD80748E3EE99D8BD1C4D5223ABC | {"arguments":{"JOB_ID":"433792192049","PHASE":"Phase-0","DEVICE_ID":"292057826315"},"literal":null,"property":"deployment:deployment_initiated_for_the_device"} |
<snip>

OmniQuery.pl -db mdb -e 'delete from notification where uuid=unhex("FB80DD80748E3EE99D8BD1C4D5223ABC");'

[Herald Sison]

"I boiled this down to there not being allocated enough resources for the virtual machine.  The second time was because of a SRU mismatch between the SRU installed on the FTD and the one installed on FMC.  Had to manually update the SRU on the FTD to solve this issue."

--- i have checked everything and seems resources are memory @30%, disk @10%

For SRU i think it has a mismatch but i am wondering why it has a mismatch, i even downloaded the latest SRU from the support site and deploy it to the FTD but the version in FTD is still outdated. do you have a tutorial or notes on how to manually update the SRU in FTD side?

So far  the deployment error has gone and did not show up again after the FMC reboot. but i need to fix this SRU mismatch between FTD and FMC. do you have a notes for this sir? Thank you in advance.

"With regard to the PBR flexconfig, I am assuming you have it set to Deployment: Once?  Change this to Deployment: Everytime.  Also I would suggest changing Type to be Append and not Prepend...if it is not already set to Append."

-- how to do (append/not prepend) this on the FMC side?

This is my flexconfig object.

i also noticed that "policy-route route-map" cli command is not supported anymore, maybe because of the FMC version or the Snort 3 version?

Thank Sir

[Marius Gunnerud · Accepted solution]

You do not have an SRU mismatch...if you did deploy would not be working at all even after reload. look at your LSP version and VDB version are the same so no issue here.

You PBR configuration you need to change the Deployment: Once to Deployment: Everytime. 

The error you are seeing is because PBR is has now been implemented into the newer releases of FMC so it can be configured through Devices > Device Management > Routing > Policy Based Routing

I see I wasn't clear in my previous post.  The issue we had with regard to resources was that the FMC was configured as Thin Provisioned in VMware and not Thick Provisioned.  So at times other resources would pull more resources and then when the FMC needed to also pull on more resources there was not enough available at that time so the deployment failed.  This is despite the FMC being allocated the correct amount of resources, so it boils down to how the virtual machine host allocates resources when Thin Provisioning is used.

[Herald Sison]

Hi Sir, i wonder why i do not have PBR in my routing under devices. I even watched it in youtube that they have PBR in their devices -> routing using 7.2.x FMC version. Mine is 7.3.0. is it ok to go to 7.2.X? I am from 7.0.4 and jumped to 7.3.0 so technically i skipped 7.2.X. Is going down to lower version is the same process as going up? Like uploading the tar file and install it in FMC the usual way?

 

[Marius Gunnerud]

it is basically the same to down grade, but you need to make sure that you downgrade the FTDs first as the FMC will not be able to manage FTDs with a higher version.

Are you sure that you are running 7.3?  It should be present. If you are running 7.3 then this might be a bug.  Have you tried reloading the FMC?

https://www.cisco.com/c/en/us/td/docs/security/secure-firewall/management-center/device-config/730/management-center-device-config-73/routing-policy-based.html#Cisco_Task.dita_27e089d1-6340-40e1-9abd-df12be8378c4

[Herald Sison]

Hi Sir, my FMC is 7.3.0 but my FTD is still 7.0.4. Would that be safe to downgrade my FMC to 7.2.1?

Regarding PBR maybe the PBR in devices-> routing is missing because i am running 7.0.4 FTD and 7.3.0 FMC?

 

 

[Marius Gunnerud]

PBR is supported from 7.1 so it could be that your FTD is on 7.0 that is the problem why you are not seeing it in the GUI

https://www.cisco.com/c/en/us/td/docs/security/firepower/710/relnotes/firepower-release-notes-710/features.html

[Herald Sison]

Hi Sir, the latest FTD upgrade i can get from the support site is 7.0.5 for ASA5508X. I am still waiting for a gold star suggested release so i can upgrade my FTD. Is there a schedule of what month would they release goldstar releases? 
thanks sir

[Marius Gunnerud]

The ASA5508X running FTD software only supports up to 7.0.x. So you will not be able to upgrade past this.

https://www.cisco.com/c/en/us/td/docs/security/secure-firewall/compatibility/threat-defense-compatibility.html

[Herald Sison]

would that mean that they will now stop producing FTD upgrades for ASA5508x? Is it okay to go higher as much as possible in FMC versions and just stick with the last FTD upgrade 7.0.5 forever?

[Marius Gunnerud]

According to the FTD compatibility document, 7.0.x is the highest FTD software version that is / will be supported for the ASA5508.  So yes, if you need PBR on the FTD software you need to upgrade your hardware.

https://www.cisco.com/c/en/us/td/docs/security/secure-firewall/compatibility/threat-defense-compatibility.html

[Marius Gunnerud]

If you are interested in a write-up on doing the SRU rollback on the FMC (we did the FMC rollback as this would save time instead of rolling back 16 other devices), you can find the write-up here.