FMC Deployment got stuck at 5% for almost 3 hours

[Herald Sison]

Hi is there a safest way to remove this deployment on my FMC that got stuck for almost 3 hours already? i hope there is a safest way to do it without breaking any database or any network interruption since the FTD is being used by the whole company.

i have restarted the FMCv for 5x already but still it get stuck at 5% deployment and i even unplug the management cable to stop the deployment but still the same.

i am using FMC 7.0.1.1 and FTD 7.0.1.1 with ASA5508X

please help!

[balaji.bandi · Accepted solution]

if you have support raise an TAC case to easy for them undestand and safe fix.

if you like to try check below :

https://www.lammle.com/post/how-to-fix-a-stuck-cisco-firepower-ftd-deployment/

[Herald Sison]

Can we consider this a bug?

[balaji.bandi]

not sure , may be bug, suggested to open a TAC case.

[Leo Laohoo]

---

@Herald Sison wrote:

Can we consider this a bug?

---

Not going to lie, it is a "feature".  

I have heard of stories with the words "two days" were used.  

Have a look at THIS thread.  Similar issue.  

[Herald Sison]

i have contacted TAC but it seems they are busy attending other concerns and they are not that proactive in emails so i just researched for solutions online and i bumped in to this blog and tried it then it worked well for me.

> expert admin@fmc01:~$ sudo su -

> root@fmc01:~# OmniQuery.pl -db mdb -e "select status,category,hex(uuid) from notification where status=7;"

> root@fmc01:~# OmniQuery.pl -db mdb -e 'delete from notification where uuid=unhex("bb0bba970b4c4423927b8f7d237edd0b");'

> OmniQuery.pl -db mdb -e "select status,category,hex(uuid) from notification where status=7;"

> root@fmc01:~# /etc/rc.d/init.d/console restart

https://dependencyhell.net/2021/fmc-task-stuck-deleting-broken-tasks-from-database

[Ciscouserz]

This is the solution if there is no actual job started, deployment stuck at nomore than 5%, Progress <=5% . 

[jsolano.fusionet]

Is it possible to apply the same workaround or something similar to a FDM?

[Leo Laohoo]

Raise a TAC Case. 

I have heard some toe-curling stories about taking 2 days.

[ianwatts]

Came to confirm the OmniQuery script to delete the task works.  I received these results when running the delete:
    Error: statement contains no result
    Command returned no results.
Rerunning the select query then returned 0 rows (the former stuck deployment line was gone).

Just wanted to add the command response when deleting the affected task.. that the "error" is in fact not a problem..
My "new" deployment appears to be moving forward normally.

[Herald Sison]

this error also appeared on my end. I just disregarded the error but i made sure the stuck deployment was deleted so that i can deploy the pending changes on my FMC.

I am pretty sure you will get several stuck deployment like that in the future.

[akhon0001]

Hi all.
I have exactly the same problem, only after deleting the message, the deployment in the GUI continues, but when I re-check for the presence of the message, it is missing.

Tried to reboot both FMC and FTD. Nothing helps.

[Marvin Rhoads]

Clearing the message with OmniQuery does not clear actual the job that is in the background, only the FMC GUI notification of it.

Deleting the job requires a more advanced set of steps and should only be done with TAC guidance.

[shariri]

It's important to note that simply clearing the task will not likely resolve the issue. The FMC may encounter the same problem again. It's crucial to understand the deployment status. Although the deployment was successful, the GUI is displaying the wrong status or is completely unresponsive. It's important to identify the root cause of this deployment issue, and TAC should be able to recognize these reasons.