FMC ghost device

[andre.ortega]

Hello there,
I have in my lab a FMCv (6.2.3.4) and a ASA5506 running FTD software.
However FMC is showing that there is a deploy in an ASA5515X, that doesn't exist.

How can I remove that ghost deployment?

I have already seen this problem before in a customer, and in that case I opened a TAC, when the engeneer remove the deployment manually, but now, in my lab, I dont have access to TAC...

Thanks.

[Roy Harrington]

I seen this happen in a case a few months ago.

This usually happens with an unsuccessful deployment which causes a roll back and typically FTD and FMC configurations are not in sync. I would first suggest checking to make sure there are no differences between the gui and the ftd.

To fix this you can deploy whats called flex configs which is basically asa cli pushed to the FTD in order to make changes. I would do whats called a deploy once flex config to remove what was not synced between the two.

You can also fix this by simply deploying the "ghost policy" it will then be removed after the successful deployment.

[andre.ortega]

But there is no 5515 in my environment, how could I check the difference in config?

[Roy Harrington]

Please attach the output from under Device>device Management where it show the devices and the names. Also please attach the screenshot of the ghost policy deployment.

[andre.ortega]

Have a look.

[Roy Harrington]

That's not a "ghost policy" that's a stuck policy push from what looks to be some 5k minutes or hrs ago i cant see it to clearly. Can you please let me know the time associated with it?

[andre.ortega]

It shows 5378 hours Roy (you can click and see it in full screen).

But again, it shows ASA5515 and I have never had a 5515 associated to this FMC.

[Roy Harrington]

Finally was able to view it. Thats 5k hrs its a pending policy push from over 200 days ago. The fix for this is quite complicated if your not familiar with linux. I would recommend you open up a case with Cisco TAC.

[andre.ortega]

One more information: this "pending policy push" appeared 2 days ago...

And after this, I have already done others deployment, that worked, and this one still stuck.

[Roy Harrington]

From fmc CLI please paste this in and let me know if any what the output is:

OmniQuery.pl -db mdb -e "select status,category,body from notification;"

[Roy Harrington · Accepted solution]

You can also try the following:

1. Connect to FMC  console and elevate to root.
root@FireSIGHT:~# sudo su -

2. Run the following command.
root@FireSIGHT:~# /etc/rc.d/init.d/console restart

Regarding the stuck notification, you can follow these steps to remove it.
1. Check in the notification table the entries with status=7.

OmniQuery.pl -db mdb -e "select status,category,hex(uuid) from notification where status=7;"
+--------+-------------------+----------------------------------+
| status | category          | hex(uuid)                        |
+--------+-------------------+----------------------------------+
| 7      | task:category.150 | 24EB1942AF4B3369B4134E3F345C03F7 |
| 7      | task:category.150 | 07EE0C1F9DF737698DDA0892FE202599 |
+--------+-------------------+----------------------------------+
2 rows in set

2. Delete those entries using the specified uuid.
OmniQuery.pl -db mdb -e 'delete from notification where uuid=unhex("24EB1942AF4B3369B4134E3F345C03F7");'
OmniQuery.pl -db mdb -e 'delete from notification where uuid=unhex("07EE0C1F9DF737698DDA0892FE202599");'

[andre.ortega]

Thank you so much Roy.

Yes, there was 3 entry with status 7.
Following your instructions restarted FMC and I removed those entries and now that notification is gone.

[andre.ortega]

Hi Roy, could you share what the status 7 means?
If you can, please share the others status as well.

[travwrig]

See notification status with the below command

root@fmc-1:/var/tmp# OmniQuery.pl -db mdb -e "select * from notification_status;"
+--------+-----------+-------+
| status | label     | level |
+--------+-----------+-------+
| 1      | info      | 1     |
| 2      | success   | 1     |
| 3      | normal    | 1     |
| 4      | recovered | 1     |
| 5      | disabled  | 1     |
| 6      | waiting   | 1     |
| 7      | running   | 1     |
| 8      | retrying  | 1     |
| 9      | suspended | 1     |
| 10     | stopped   | 1     |
| 11     | warning   | 2     |
| 12     | critical  | 3     |
| 13     | failure   | 3     |
| 14     | error     | 3     |
+--------+-----------+-------+

[travwrig]

Also to delete tasks in status 7 with task category of 149 you can use this: OmniQuery.pl -db mdb -e 'delete from notification where notification.status=7 and notification.category="task:category.149";'

root@FTDHOSTNAME-removed:/Volume/home/admin# OmniQuery.pl -db mdb -e "select status,category,hex(uuid),body from notification where status=7;"
+--------+-------------------+----------------------------------+----------------------------------------------------------------------------------------+
| status | category | hex(uuid) | body |
+--------+-------------------+----------------------------------+----------------------------------------------------------------------------------------+
| 7 | task:category.149 | 100000500010059E0000004F00005A3C | {"literal":null,"arguments":{"DEVICE":"FTDHOSTNAME-removed"},"property":"task:ngfw_notified"} |