topic Re: FMC ghost device in Network Security

FMC ghost device

andre.ortega — Fri, 21 Feb 2020 16:14:18 GMT

Hello there,
I have in my lab a FMCv (6.2.3.4) and a ASA5506 running FTD software.
However FMC is showing that there is a deploy in an ASA5515X, that doesn't exist.

How can I remove that ghost deployment?

I have already seen this problem before in a customer, and in that case I opened a TAC, when the engeneer remove the deployment manually, but now, in my lab, I dont have access to TAC...

Thanks.

Re: FMC ghost device

Roy Harrington — Thu, 13 Sep 2018 17:14:10 GMT

I seen this happen in a case a few months ago.

This usually happens with an unsuccessful deployment which causes a roll back and typically FTD and FMC configurations are not in sync. I would first suggest checking to make sure there are no differences between the gui and the ftd.

To fix this you can deploy whats called flex configs which is basically asa cli pushed to the FTD in order to make changes. I would do whats called a deploy once flex config to remove what was not synced between the two.

You can also fix this by simply deploying the "ghost policy" it will then be removed after the successful deployment.

Re: FMC ghost device

andre.ortega — Thu, 13 Sep 2018 20:57:14 GMT

But there is no 5515 in my environment, how could I check the difference in config?

Re: FMC ghost device

Roy Harrington — Thu, 13 Sep 2018 21:01:53 GMT

Please attach the output from under Device>device Management where it show the devices and the names. Also please attach the screenshot of the ghost policy deployment.

Re: FMC ghost device

andre.ortega — Thu, 13 Sep 2018 21:06:23 GMT

Have a look.

Re: FMC ghost device

Roy Harrington — Thu, 13 Sep 2018 21:09:13 GMT

That's not a "ghost policy" that's a stuck policy push from what looks to be some 5k minutes or hrs ago i cant see it to clearly. Can you please let me know the time associated with it?

Re: FMC ghost device

Roy Harrington — Thu, 13 Sep 2018 21:12:01 GMT

Finally was able to view it. Thats 5k hrs its a pending policy push from over 200 days ago. The fix for this is quite complicated if your not familiar with linux. I would recommend you open up a case with Cisco TAC.

Re: FMC ghost device

andre.ortega — Thu, 13 Sep 2018 21:12:35 GMT

It shows 5378 hours Roy (you can click and see it in full screen).

But again, it shows ASA5515 and I have never had a 5515 associated to this FMC.

Re: FMC ghost device

andre.ortega — Thu, 13 Sep 2018 21:15:51 GMT

One more information: this "pending policy push" appeared 2 days ago...

And after this, I have already done others deployment, that worked, and this one still stuck.

Re: FMC ghost device

Roy Harrington — Thu, 13 Sep 2018 21:19:44 GMT

From fmc CLI please paste this in and let me know if any what the output is:

OmniQuery.pl -db mdb -e "select status,category,body from notification;"

Re: FMC ghost device

Roy Harrington — Thu, 13 Sep 2018 21:25:00 GMT

You can also try the following:

1. Connect to FMC  console and elevate to root.
root@FireSIGHT:~# sudo su -

2. Run the following command.
root@FireSIGHT:~# /etc/rc.d/init.d/console restart

Regarding the stuck notification, you can follow these steps to remove it.
1. Check in the notification table the entries with status=7.

OmniQuery.pl -db mdb -e "select status,category,hex(uuid) from notification where status=7;"
+--------+-------------------+----------------------------------+
| status | category          | hex(uuid)                        |
+--------+-------------------+----------------------------------+
| 7      | task:category.150 | 24EB1942AF4B3369B4134E3F345C03F7 |
| 7      | task:category.150 | 07EE0C1F9DF737698DDA0892FE202599 |
+--------+-------------------+----------------------------------+
2 rows in set

2. Delete those entries using the specified uuid.
OmniQuery.pl -db mdb -e 'delete from notification where uuid=unhex("24EB1942AF4B3369B4134E3F345C03F7");'
OmniQuery.pl -db mdb -e 'delete from notification where uuid=unhex("07EE0C1F9DF737698DDA0892FE202599");'

Re: FMC ghost device

andre.ortega — Thu, 13 Sep 2018 21:42:45 GMT

Thank you so much Roy.

Yes, there was 3 entry with status 7.
Following your instructions restarted FMC and I removed those entries and now that notification is gone.

Re: FMC ghost device

andre.ortega — Fri, 14 Sep 2018 13:17:08 GMT

Hi Roy, could you share what the status 7 means?
If you can, please share the others status as well.

Re: FMC ghost device

travwrig — Wed, 23 Jan 2019 17:02:11 GMT

See notification status with the below command

root@fmc-1:/var/tmp# OmniQuery.pl -db mdb -e "select * from notification_status;"
+--------+-----------+-------+
| status | label     | level |
+--------+-----------+-------+
| 1      | info      | 1     |
| 2      | success   | 1     |
| 3      | normal    | 1     |
| 4      | recovered | 1     |
| 5      | disabled | 1     |
| 6      | waiting   | 1     |
| 7      | running   | 1     |
| 8      | retrying | 1     |
| 9      | suspended | 1     |
| 10     | stopped   | 1     |
| 11     | warning   | 2     |
| 12     | critical | 3     |
| 13     | failure   | 3     |
| 14     | error     | 3     |
+--------+-----------+-------+

Re: FMC ghost device

travwrig — Wed, 23 Jan 2019 20:30:05 GMT

Also to delete tasks in status 7 with task category of 149 you can use this: OmniQuery.pl -db mdb -e 'delete from notification where notification.status=7 and notification.category="task:category.149";'

root@FTDHOSTNAME-removed:/Volume/home/admin# OmniQuery.pl -db mdb -e "select status,category,hex(uuid),body from notification where status=7;"
+--------+-------------------+----------------------------------+----------------------------------------------------------------------------------------+
| status | category | hex(uuid) | body |
+--------+-------------------+----------------------------------+----------------------------------------------------------------------------------------+
| 7 | task:category.149 | 100000500010059E0000004F00005A3C | {"literal":null,"arguments":{"DEVICE":"FTDHOSTNAME-removed"},"property":"task:ngfw_notified"} |

Re: FMC ghost device

Mashabane — Tue, 07 Jul 2020 03:43:43 GMT

Hello

How do I check the notification table, I illiterate when it comes to Linux , I am clueless, I managed to restart hoping the notification table will just appear but Nope.

here is where I am stuck please please help me

root@DPE-Firepower-manager:~# /etc/rc.d/init.d/console restart
Stopping Cisco Firepower Management Center for VMWare......ok
Starting Cisco Firepower Management Center for VMWare, please wait......started.
root@DPE-Firepower-manager:~#
root@DPE-Firepower-manager:~#
root@DPE-Firepower-manager:~#

@Roy Harrington wrote:

You can also try the following:

1. Connect to FMC  console and elevate to root.
root@FireSIGHT:~# sudo su -

2. Run the following command.
root@FireSIGHT:~# /etc/rc.d/init.d/console restart

Regarding the stuck notification, you can follow these steps to remove it.
1. Check in the notification table the entries with status=7.

OmniQuery.pl -db mdb -e "select status,category,hex(uuid) from notification where status=7;"
+--------+-------------------+----------------------------------+
| status | category          | hex(uuid)                        |
+--------+-------------------+----------------------------------+
| 7      | task:category.150 | 24EB1942AF4B3369B4134E3F345C03F7 |
| 7      | task:category.150 | 07EE0C1F9DF737698DDA0892FE202599 |
+--------+-------------------+----------------------------------+
2 rows in set

2. Delete those entries using the specified uuid.
OmniQuery.pl -db mdb -e 'delete from notification where uuid=unhex("24EB1942AF4B3369B4134E3F345C03F7");'
OmniQuery.pl -db mdb -e 'delete from notification where uuid=unhex("07EE0C1F9DF737698DDA0892FE202599");'

Check in the notification table the entries with status=7

Re: FMC ghost device

A.Foerby — Thu, 10 Sep 2020 12:08:06 GMT

Thank you for this Roy.

Used the same process to delete another stuck notification on FMC v.6.6.0.