FMC Health Alert not clearable (Interface Status (Modified))

[Legusol]

We just upgrade to FMC v7.0.1 and afterwards we seeing an issue with a Health Alert that we cannot clear. The following is showing for one of our FTDs. At the interfaces shown are disabled interfaces on the FTD that are not in use which they should be since we don't use them. We cannot figure out how to clear this alert. If you click the alert, it takes us to the Health Monitoring page where both the FMC and FTD are showing Green without and alerts.

Prior to raising a TAC case I figured I check here.

[Marvin Rhoads · Accepted solution]

Here is the fix to delete the messages. The following commands need to be run as root from the FMC cli (expert mode):

OmniQuery.pl -db mdb -e "select status,category,hex(uuid),body from notification where status=11;"
OmniQuery.pl -db mdb -e 'delete from notification where uuid=unhex("<insert the uuid that the first command returns here>");'
OmniQuery.pl -db mdb -e "select status,category,hex(uuid),body from notification where status=11;"

repeat line 2 as necessary if there is more than one entry to be removed. The output after line 2 should be "Command returned no results." That is normal. Line 3 (same as line 1) should return "0 rows in set" if you removed the messages successfully.

The following bugID applies: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwa14518
(internal only as of this writing since it is still under investigation with only a work around in place and not a proper fix)

[Legusol]

Thank you! That worked perfectly!

[tjmann]

Just worked for me. I had to remember to do a sudo SU first, which gives you root user access. Then run the commands.

Thank you

[mdelvecc]

Hi Marvin,

I am running image 7.0.2-88 on my 4600 FMC, and tried running the commands you listed above. For some reason, they just won't clear. I ran all lines several times, but still display in notification window. It never returns a "0 rows" after running the 3rd line. Any other suggestions?

Thanks,

Mark

[Marvin Rhoads]

@mdelvecc is your FMC in HA by chance? If so be sure to run the commands on the active FMC.

If it's stand alone, then you may have some other root cause and would need to open a TAC case for resolution.

[mdelvecc]

Yes, I am running H/A pair, but ran it on the active. I see the table populated with the warnings, after I run the commands. I ran the same commands on the standby FMC and it returned "0 rows", which I would expect.

[Priit Kosmopoliit]

The solution worked, but we are running FMC 7.1.0.1 (build 28) and in the bug tool it says:

Known Fixed Releases (2 of 2)

7.1.0

7.0.1

Which seems to be not true.

[longlong_liu]

Hi,

I also encountered this bug, what I want to know is: 

1: Is the interface displayed by the health alarm unavailable? (modified [eht1/13 , eth1/14 , eth1/15 , eth1/16] , even if the indicator lights are normal after I connect eth1/16, but there is no data interaction)

2: The following commands need to be run as root from the FMC cli (expert mode): How to enter root mode

3: After clearing the error message, whether the interface can be used normally.

Thank you very much!

[Marvin Rhoads]

1. I was told by TAC that the bug has to do with the newer software parsing the MTU setting in the interface configuration where it did not do so before.

2. From clish (> prompt) type "expert" (without quotes). that will switch to the Linux shell as user admin. Then type "sudo su -" (also without quotes) to change to root user. The Linux prompt will change to #, indicating you are root user.

3. Yes.

[sammer.wehaib]

Hi @Marvin Rhoads thanks for sharing a solution for this, I running those commands and I'm still seeing those warning, could you please have a look?

root@fmc:/Volume/home/admin# OmniQuery.pl -db mdb -e "select status,category,hex(uuid),body from notification where status=11;"
************ Applying dynamic update files ************
Dynamic update files directory: /usr/local/sf/etc/dynamic_db_updates
Applying file remove_ref_check_rna_ip_os_map.yaml.
Status: Success.
************ Applying dynamic update files finished ************
+--------+------------------------------------------------------+----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------+
| status | category | hex(uuid) | body |
+--------+------------------------------------------------------+----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------+
| 11 | health:category.qwerjyue-andv-7654-2345-g4678494gi94 | FF93C555564B3A4CA13B5C80A63DB8A2 | {"property":"task:category.99","literal":"Modified: [Ethernet1/5, Ethernet1/6, Ethernet1/7, Ethernet1/8]","arguments":{}} |
| 11 | health:category.qwerjyue-andv-7654-2345-g4678494gi94 | 5CDC6A8CB2D43D559E38A39B6A1D434F | {"arguments":{},"literal":"Modified: [Ethernet1/3, Ethernet1/4, Ethernet1/5, Ethernet1/6, Ethernet1/7, Ethernet1/8]","property":"task:category.99"} |
| 11 | health:category.qwerjyue-andv-7654-2345-g4678494gi94 | A5F93E573B113B2FA571DB9E52AACC7D | {"arguments":{},"literal":"Modified: [Ethernet1/3, Ethernet1/4, Ethernet1/5, Ethernet1/6, Ethernet1/7, Ethernet1/8]","property":"task:category.99"} |
+--------+------------------------------------------------------+----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------+
3 rows in set (0.000615 seconds)

root@fmc:/Volume/home/admin# OmniQuery.pl -db mdb -e 'delete from notification where uuid=unhex("<insert the uuid that the first command returns here>");'
************ Applying dynamic update files ************
Dynamic update files directory: /usr/local/sf/etc/dynamic_db_updates
Applying file remove_ref_check_rna_ip_os_map.yaml.
Status: Success.
************ Applying dynamic update files finished ************
Command returned no results.

root@fmc:/Volume/home/admin# OmniQuery.pl -db mdb -e 'delete from notification where uuid=unhex("<insert the uuid that the first command returns here>");'
************ Applying dynamic update files ************
Dynamic update files directory: /usr/local/sf/etc/dynamic_db_updates
Applying file remove_ref_check_rna_ip_os_map.yaml.
Status: Success.
************ Applying dynamic update files finished ************
Command returned no results.

root@fmc:/Volume/home/admin# OmniQuery.pl -db mdb -e "select status,category,hex(uuid),body from notification where status=11;"
************ Applying dynamic update files ************
Dynamic update files directory: /usr/local/sf/etc/dynamic_db_updates
Applying file remove_ref_check_rna_ip_os_map.yaml.
Status: Success.
************ Applying dynamic update files finished ************
+--------+------------------------------------------------------+----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------+
| status | category | hex(uuid) | body |
+--------+------------------------------------------------------+----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------+
| 11 | health:category.qwerjyue-andv-7654-2345-g4678494gi94 | FF93C555564B3A4CA13B5C80A63DB8A2 | {"property":"task:category.99","literal":"Modified: [Ethernet1/5, Ethernet1/6, Ethernet1/7, Ethernet1/8]","arguments":{}} |
| 11 | health:category.qwerjyue-andv-7654-2345-g4678494gi94 | 5CDC6A8CB2D43D559E38A39B6A1D434F | {"arguments":{},"literal":"Modified: [Ethernet1/3, Ethernet1/4, Ethernet1/5, Ethernet1/6, Ethernet1/7, Ethernet1/8]","property":"task:category.99"} |
| 11 | health:category.qwerjyue-andv-7654-2345-g4678494gi94 | A5F93E573B113B2FA571DB9E52AACC7D | {"arguments":{},"literal":"Modified: [Ethernet1/3, Ethernet1/4, Ethernet1/5, Ethernet1/6, Ethernet1/7, Ethernet1/8]","property":"task:category.99"} |
+--------+------------------------------------------------------+----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------+
3 rows in set (0.00063 seconds)

[Marvin Rhoads]

When you run:

OmniQuery.pl -db mdb -e 'delete from notification where uuid=unhex("<insert the uuid that the first command returns here>");'

...you must substitute your specific UUID(s) for the portion of that command within the quotes. For instance:

OmniQuery.pl -db mdb -e 'delete from notification where uuid=unhex("FF93C555564B3A4CA13B5C80A63DB8A2");'

...and so forth for each UUID that the query returned.

[sammer.wehaib]

Thanks Marvin, worked like a charm, appreciate your always support and help.

[ste.ant]

Worked with FMCv v6.7.0.3 as well! Thank you.

[Martin Ostberg]

Running a newly installed 7.0.1 here and I'm getting this as well.

Running 

OmniQuery.pl -db mdb -e "select status,category,hex(uuid),body from notification where status=11;" 

 Doesn't return any results though.

admin@DJ-FW01:~$ sudo OmniQuery.pl -db mdb -e "select status,category,hex(uuid),body from notification where status=11;"
Password:
+--------+----------+-----------+------+
| status | category | hex(uuid) | body |
+--------+----------+-----------+------+

0 rows in set (0.000796 seconds)

Edit: my bad, I was in FTD, not  FMC.

This worked!