"Oh snap!" - FMC Health Monitor features gone.

[HQuest]

Anyone else having trouble to access the health data of FMC and/or sensors on version 10?

Also, yes, that is a 100% genuine image. I would expect this from an open-source code, not Cisco having a sense of humor.

[Marvin Rhoads]

That's actually more commonly an error from the browser itself being unable to render a page, often due to running out of memory on the local computer. Is it persisting for you when accessing from another computer or a different browser?

I am using FMC 10.0.1 across more than a dozen production deployments and have never encountered this error. 

[HQuest · Accepted solution]

I can confirm memory, different browsers or different computers make no difference. While this is a lab environment, health items were working on 7.6 (tested up to 7.6.4) before upgraded to 10.0.0 and never worked since then.

Given Firepower solution is (mostly) built on top of standard Linux solutions, I was able to fix it by clearing the Prometheus data, used for the device monitoring component. Somehow there was leftover data from well over a year ago, which was causing the component to fail to create new files and not starting properly, even while the system has a nearly limitless FD limit.

ts=2026-06-07T00:02:28.262Z caller=head.go:532 level=error component=tsdb msg="Failed to calculate size of \"chunks_head\" dir" err="open /var/opt/prometheus/data/chunks_head: too many open files"
ts=2026-06-07T00:02:28.262Z caller=wlog.go:245 level=error component=tsdb msg="Failed to calculate size of \"wal\" dir" err="open /var/opt/prometheus/data/wal: too many open files"
ts=2026-06-07T00:02:40.838Z caller=compact.go:562 level=error component=tsdb msg="removed tmp folder after failed compaction" err="open /var/opt/prometheus/data: too many open files"
ts=2026-06-07T00:02:40.838Z caller=db.go:982 level=error component=tsdb msg="compaction failed" err="compact head: persist head block: open chunk writer: open /var/opt/prometheus/data/01KTFP7226HFSNZ4PVYWV1K85Z.tmp-for-creation/chunks: too many open files"
ts=2026-06-07T00:03:28.261Z caller=head.go:532 level=error component=tsdb msg="Failed to calculate size of \"chunks_head\" dir" err="open /var/opt/prometheus/data/chunks_head: too many open files"
ts=2026-06-07T00:03:28.261Z caller=wlog.go:245 level=error component=tsdb msg="Failed to calculate size of \"wal\" dir" err="open /var/opt/prometheus/data/wal: too many open files"
ts=2026-06-07T00:03:40.839Z caller=compact.go:562 level=error component=tsdb msg="removed tmp folder after failed compaction" err="open /var/opt/prometheus/data: too many open files"
ts=2026-06-07T00:03:40.839Z caller=db.go:982 level=error component=tsdb msg="compaction failed" err="compact head: persist head block: open chunk writer: open /var/opt/prometheus/data/01KTFP8WN74E9HK5NZTVGK3H3W.tmp-for-creation/chunks: too many open files"

It doesn't help the Prometheus version Cisco packs on FMC is from 2023. Outdated, with published CVEs including DoS and credentials leaking, and without options found on modern versions such as compact and delete functions for its tsdb management, a bit of fiddling with the filesystem and processes was required.

root@fmc:/var/opt/prometheus# ./prometheus --version
prometheus, version 2.45.1 (branch: HEAD, revision: c62de5d1075d9c24a281b4b719f124f14e730fa5)
build user: root@827ead67e852
build date: 20230929-21:10:51
go version: go1.20.8
platform: linux/amd64
tags: netgo,builtinassets,stringlabels

I can confirm my graphs are all back - time to repopulate them.

[Marvin Rhoads]

You were likely hitting this bug:
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwo49337
While Cisco reports it is fixed in 10.0.0. I have had two customers hit it on FMC 10.0.1. In their case, the health data was not populating correctly. They did get the page to open, just with incomplete data.