FMCv web interface error 429 Too Many Requests

[s_SiD_s]

Firewall Management Center Version 7.6.5 (build 106)

OS Cisco Firepower Extensible Operating System (FX-OS) 2.16.1 (build 147)

I have noticed today very slow web interface ... Health Monitor
grapthics are cutted, or "no data available"
Then i caught a pop-up message - 

 429 Too Many Requests

Error in fetching device details
Error: {"code":429,"description":"The server is refusing to service the request because the user has sent too many requests in a given amount of time (\"rate limiting\")","homeRef":"/","reasonPhrase":"Too Many Requests","uri":"http://tools.ietf.org/html/rfc6585#section-4"}

RFC 6585 - Additional HTTP Status Codes 

dashboard shows no errors.
tail -f /var/log/messages

watching about an 2 hours...no errors.
May be just restart entire server? 🙂
-uptime is 70 days.
a lot of free disk space. server esxi is OK
network interfaces on switch connected to ESXi - is OK,
didn't find on community same problem.

[s_SiD_s]

may be i missed something in var/log/messages?

[Aref Alsouqi]

That is a notification from the FMC side to ask the clients to slow down their requests. I think you should look at the clients that would be connected/connecting to that FMC and see if there is any that is sending bulk requests or doing some weird activities. One thing you could potentially do would be to narrow down the accesses to the FMC management console to one client at a time and see which one is causing that issue.

[s_SiD_s]

oh...i see now
PRTG, Zabbix, Graylog are pointing to FMC... could this be reason of "too many connections"?
Or does amount of connections are hardcoded or I could modify it?

[Aref Alsouqi]

It could but it depends on how those systems try to connect to the FMC but I think what @Marvin Rhoads mentioned would be the case here, so I would try to engage with TAC and see what they say.

[Marvin Rhoads]

There is a known bug in the Prometheus process that serves up the health monitoring page in FMC.

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwo49337

Cisco TAC can confirm if you are hitting it and provide a work around. It involves deleting some files in the FMC file system and is thus not recommended for most end users to attempt directly.