High unmanaged disk usage on /ngfw/var on 7.0.4 FTD

[Herald Sison]

anyone experienced this weird error. i dont find any reason for the disk to be full since it is still running at 45%.

tried clearing some log files on these directories but still the error still present

• /var/sf/detection_engines/<some GUID>/backup/
• /var/sf/detection_engines/<some GUID>/instance-1/backup/
• /var/sf/detection_engines/<some GUID>/instance-2/backup/
• /var/sf/detection_engines/<some GUID>/instance-3/backup/

and also tried from these forum.

https://www.lammle.com/post/fn-70466-ftd-high-unmanaged-disk-utilization-on-firepower-appliances-due-to-untracked-files/?unapproved=223398&moderation-hash=5b9456c268d5ce0ddbf2b6f63d3e882e#comment-223398

despite all of those actions the error still present.

[ppejjorgensen]

Hi All

I have a customer who upgraded from 7.0.4 to 7.2.1 due to "High Unmanaged Disk Usage on /ngfw" bugs (yes - it is several bugs since version 6.4) and here the problem is unfortunately still present. Hope Cisco finds a more permanent solution soon, because I have quite a few customers with this bug, and frankly, they're getting a little tired of all these cosmetic bugs and alarms in FMC. 

[Chess Norris]

Hi,

I had some customers with this issue lately, but there is a workarround and it's described here https://bst.cisco.com/bugsearch/bug/CSCwb34240 

/Chess

[Marvin Rhoads]

There are at least five "high unmanaged disk space" bugs.

I've been collecting a list:

https://bst.cisco.com/bugsearch/bug/CSCvt77813
https://bst.cisco.com/bugsearch/bug/CSCvo74833
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwb34240
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwc30487
https://bst.cisco.com/bugsearch/bug/CSCvy26511

[ethutchinson]

Marvin,

I am having the same issue with my FTD 1140 running 7.0.4. When I run pidof syslog-ng I get three PIDS

8058 8057 and 6464. Which would be the correct PID to kill?

[Marvin Rhoads]

@ethutchinson generally speaking the last listed one does the trick. So in your case at hand, "kill 6464".

[ethutchinson]

Marvin,

Thanks for the assist. Killing the syslog-ng pid (third one in list) worked.

[councilm]

Run the LSOF command again but also grep for syslog-ng.

lsof | grep deleted | grep syslog-ng

Then kill any PIDs that are also in the list from "pidof".

kill -n 1 <PID>

[Marius Gunnerud]

@ethutchinson Please open a new post for this so we can help you better and easier for other to find should the solution be different than that of this post.

[CiscoBrownBelt]

Hi Marvin. I get this pretty often on our FTDv. usually doing syslog-ng restart does the trick to clear it, but not anymore on one of them. Any ideas besides upgrading or something from 7.0.4?

[Marvin Rhoads]

@CiscoBrownBelt definitely plan to upgrade. Even staying in 7.0.x is OK as long as you keep up with the latest maintenance release and patch.

Meanwhile, look for largest files using one of the following commands (based on platform):

find /ngfw -type f -exec du -Sh {} + | sort -rh | head -n 15

find /var -type f -exec du -Sh {} + | sort -rh | head -n 15

We go from there based on what is observed. As always be VERY CAREFUL and don't delete anything that you are not 100% sure is unneeded! Often, TAC can provide workarounds for specific disk space issues that will ensure you don't render the device inoperable

[Marius Gunnerud]

You might just need to force a logrotate.  Mant times this will cleanup some stale / old log data that have not been overwritten.

expert

logrotate -v /ngfw/etc/logrotate-5min.d/pm.logrotate

[ppejjorgensen]

This fix seems to solve the problem permanently. I used it with a customer on 28 Dec. 22 and so far I haven't seen any errors related to "High unmanaged disk usage". Thank you Chess Norris.

[Herald Sison]

I am sick of manually clearing these files just to lower the unmanaged disk usage. i do this every 10 days.

every 10days i always get this error. before, even if my disk usage is only at 60% the error keeps popping out and i followed what TAC told me that to change some values in diskmanager.conf file, i thought that the error will be gone but when the disk usage reaches 80% the same error pops up again. So i am back again at clearing the freaking log files. As per TAC this bug has been fixed in FMC 7.3.0 and FTD 7.0.5 but i am already running 7.3.0 but still this errors pops up and some new bugs came out. Are we expecting a chain of BUGS here? 

i will try to upgrade my FTD to 7.0.5 once i can ask for maintenance window maybe during Sundays and hoped that this freaking bug will be gone forever. and by the way my device ASA5508X will have its last FTD version which is 7.0.5 (and it is already gold star) so i am really expecting that this version would really be it, fingers crossed.

/etc/sf/diskmanager.conf file

- Change:

          percent_exceeded 60;

     TO:

           percent_exceeded 25;

- Restart diskmanager process using pmtool. "pmtool restartbyid diskmanager"

  

This is the disk usage after clearing up the log files in:

/ngfw/var/sf/detection_engines/<uuid>/instance-*/fileperfstats.log.*
/ngfw/var/sf/detection_engines/<uuid>/instance-*/ssl-certs-unified.log.*
/ngfw/var/sf/detection_engines/<uuid>/instance-*/ssl-nse-debug.log.*
/ngfw/var/sf/detection_engines/<uuid>/instance-*/ssl-stats-unified.log.*

This is the disk usage after performing:

lsof | grep deleted
pidof syslog-ng
kill <pid returned from previous command>

pmtool restartbyid diskmanager

[Milos_Jovanovic]

I agree with @Chess Norris that this is most likely caused by CSCwb34240 (at least on 7.0.4). I observed the behavior, right after I manually killed syslog-ng process, and noticed that behavior is indeed tied to log rotation (v7.0.4). As soon as I kill syslog-ng, file disappears, but soon file is recreated with the same name, and continues to grow.

Since then, I've implemented this workaround on 20+ devices, and issue never reappeared. I'm speaking from the experience of this issue and v7.0.x only.

Kind regards,

Milos

[th3r1dd1ck]

As far as my issue, this was my solution.

Cisco Firepower Extensible Operating System (FX-OS) v2.10.1 (build 208)
Cisco Firepower 2140 Threat Defense v7.0.4 (build 55)