High unmanaged disk usage on /ngfw/var on 7.0.4 FTD

[Herald Sison]

anyone experienced this weird error. i dont find any reason for the disk to be full since it is still running at 45%.

tried clearing some log files on these directories but still the error still present

• /var/sf/detection_engines/<some GUID>/backup/
• /var/sf/detection_engines/<some GUID>/instance-1/backup/
• /var/sf/detection_engines/<some GUID>/instance-2/backup/
• /var/sf/detection_engines/<some GUID>/instance-3/backup/

and also tried from these forum.

https://www.lammle.com/post/fn-70466-ftd-high-unmanaged-disk-utilization-on-firepower-appliances-due-to-untracked-files/?unapproved=223398&moderation-hash=5b9456c268d5ce0ddbf2b6f63d3e882e#comment-223398

despite all of those actions the error still present.

[Chess Norris]

Hi,

I had some customers with this issue lately, but there is a workarround and it's described here https://bst.cisco.com/bugsearch/bug/CSCwb34240 

/Chess

[Marvin Rhoads]

There are at least five "high unmanaged disk space" bugs.

I've been collecting a list:

https://bst.cisco.com/bugsearch/bug/CSCvt77813
https://bst.cisco.com/bugsearch/bug/CSCvo74833
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwb34240
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwc30487
https://bst.cisco.com/bugsearch/bug/CSCvy26511

[ethutchinson]

Marvin,

I am having the same issue with my FTD 1140 running 7.0.4. When I run pidof syslog-ng I get three PIDS

8058 8057 and 6464. Which would be the correct PID to kill?

[Marvin Rhoads]

@ethutchinson generally speaking the last listed one does the trick. So in your case at hand, "kill 6464".

[ethutchinson]

Marvin,

Thanks for the assist. Killing the syslog-ng pid (third one in list) worked.

[councilm]

Run the LSOF command again but also grep for syslog-ng.

lsof | grep deleted | grep syslog-ng

Then kill any PIDs that are also in the list from "pidof".

kill -n 1 <PID>

[Marius Gunnerud]

@ethutchinson Please open a new post for this so we can help you better and easier for other to find should the solution be different than that of this post.

[ppejjorgensen]

This fix seems to solve the problem permanently. I used it with a customer on 28 Dec. 22 and so far I haven't seen any errors related to "High unmanaged disk usage". Thank you Chess Norris.

[Herald Sison]

I am sick of manually clearing these files just to lower the unmanaged disk usage. i do this every 10 days.

every 10days i always get this error. before, even if my disk usage is only at 60% the error keeps popping out and i followed what TAC told me that to change some values in diskmanager.conf file, i thought that the error will be gone but when the disk usage reaches 80% the same error pops up again. So i am back again at clearing the freaking log files. As per TAC this bug has been fixed in FMC 7.3.0 and FTD 7.0.5 but i am already running 7.3.0 but still this errors pops up and some new bugs came out. Are we expecting a chain of BUGS here? 

i will try to upgrade my FTD to 7.0.5 once i can ask for maintenance window maybe during Sundays and hoped that this freaking bug will be gone forever. and by the way my device ASA5508X will have its last FTD version which is 7.0.5 (and it is already gold star) so i am really expecting that this version would really be it, fingers crossed.

/etc/sf/diskmanager.conf file

- Change:

          percent_exceeded 60;

     TO:

           percent_exceeded 25;

- Restart diskmanager process using pmtool. "pmtool restartbyid diskmanager"

  

This is the disk usage after clearing up the log files in:

/ngfw/var/sf/detection_engines/<uuid>/instance-*/fileperfstats.log.*
/ngfw/var/sf/detection_engines/<uuid>/instance-*/ssl-certs-unified.log.*
/ngfw/var/sf/detection_engines/<uuid>/instance-*/ssl-nse-debug.log.*
/ngfw/var/sf/detection_engines/<uuid>/instance-*/ssl-stats-unified.log.*

This is the disk usage after performing:

lsof | grep deleted
pidof syslog-ng
kill <pid returned from previous command>

pmtool restartbyid diskmanager

[Milos_Jovanovic]

I agree with @Chess Norris that this is most likely caused by CSCwb34240 (at least on 7.0.4). I observed the behavior, right after I manually killed syslog-ng process, and noticed that behavior is indeed tied to log rotation (v7.0.4). As soon as I kill syslog-ng, file disappears, but soon file is recreated with the same name, and continues to grow.

Since then, I've implemented this workaround on 20+ devices, and issue never reappeared. I'm speaking from the experience of this issue and v7.0.x only.

Kind regards,

Milos

[th3r1dd1ck]

As far as my issue, this was my solution.

Cisco Firepower Extensible Operating System (FX-OS) v2.10.1 (build 208)
Cisco Firepower 2140 Threat Defense v7.0.4 (build 55)

[derek.small]

What about if you are getting this error in FMC, but the firewalls don't show any signs of what I would consider excessive disk use, nor do I find any deleted files when I run the command everyone refers to "lsof | grep deleted".  I don't see anything that would merit an alert about disk space or disk usage in the output below.

 admin@firepower:/$ df
Filesystem 1K-blocks Used Available Use% Mounted on
rootfs 7862912 589412 7273500 8% /
devtmpfs 7966776 649188 7317588 9% /dev
tmpfs 8056044 496 8055548 1% /run
tmpfs 8056044 4548 8051496 1% /var/volatile
/dev/sda1 945144 272712 623588 31% /opt/cisco/config
/dev/sda2 944120 49568 845760 6% /opt/cisco/platform/logs
/dev/sda3 11403544 28764 10788848 1% /var/data/cores
/dev/sda4 83948496 26830936 57117560 32% /opt/cisco/csp
/dev/sdb1 7676252 2199012 5477240 29% /mnt/boot
cgroup_root 8056044 0 8056044 0% /dev/cgroups
tmpfs 8056044 0 8056044 0% /sys/fs/cgroup
tmpfs 8056044 0 8056044 0% /sys/fs/cgroup/pm
none 363520 12 363508 1% /dev/shm/snort
tmpfs 1024 0 1024 0% /var/data/cores/sysdebug/tftpd_logs
admin@firepower:/$ 

[Herald Sison]

Hi Sir,

Try checking the log files from these directories below and if you find something that is defined below then you can delete it then run the "lsof | grep deleted" command again.

• /var/sf/detection_engines/<uuid>/instance-*/fileperfstats.log.*
• /var/sf/detection_engines/<uuid>/instance-*/ssl-certs-unified.log.*
• /var/sf/detection_engines/<uuid>/instance-*/ssl-nse-debug.log.*
• /var/sf/detection_engines/<uuid>/instance-*/ssl-stats-unified.log.*
• /var/sf/detection_engines/<some GUID>/backup/
• /var/sf/detection_engines/<some GUID>/instance-1/backup/
• /var/sf/detection_engines/<some GUID>/instance-2/backup/

after deleting you need to run the restart diskmanager "pmtool restartbyid diskmanager"

then run the "lsof | grep deleted" command

[Marvin Rhoads]

@derek.small it's the /ngfw folder that the alert is triggering on. So check it with "df -k /ngfw".

[gsalcedo]

Having the same issue but i don't have output for lsof | grep deleted

bpnfw04:~$ df -k /ngfw
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda6 41943040 41660476 282564 100% /ngfw
bpnfw04:~$ lsof | grep deleted
bpnfw04:~$