High unmanaged disk usage on /ngfw/var on 7.0.4 FTD

[Herald Sison]

anyone experienced this weird error. i dont find any reason for the disk to be full since it is still running at 45%.

tried clearing some log files on these directories but still the error still present

• /var/sf/detection_engines/<some GUID>/backup/
• /var/sf/detection_engines/<some GUID>/instance-1/backup/
• /var/sf/detection_engines/<some GUID>/instance-2/backup/
• /var/sf/detection_engines/<some GUID>/instance-3/backup/

and also tried from these forum.

https://www.lammle.com/post/fn-70466-ftd-high-unmanaged-disk-utilization-on-firepower-appliances-due-to-untracked-files/?unapproved=223398&moderation-hash=5b9456c268d5ce0ddbf2b6f63d3e882e#comment-223398

despite all of those actions the error still present.

[osn]

Hi !

I have the same problem but i cant perform the "kill" command it says operation not permitted. I have logged in in expert mode on a FTD 

[Yordan1]

Hi,

did you tried sudo su first?

[osn]

Thanks it helped 

[alopezc105]

HI, please help,  iam using user -admin,  ASA5516    FMC 7.0.4

,mode expert   

pidof syslog-ng

2022

Kill 2022   

bash: kill (2022) - Operation not permitted

service syslog-ng restart :  unrecognized service

HAve I to use user root maybe?

[Marvin Rhoads]

Yes, killing a process requires you be root / su (super user).

So "sudo su -" first to change to root.

@Herald Sison noted this earlier in this thread.

[CJ Bird]

This is good info, as I am experienceing the same issue - thanks.  Now, the million-dollar question is, should I upgrade to 7.0.5, which is the 'recommended' version, or do I jump to the latest & greatest 7.2.3?  Which also begs to ask, if I elect to upgrade to 7.2.3, will that be an incremental upgrade, where I am required to upgrade to an intermediate IOS version?  I've run into this issue with both MS and Catalyst OS/IOS upgrades.  

[Marvin Rhoads]

7.0.5 is safest and includes the "permanent" fixes for the known (to date) high unmanaged disk space bugs.

7.2.3 is also a good release and may be required if you have the new 3100 series - those need at least 7.1.x. Upgrading to 7.2.3 is a single step upgrade from any supported earlier release.

7.2.4 will probably be the next suggested release but it won't be released until sometime in April 2023 (current projection, subject to change).

[CiscoBrownBelt]

I have this problem too. W ill delete and of the reference "Deleted" files reference previously then restart the syslog service 

[Marius Gunnerud]

I had this same issue last week where /ngfw went in 100% used and lost connectivity to FMC as well as via SSH.  I did an internal writeup of it and I'll post it here.

##############################

FMC complains about an FTD with High Unmanaged Disk Usage. In this instance the /ngfw was consuming 100% of its allocated disk space causing it to lose connectivity to the FMC as well as the device was unreachable on SSH. Associated with bug ID CSCwd87227

The issue was due to syslog_ng not overwriting (rotating) older logs.

root@fpr:Volume# df
Filesystem 1K-blocks Used Available Use% Mounted on
rootfs 3412056 6364 3405692 1% /
devtmpfs 3437312 9336 3427976 1% /dev
tmpfs 4053836 500 4053336 1% /run
/dev/sda1 7558312 1794576 5373132 26% /mnt/boot
/dev/sda2 945144 199392 696908 23% /opt/cisco/config
/dev/sda3 945144 68436 827864 8% /opt/cisco/platform/logs
/dev/sda5 156177912 125307820 30870092 81% /opt/cisco/csp
/dev/sda4 28705788 45116 27195840 1% /var/data/cores
cgroup_root 4053836 0 4053836 0% /dev/cgroups
none 140288 90936 49352 65% /dev/shm/snort
tmpfs 1024 0 1024 0% /var/data/cores/sysdebug/tftpd_logs

root@fpr:/# lsof | grep deleted
syslog-ng 6111 root 32w REG 8,5 110027440128 807643062 /ngfw/var/log/process_stderr.log.1 (deleted)
syslog-ng 6111 root 33w REG 8,5 4116480 807643060 /ngfw/var/log/process_stdout.log.1 (deleted)
syslog-ng 6111 6112 syslog-ng root 32w REG 8,5 110027440128 807643062 /ngfw/var/log/process_stderr.log.1 (deleted)
syslog-ng 6111 6112 syslog-ng root 33w REG 8,5 4116480 807643060 /ngfw/var/log/process_stdout.log.1 (deleted)
syslog-ng 6111 37078 syslog-ng root 32w REG 8,5 110027440128 807643062 /ngfw/var/log/process_stderr.log.1 (deleted)
syslog-ng 6111 37078 syslog-ng root 33w REG 8,5 4116480 807643060 /ngfw/var/log/process_stdout.log.1 (deleted)

Change /var/log/process_std*.log { to /ngfw/var/log/process_std*.log {

root@fpr:logrotate-5min.d# pwd
/ngfw/etc/logrotate-5min.d

root@fpr:logrotate-5min.d# cat pm.logrotate
/ngfw/var/log/process_std*.log {
missingok
compress
copytruncate
maxsize 1G
rotate 4
sharedscripts
}

root@fpr:logrotate-size.d# pwd
/ngfw/etc/logrotate-size.d

root@fpr:logrotate-size.d# cat pm.logrotate
/ngfw/var/log/process_std*.log {
missingok
compress
nocreate
}

Next remove the pm.logrotate files and restart the syslog-ng process.

rm -f /ngfw/etc/logrotate.d/pm.logrotate
rm -f /ngfw/etc/logrotate-size.d/pm.logrotate

/ngfw/etc/rc.d/init.d/syslog-ng restart

Once the syslog-ng process is started initiate a logrotate job:

logrotate -v /ngfw/etc/logrotate-5min.d/pm.logrotate

Check disk usage:

root@fpr:logrotate-size.d# df
Filesystem 1K-blocks Used Available Use% Mounted on
rootfs 3412056 6368 3405688 1% /
devtmpfs 3437312 9336 3427976 1% /dev
tmpfs 4053836 500 4053336 1% /run
/dev/sda1 7558312 1794576 5373132 26% /mnt/boot
/dev/sda2 945144 199376 696924 23% /opt/cisco/config
/dev/sda3 945144 72032 824268 9% /opt/cisco/platform/logs
/dev/sda5 156177912 17857168 138320744 12% /opt/cisco/csp
/dev/sda4 28705788 45116 27195840 1% /var/data/cores
cgroup_root 4053836 0 4053836 0% /dev/cgroups
none 140288 88064 52224 63% /dev/shm/snort
tmpfs 1024 0 1024 0% /var/data/cores/sysdebug/tftpd_logs

[CiscoBrownBelt]

Awesome! Actually I have a bunch of deleted files when viewing grep. Does nto look like any are from syslog, but from snort.

[mgawrylczyk]

While this workaround is ok, you may just wish to get rid of the log file affecting the file system. These steps may be repeated whenever needed, but I would suggest to upgrade to 7.2.4, which is supposed to fix the issue.

cd /ngfw/var/log
ls -al | grep process_st
-rw-r----- 1 root adm 57927451 May 18 19:40 process_stderr.log
-rw-r----- 1 root adm 2585122 May 18 19:40 process_stdout.log

rm process_stderr.log
/ngfw/etc/rc.d/init.d/syslog-ng restart

df -h
Filesystem Size Used Avail Use% Mounted on
rootfs 7.2G 11M 7.2G 1% /
devtmpfs 7.3G 44M 7.2G 1% /dev
tmpfs 7.9G 504K 7.9G 1% /run
/dev/sda1 7.3G 949M 6.0G 14% /mnt/boot
/dev/sda2 923M 23M 853M 3% /opt/cisco/config
/dev/sda3 923M 36M 840M 5% /opt/cisco/platform/logs
/dev/sda5 149G 18G 132G 12% /opt/cisco/csp
/dev/sda4 28G 45M 26G 1% /var/data/cores
cgroup_root 7.9G 0 7.9G 0% /dev/cgroups
none 375M 0 375M 0% /dev/shm/snort
tmpfs 1.0M 0 1.0M 0% /var/data/cores/sysdebug/tftpd_logs

[Marius Gunnerud]

In my case I also saw that /ngfw/Volume was the largest used partition.

[CiscoBrownBelt]

thanks!

[Marvin Rhoads]

The syslog-ng deleted files is just the most common of several bugs that can cause this issue.

If you are unable to upgrade to 7.2.4 (which fxies these and many other bugs) or even 7.0.5 (7.0.6 as soon as it is released) then I would recommend opening a TAC case.

Further deletion of files not already documented in this thread can be quite dangerous and, if the wrong files are deleted, can leave your system unusable.

[harishbau084]

@Marvin Rhoads 

It worked for me. Thank you.

root@FW:/ngfw# lsof | grep deleted
syslog-ng 4938 root 13w REG 8,4 1212416 823380 /ngfw/var/log/process_stdout.log.1 (deleted)
syslog-ng 4938 root 27w REG 8,4 39229030400 823383 /ngfw/var/log/process_stderr.log.1 (deleted)
syslog-ng 4938 4939 syslog-ng root 13w REG 8,4 1212416 823380 /ngfw/var/log/process_stdout.log.1 (deleted)
syslog-ng 4938 4939 syslog-ng root 27w REG 8,4 39229030400 823383 /ngfw/var/log/process_stderr.log.1 (deleted)
syslog-ng 4938 60269 syslog-ng root 13w REG 8,4 1212416 823380 /ngfw/var/log/process_stdout.log.1 (deleted)
syslog-ng 4938 60269 syslog-ng root 27w REG 8,4 39229030400 823383 /ngfw/var/log/process_stderr.log.1 (deleted)
mariadbd 5857 mysql 6u REG 0,18 0 24946 /var/volatile/tmp/#24946 (deleted)
mariadbd 5857 mysql 7u REG 0,18 0 24947 /var/volatile/tmp/#24947 (deleted)
mariadbd 5857 mysql 8u REG 0,18 0 29409 /var/volatile/tmp/#29409 (deleted)
mariadbd 5857 mysql 11u REG 0,18 0 29410 /var/volatile/tmp/#29410 (deleted

pidof syslog-ng
kill 4938

Br-