Re: Prime 3.5 logrotate not working

patoberli · ‎08-20-2019

Hello All

My PI 3.5 installation has the latest Update 3 installed and I just wanted to install the latest Upgrade 3.5.1. This failed, because the /var was full (once again....).

I dug a little bit deeper and discovered that logrotate is not working in this version. Because of that is my /var/wtmp > 2.7 GB and the /var/messages > 0.5 GB.

Luckily I had a VM Snapshot and could restore the working installation (upgrade failed and left me with a broken installation).

I manually cleaned the wtmp file by doing this:

#enter shell:
shell
#get root
sudo su -
#write an empty new wtmp file (this can take a minute)
cat /dev/null > /var/log/wtmp

After this I started to troubleshoot the logrotate. It seems that SELINUX is blocking the run (or a faulty logrotate.d/syslog config file).

Error messages on CLI:

[root@cpi1 logrotate.d]# logrotate -df /etc/logrotate.d/syslog
reading config file /etc/logrotate.d/syslog
Allocating hash table for state file, size 15360 B

Handling 1 logs

rotating pattern: /var/log/cron
/var/log/maillog
/var/log/messages
/var/log/secure
/var/log/spooler
 forced from command line (no old logs will be kept)
empty log files are rotated, old logs are removed
considering log /var/log/cron
error: skipping "/var/log/cron" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /var/log/maillog
error: skipping "/var/log/maillog" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /var/log/messages
error: skipping "/var/log/messages" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /var/log/secure
error: skipping "/var/log/secure" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
considering log /var/log/spooler
error: skipping "/var/log/spooler" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
[root@cpi1 logrotate.d]#

[root@cpi1 logrotate.d]# run-parts /etc/cron.daily/
/etc/cron.daily/cronCleanup:

find: â€˜/localdisk/tftp/RUNNINGCONFIG*.cfgâ€™: No such file or directory
find: â€˜/localdisk/tftp/STARTUPCONFIG*.cfgâ€™: No such file or directory
find: â€˜/localdisk/tftp/2019*.cfgâ€™: No such file or directory
find: â€˜/localdisk/tftp/2018*.cfgâ€™: No such file or directory
/etc/cron.daily/logrotate:

error: Ignoring ADE because of bad file mode - must be 0644 or 0444.
error: Ignoring charon.logrotate because of bad file mode - must be 0644 or 0444.
error: skipping "/var/log/boot.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
error: skipping "/opt/CSCOlumos/logs/ftpadmin.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
error: skipping "/opt/CSCOlumos/logs/tftpadmin.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
error: skipping "/var/log/squid/*.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
error: skipping "/var/log/cron" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
error: skipping "/var/log/maillog" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
error: skipping "/var/log/messages" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
error: skipping "/var/log/secure" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
error: skipping "/var/log/spooler" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
error: skipping "/var/log/vsftpd.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
error: skipping "/var/log/xferlog" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
error: skipping "/var/log/wpa_supplicant.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
error: skipping "/var/log/yum.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
error: skipping "/var/log/wtmp" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
error: skipping "/var/log/btmp" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
[root@cpi1 logrotate.d]#

Digging a little bit deeper now, I think I should be able to solve it (temporarily) with setting permissions inside the /etc/logrotate.d/syslog file.

Velin Georgiev · ‎08-22-2019

Hello,

Did you manage to fix this issue? Recently I've noticed I have the same problem on two of my CPI installations.

patoberli · ‎08-22-2019

See my new messages.

patoberli · ‎08-22-2019

Issue is the /var/log/wtmp file, which uses nearly 80% of the partition. First I have to clean this.

#enter shell:
shell
#get root
sudo su -
#write an empty new wtmp file (this can take a minute)
cat /dev/null > /var/log/wtmp

#In my case the messages file was also already > 500 MB, I'm not sure yet if I should also clean it. wtmp file was 2.7 GB though.

Now that this file is clean, time to fix the not working logrotate function.

Fixed it (probably until the next patch/major release) by editing the /etc/logrotate.d/syslog file (plus all the others for good measure, where I also received an error message). I added the line ' su prime gadmin '

[root@cpi1 logrotate.d]# vi syslog 
/var/log/cron
/var/log/maillog
/var/log/messages
/var/log/secure
/var/log/spooler
{
    su prime gadmin    
    missingok
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}

# To safe enter :wq
# It will logrotate all the files mentioned in this file within the next 24 hours automatically
# To manually rotate enter 'run-parts /etc/cron.daily/'

Please note, this will NOT cleanup the used space, as the files will not be compressed nor removed (they will be kept for 1-4 weeks). So if you need space now, remove the files with a .1, .2, .3 or .4 ending after logrotate has run.

Please also note, logrotate can only successfully run if there is at least some space left. If the partition is completely full, clean the wtmp file first (or any other large file).

To rotate wtmp and btmp, edit the /etc/logrotate.conf to:

…
…
…
# other lines removed
/var/log/wtmp {
    su root guser    
    monthly
    create 0640 root guser
    rotate 1
}
/var/log/btmp {
    su root guser    
    weekly
    create 0600 root guser
    rotate 4
}
# other lines removed
…
…
.

Velin Georgiev · ‎08-22-2019

Thank you for sharing this!
Now I started the 3.5.1 maintenance upgrade and will see if it's fixed there. If not, I will try your solution.

Velin Georgiev · ‎08-22-2019

Upgrading to 3.5.1 did not resolve the issue, so I used your method.

Thanks again!

jon.ellis · ‎11-20-2019

Very good write up patoberli, I've recently hit this and found the following bugs

CSCvp38163

Although the bug does state /secure files will take/use all the var space the description includes the wtmp file also.

PI version 3.6 includes the fix for this issue

Your workaround did the trick for me :)

Kind Regards Jon

andrewswanson · ‎01-29-2020

Thanks for posting this. I ran into the same issue with PI 3.6 patch 1 - primary server in an ha pair ground to a halt with 100% usage on /dev/mapper/smosvg-varvol. I contacted TAC who referred me to this thread - the fix worked for me.

cheers

Andy

samuel.heinrich · ‎06-05-2020

Same problem here.

NCS did not start

xxxx/admin# ncs status

Health Monitor Server is stopped.

Database server is stopped

FTP Service is Stopped

TFTP Service is Stopped

Matlab Server is Stopped

Matlab Server Instance 1 is Stopped

Matlab Server Instance 2 is Stopped

Matlab Server Instance 3 is Stopped

NMS Server is stopped.

Coral Service is stopped..

WSA Service is stopped..

SAM Daemon is stopped.

DA Daemon is stopped.

Compliance engine is not running

/dev/mapper/smosvg-varvol 3966144 3966144 0 100% /var

[root@xx log]# cat boot.log

…

[ OK ] Started Import network configuration from initramfs.

Starting Create Volatile Files and Directories...

[FAILED] Failed to start Create Volatile Files and Directories.

See 'systemctl status systemd-tmpfiles-setup.service' for details.

Mounting RPC Pipe File System...

Starting Security Auditing Service...

[root@xxxlog]# ls -lha

total 3.7G

-rw-r-----. 1 root guser 2.6G Jun 5 09:06 wtmp

after deleting wtmp

[root@vbsgcpi01 log]# rm wtmp

rm: remove regular file ‚Äòwtmp‚Äô? y

ncs started again.

implemented the workaround.

now file size of wtmp looks stable

-rw-r-----. 1 root guser 129K Jun 5 10:35 wtmp
-rw-r-----. 1 root guser 957K Jun 5 10:26 wtmp.1