LogPartitionHighWaterMarkExceeded

[Laurent Protti]

Hello,

upgraded our cluster to 12.5.1.12900-115.

on 2 nodes, we get this LogPartitionHighWaterMarkExceeded and common partition usage disk is high (between 87 and 95).

Files get purged but never go below 85-87 %.

Tried to use the free common space cop no difference

common space used on the problem node

admin:show diskusage common sort

This command can take significantly long time,
and can also effect the system wide IOWAIT on your system.
Continue (y/n)?y
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda6 69235192 61198896 4496280 94% /common
61144032 /common/
61128100 /common/log
61097092 /common/log/taos-log-a
53141964 /common/log/taos-log-a/syslog
23770472 /common/log/taos-log-a/syslog/AlternateSyslog
14888716 /common/log/taos-log-a/syslog/CiscoSyslog
14359932 /common/log/taos-log-a/syslog/messages

common space used on a normal node

admin:show diskusage common sort

This command can take significantly long time,
and can also effect the system wide IOWAIT on your system.
Continue (y/n)?y
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda6 69235192 32090128 33605048 49% /common
32037424 /common/
32021492 /common/log
31990336 /common/log/taos-log-a
15665412 /common/log/taos-log-a/syslog
12772228 /common/log/taos-log-a/cm
9841284 /common/log/taos-log-a/syslog/AlternateSyslog
7536020 /common/log/taos-log-a/cm/trace
5238932 /common/log/taos-log-a/syslog/messages

could the difference come from a difference in trace settings ? I tried looking at them between the 2 nodes (on call manager and cti service mostly) but could not find a difference...

any help would be welcome :-)

[Adam Pawlowski]

Have you tried to change the high and low watermarks to lower levels to cause the system to purge? It purges when it hits the high until it's at the low.

[Laurent Protti]

Hi Adam,

Yes I tried it but the disk consumption never lowered below 84 %.

[Laurent Protti · Accepted solution]

Well, after inspecting the filesystem, I noticed that CiscoSylog and AlternateSyslog files were huge.

Tried to delete them but they were in use, so I rebooted the node and after a few hours, I noticed that log rotation begun. before there was no file indentation. Might be a leftover of our migration...

admin:file list activelog /syslog detail
15 Aug,2020 15:32:05 11,437,477 AlternateSyslog
15 Aug,2020 15:01:03 22,111,503 AlternateSyslog.1
15 Aug,2020 14:01:02 22,118,142 AlternateSyslog.2
15 Aug,2020 13:01:02 22,116,963 AlternateSyslog.3
15 Aug,2020 12:01:02 22,081,090 AlternateSyslog.4
20 Jun,2020 13:13:04 0 CallManager
15 Aug,2020 15:32:05 7,168,068 CiscoSyslog
15 Aug,2020 15:01:02 13,851,767 CiscoSyslog.1
15 Aug,2020 14:01:02 13,856,123 CiscoSyslog.2
15 Aug,2020 13:01:02 13,858,293 CiscoSyslog.3
15 Aug,2020 12:01:02 13,834,927 CiscoSyslog.4
14 Aug,2020 14:03:59 162,693 boot.log
15 Aug,2020 15:32:01 5,069,951 cron
14 Aug,2020 13:57:55 825 cucminvstart.log
15 Aug,2020 15:32:02 1,839,519 maillog
15 Aug,2020 15:32:05 3,796,899 messages
15 Aug,2020 15:01:02 7,324,547 messages.1
15 Aug,2020 14:01:02 7,329,878 messages.2
15 Aug,2020 13:01:02 9,048,451 messages.3
15 Aug,2020 12:01:01 7,312,749 messages.4
20 Jun,2020 20:17:14 0 nscd.log
15 Aug,2020 15:31:47 31,570 ntp_reachability.sh.log
14 Aug,2020 13:58:26 1,816 ntp_start.sh.log
14 Aug,2020 13:56:15 2,364 ntp_stop.sh.log
15 Aug,2020 15:31:46 20,667 sd_ntp.log
15 Aug,2020 15:31:58 412,823 secure
14 Aug,2020 14:00:58 29,099,551 secure.1
14 Aug,2020 14:01:49 89,740 setkeysdetails.sh.log
20 Jun,2020 13:13:04 0 spooler
15 Aug,2020 15:31:47 4,108 sysconfig_validation.sh.log