log4j:ERROR Failed to flush writer, java.io.IOException: No space left

[jaheshkhan]

while SSH to CUCM we are getting the following messages:

Command Line Interface is starting up, please wait ...
log4j:ERROR Failed to flush writer,
java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:345)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
at org.apache.log4j.helpers.QuietWriter.flush(QuietWriter.java:59)
at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:324)
at org.apache.log4j.RollingFileAppender.subAppend(RollingFileAppender.ja va:276)
at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
at com.cisco.iptplatform.fappend.ciscoRollingFileAppender.append(ciscoRo llingFileAppender.java:81)
at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders (AppenderAttachableImpl.java:66)
at org.apache.log4j.Category.callAppenders(Category.java:206)
at org.apache.log4j.Category.forcedLog(Category.java:391)
at org.apache.log4j.Category.debug(Category.java:260)
at com.cisco.iptplatform.cli.CliSettings.getInstance(CliSettings.java:11 6)
at sdMain.initialize(sdMain.java:521)
at sdMain.main(sdMain.java:676)

admin:show status

Host Name : CUCM01.sst.local
Date : Tue Mar 21, 2023 18:50:07
Time Zone : Gulf Standard Time (Asia/Dubai)
Locale : en_US.UTF-8
Product Ver : 12.5.1.10000-22
Unified OS Version : 7.0.0.0-4

Uptime:
18:50:10 up 434 days, 9:13, 1 user, load average: 3.94, 2.66, 2.37

CPU Idle: 00.00% System: 13.13% User: 86.87%
IOWAIT: 00.00% IRQ: 00.00% Soft: 00.00%

Memory Total: 5945700K
Free: 141756K
Used: 4722744K
Cached: 121168K
Shared: 666356K
Buffers: 1081200K

Total Free Used
Disk/active 14154188K 954052K 13038624K (94%)
Disk/inactive 14154188K 13371164K 40984K (1%)
Disk/logging 49573456K 0K 49557072K (100%)

logging showing 100% and active disk shows 94%

In this case what we need to do? do we need to reboot? if tried to reboot will it not get bootup?

do I need to run cop  file for this to increase the space? how can I increase diskspace?

[Nithin Eluvathingal]

Try this method.

https://www.cisco.com/c/en/us/support/docs/unified-communications/unified-communications-manager-callmanager/200581-Procedure-to-Adjust-WaterMark-in-RTMT-of.html

2.12.0.0

[jaheshkhan]

in this case do I need to reduce the value from 95 to 90 for highwatermark and 90 to 85 for lowwatermark?

[Jonathan Schulenberg]

Unless I’m overlooking something, adjusting the watermark will not actually clear up space; that’s just the alarm.

For logging you can download log files, say older than a month, and tell RTMT to delete those files from the server.

For the active partition, removing old/unused firmware will likely have the greatest impact.

[jaheshkhan]

how to delete old log file using RTMT?

active shows 94%

we also facing DRS not working properly in same server. service is not getting started even after trying to restart it. 

[Jonathan Schulenberg]

The first Google result: How to gather logs with RTMT

I wouldn’t expect DRS to work until you have useable space; it can’t create the tarballs if there is no disk space.

[jaheshkhan]

so are you telling to remove old and unused firmware? is there any other way to increase the disk space?

if disk logging dont have enough space DRS will fail? 

[Jonathan Schulenberg]

There is no supported method to expand the virtual disks; the size is determined by the OVA you deployed: 80GB for 2,500 user and below or 110GB for 7,500 users and above. There is also a COP file to free up disk space on the common partition that I should have mentioned earlier though: ciscocm.free_common_space_v1.9.k4.cop.sha512.

[iptsupport]

Hello , I same problem. 

I try to remove logs from RTMT and change Adjust WaterMark and it still disk logging is 100%

if phone preform reset, he can't register again to this CUCM .

My CUCM version is 12.5

any idea?

[Carlo Poggiarelli]

Hi,

Did you try what Jonathan suggested?

The other option is to make a full DRS backup of your cluster, Install a new one using a VM with 110GB Disk size and recover the backup.

There are different methods to perform this migration with minimum impact to the users. One of these is to create the new culster's VM putting Network interface to a different VLAN while installling and recovering it. Once done, just disconnect previous CUCM network interface and put the new cluster in the production VLAN.

During installation, CUCM checks the connectivity to DNS , NTP and Gateway, in this case you can Use a CSR router (Eg. V1000) and configure NTP and DNS service on it just for the time you need to perform the CUCM installation.

If you need further info, don't hesitate to ask.

HTH

Regards

Carlo