CSCul18667 - CUIC doesnt purge JMX logs - 1

[ibonsall]

REF - CSCul18667

Release notes for Unified Contact Center Express 9.0(2)SU2

These mention that this bug is fixed in this version but this is not reflected in the BUG ID as the known fixed version ?

Also this bug ID has 27th Jan 2015 as the Last Modified date.

Would anyone know if this is indeed fixed or is still an open BUG before we attempt to patch a customer to resolve this issue.

Thanks

Ian

[stuartbedale]

Hi Ian

I had this problem too and I had to manually delete files.

file list activelog cuic/logs/jmx/

and then

file delete .....

I am testing a work-around where I don't allow that many files to build up in the first place, the default is 1500 @ 50MB each.  Hopefully then we never have to rely on the High Water Mark purge process to delete files.

The trace settings are in UCCX Serviceability/Trace/Configuration/Select Service = Cisco Unified Intelligence Centre Services.

Hope this helps.

Stuart

[stuartbedale]

Hi Ian

Hmm.. the work-around does not work.  Having changed the setting to 5 files at just 5MB in size, I still get more logs.

I have also tried upgrading a lab box to 9.0(2)SU2 as the release notes say this is fixed, but with the same settings, I still have more logs and at they keep at 50MB.

admin:file list activelog cuic/logs/jmx/ detail
27 Feb,2015 23:01:01            0  JMX-INFO-jmx.2015-02-27T14-54-17.298.startup.log
27 Feb,2015 23:01:01       56,010  JMX-INFO-jmx.2015-02-27T14-54-17.298.startup.log.1.gz
27 Feb,2015 22:01:03       92,513  JMX-INFO-jmx.2015-02-27T14-54-17.298.startup.log.2.gz
27 Feb,2015 21:01:02       92,478  JMX-INFO-jmx.2015-02-27T14-54-17.298.startup.log.3.gz
27 Feb,2015 20:01:03       92,597  JMX-INFO-jmx.2015-02-27T14-54-17.298.startup.log.4.gz
27 Feb,2015 19:01:03       92,520  JMX-INFO-jmx.2015-02-27T14-54-17.298.startup.log.5.gz
27 Feb,2015 18:01:04       92,691  JMX-INFO-jmx.2015-02-27T14-54-17.298.startup.log.6.gz
27 Feb,2015 17:01:02       92,137  JMX-INFO-jmx.2015-02-27T14-54-17.298.startup.log.7.gz
27 Feb,2015 16:01:04      120,718  JMX-INFO-jmx.2015-02-27T14-54-17.298.startup.log.8.gz
27 Feb,2015 15:01:02      109,443  JMX-INFO-jmx.2015-02-27T14-54-17.298.startup.log.9.gz
28 Feb,2015 06:45:46   50,000,331  JMX-INFO-jmx.2015-02-27T22-37-01.946.log
28 Feb,2015 14:54:16   50,001,092  JMX-INFO-jmx.2015-02-28T06-45-46.940.log
28 Feb,2015 23:03:01   50,001,351  JMX-INFO-jmx.2015-02-28T14-54-16.893.log
01 Mar,2015 07:11:46   50,000,331  JMX-INFO-jmx.2015-02-28T23-03-01.881.log
01 Mar,2015 15:20:16   50,001,092  JMX-INFO-jmx.2015-03-01T07-11-46.868.log
01 Mar,2015 23:29:01   50,001,351  JMX-INFO-jmx.2015-03-01T15-20-16.883.log
02 Mar,2015 07:37:46   50,000,331  JMX-INFO-jmx.2015-03-01T23-29-01.940.log
02 Mar,2015 15:46:16   50,001,092  JMX-INFO-jmx.2015-03-02T07-37-46.903.log
02 Mar,2015 23:55:01   50,001,351  JMX-INFO-jmx.2015-03-02T15-46-16.823.log
03 Mar,2015 08:03:46   50,000,331  JMX-INFO-jmx.2015-03-02T23-55-01.754.log
03 Mar,2015 11:18:31   19,948,662  JMX-INFO-jmx.2015-03-03T08-03-46.752.log
dir count = 0, file count = 21
admin:
admin:sho status

Executed command unsuccessfully
Incomplete command entered
admin:show status

Host Name    : ucs-uccx-pri
Date         : Tue Mar 3, 2015 11:25:15
Time Zone    : Greenwich Mean Time (Europe/London)
Locale       : en_US.UTF-8
Product Ver  : 9.0.2.11002-27

It is as if the Callmanager takes no notice of these settings, or that JMX files are not included in these limits.  Maybe the disk space purge works when a limit is reached, I will test further......

Stuart

[stuartbedale]

Hmm... I have set the Low and High water marks to 50 and 55%.  The alerts have been triggered and usage is now down to 52%, but I still have all the JMX files.  I am not sure how I can get Callmanager to delete these JMX files and nothing else.....maybe disable all other logging and wait for JMX to trigger the alerts.

[bvanbenschoten]

Just ran into this myself.

Can delete log files via RTMT by downloading and deleting or via CLI
CLI: 

file list activelog /cuic/logs/jmx/*
file delete activelog  /cuic/logs/jmx/* noconfirm

They are huge files.  I don't think the high and low watermark settings will remove them

Need to do it before the disk fills up and the server crashes.  Then you need TAC's help.

[Jason Hendrix]

I had this issue today and this was the fix for me...

Upgrading from UCCX Version: 9.0.2.11001-24 to UCSInstall_UCCX_10_6_1_UCOS_10.6.1.10000-39.sgn.iso.

Don't forget the ciscouccx.refresh_upgrade_v1.7.cop.sgn file first if u r on my versions.

BUT PUB system did not comeback online after reboot. TAC said to VMware into server and SHUTDOWN and RESTART. Then the upgrade went as normal. Took a couple of hours but working.

http://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cust_contact/contact_center/crs/express_10_6/install/guide/UCCX_BK_C2956BDB_00_uccx-install-upgrade-guide-106.pdf
 

Some CLI cmd's

utils dbreplication runtimestate
show hardware
show myself
show status <-- License MAC  : for TAC for licensing team for new licenses
show uccx version
show version active
show version inactive
utils system upgrade status
utils uccx dbreplication status
utils service list
utils ntp status
show uccx license <-- Cisco Unified CCX Premium <-- software type

utils system switch-version

[mvf]

This issue is very dangerous. My UCCX system has completely broken due to zero space on the log filesystem.

Even after manual deletion of JMX logs the UCCX engine failed to start. A tons of strange JavaBeans errors are in the logs. So the only solution was to completely rebuild the system, so be careful.

[lguyomard]

I had the same issue yesterday but fortunately Cisco has managed to restore the database with root account.