RTMT failed to initialize with RTMT 8.91 and JRE 1.8.0_65

[hviridis31]

We experienced that RTMT displays a window titled "Error" with the following message (see attachment):

RTMT failed to initialize. Exiting...

Log file ~/.jrtmt/log/rtmt.log ends with the following messages:

2016-XX-XX XX:17:03,453 [SplashThread] INFO  rtmt.control - doMeat(): after ServerVersionChecker.validMLALogin(): ServerVersionChecker.gStatusCode=200, bValidLogin=true
2016-XX-XX XX:17:03,458 [SplashThread] ERROR rtmt.control - TzDataManager:getStrClientTzVersion:[ERROR]:Ex: /opt/java/jre1.8.0_65/lib/zi/ZoneInfoMappings (No such file or directory)
2016-XX-XX XX:17:03,458 [SplashThread] ERROR rtmt.control - [ERROR] In run thread SplashWindow: java.lang.NullPointerException
2016-XX-XX XX:17:03,523 [SplashThread] ERROR rtmt.control - [ERROR] In run thread SplashWindow: java.lang.NullPointerException
        at com.cisco.ccm.serviceability.rtmt.utils.TzDataManager.findClientTzVersion(Unknown Source)
        at com.cisco.ccm.serviceability.rtmt.utils.TzDataManager.<init>(Unknown Source)
        at com.cisco.ccm.serviceability.rtmt.ui.JRtmtMain.doMeat(Unknown Source)
        at com.cisco.ccm.serviceability.rtmt.ui.JRtmtMain$SplashWindow$2.run(Unknown Source)
        at java.lang.Thread.run(Thread.java:745)

2016-XX-XX XX:19:39,401 [SplashThread] INFO  rtmt.control - ======>Exit System

In fact a directory /opt/java/jre1.8.0_65/lib/zi does not exist. It looks like java time zone database has changed. With JRE 1.7 database is found at some location like /opt/java/jre1.7.0_60/lib/zi/ZoneInfoMappings (see https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/time-zone001.html). Since JRE 1.8 you rather find a file like /opt/java/jre1.8.0_65/lib/tzdb.dat. RTMT 8.91 comes with Timezone Updater Tool Version 1.3.62-b01 that looks for time zone information at directory <install_dir>/lib/zi. Recent Versions of tzupdate.jar from Oracle support both database types.

user@host:~$ java -jar /opt/cisco/jrtmt/tzupdater.jar -V

The Java runtime doesn''t have time zone data (zi directory).

A workaround is to copy a legacy time zone database file (shipped by Oracle with legacy jre packages) to <install_dir>/lib/zi. I found a ZoneInfoMappings file in RTMT install directory:

user@host:~$ ls -l /opt/cisco/jrtmt/jre/lib/zi/ZoneInfoMappings
-rw-r--r-- 1 root root 14488 Jul 28  2011 /opt/cisco/jrtmt/jre/lib/zi/ZoneInfoMappings

Some people discovered that it is sufficent just to create an empty database file (see http://www.ucguerrilla.com/2015/10/installing-cisco-rtmt-105-on-apple-os-x.html).

It shows that RTMT 8.91 installation directory contains tzdata version tzdata2011g where Cisco Unified CM Administration server version 8.6.2.24901-1 comes with tzdata2013i. My local copy of JRE 1.7.0 Update 60 provided tzdata2014b. This version mismatch is why RTMT (/opt/cisco/jrtmt/JRtmt) prompts whether to update timezone tables on the client (see screenshot timezone-data-version-mismatch.png attached):

Window title: Timezone data version mismatch !!!

There is a mismatch between timezone version on this RTMT and the server you are trying to connect. Do you want to update the timzone tables on this RTMT to match to that of the server now ?

This will need a restart of RTMT.

Note: Choosing "No" would let you launch RTMT with current timezone version, but you may face some issues related to mismatch in time.

Log file ~/.jrtmt/log/rtmt.log shows:

[...]

2016-XX-XX XX:13:26,212 [SplashThread] INFO  rtmt.control - doMeat(): after ServerVersionChecker.validMLALogin(): ServerVersionChecker.gStatusCode=200, bValidLogin=true
2016-XX-XX XX:13:26,397 [SplashThread] INFO  rtmt.control - [INFO]Timezone ServerVersion:tzdata2013i
2016-XX-XX XX:13:26,397 [SplashThread] INFO  rtmt.control - [INFO]Timezone ClientVersion:tzdata2014b
2016-XX-XX XX:13:26,397 [SplashThread] INFO  rtmt.control - [INFO]Timezone isUpdateRequired:true
2016-XX-XX XX:13:26,398 [SplashThread] INFO  rtmt.control - [INFO] No tzUpdateFlagfile prompting the user.

You can use tzupdater.jar from CUCM (which is downloaded to RTMT install directory) to query time zone data version and update the data. Alternativly download a current version of Oracle Timezone Updater Tool at http://www.oracle.com/technetwork/java/javase/downloads/tzupdater-download-513681.html and install at some location like /opt/java/tzupdater-2.0.0-2015a/tzupdater.jar:

user@host:~$ java -jar /opt/java/tzupdater-2.0.0-2015a/tzupdater.jar
Usage: java -jar tzupdater.jar options
options:
    -h, --help     Print this message and exit.
    -V, --version  Print tool version, tzdata version in JRE and tzdata version embedded in tool and exit.
    -u, --update   Update time zone data.
    -l, --location <URL link to resource bundle>
                   Compile, test and update JRE timezone data from provided tzdata.tar.gz bundle
                   e.g -l http://www.iana.org/time-zones/repository/tzdata-latest.tar.gz
                   Supported URL protocols : http://, https://, file://
    -f, --force    Force to update tzdata.
    -bc, --backwardcompatible
                   (JDK 7 and earlier only) Keep JDK1.1 time zone ID compatibility. Requires -u, -f or -t.
    -t, --test     Perform verification tests and exit.
    -v, --verbose  Display detailed messages to stdout.

root@host:~# java -jar /opt/java/tzupdater-2.0.0-2015a/tzupdater.jar -v -l http://www.iana.org/time-zones/repository/releases/tzdata2015g.tar.gz
java.home: /opt/java/jre1.7.0_60
java.vendor: Oracle Corporation
java.version: 1.7.0_60
Downloaded file to /tmp/tz.tmp_2/tzdata.tar.gz
JRE tzdata version: tzdata2014b
tzupdater tool would update with tzdata version: tzdata2015g
Downloaded file to /tmp/tz.tmp_2/sha512hash
Extracting files... done.
Renaming /opt/java/jre1.7.0_60/lib/zi to /opt/java/jre1.7.0_60/lib/zi.tzdata2014b
Renaming /opt/java/jre1.7.0_60/lib/zi.tzdata2015g_1 to /opt/java/jre1.7.0_60/lib/zi
Validating for : tzdata2015g
Validation complete
JRE updated to version : tzdata2015g

For the following demonstration I switched to JRE 1.7.0 Update 60 using update-alternatives (detailed information below). Moving file ZoneInfoMappingsing we accomplish a downgrade from tzdata2014b to tzdata2011g:

user@host:~$ java -jar /opt/java/tzupdater-2.0.0-2015a/tzupdater.jar -V
tzupdater version 2.0.0-b03
JRE tzdata version: tzdata2014b
tzupdater tool would update with tzdata version: tzdata2015a

root@host:~$ mv /opt/java/jre1.7.0_60/lib/zi /opt/java/jre1.7.0_60/lib/zi.tzdata2014b
root@host:~$ mkdir /opt/java/jre1.7.0_60/lib/zi
root@host:~$ cp /opt/cisco/jrtmt/jre/lib/zi/ZoneInfoMappings /opt/java/jre1.7.0_60/lib/zi/ZoneInfoMappings

user@host:~$ java -jar /opt/java/tzupdater-2.0.0-2015a/tzupdater.jar -V
tzupdater version 2.0.0-b03
JRE tzdata version: tzdata2011g
tzupdater tool would update with tzdata version: tzdata2015a

user@host:~$ java -jar /opt/cisco/jrtmt/tzupdater.jar -V
tzupdater version 1.3.62-b01
JRE time zone data version: tzdata2011g
Embedded time zone data version: tzdata2013i

As you can see from the listing above time zone data with version tzdata2013i is shipped with tzupdater.jar from RTMT plugin. Option -u (update) will instruct tzupdater.jar to set up time zone data (requires permission to write to /opt/java/jre1.7.0_60/lib). While current tzupdater.jar from Oracle includes tzdata2015a this version additionally allows to update time zone data from Internet Assigned Numbers Authority (see http://www.iana.org/time-zones). Through option -v we ask tzupdater.jar to increase verbosity. Use option -f (force) to downgrade time zone data.

root@host:~# java -jar /opt/cisco/jrtmt/tzupdater.jar -v -u -f
java.home: /opt/java/jre1.7.0_60
java.vendor: Oracle Corporation
java.version: 1.7.0_60
JRE time zone data version: tzdata2015a
Embedded time zone data version: tzdata2013i
Extracting files... done.
Renaming directories... done.
Validating the new time zone data... done.
Time zone data update is complete.

root@host:~# java -jar /opt/cisco/jrtmt/tzupdater.jar -V
tzupdater version 1.3.62-b01
JRE time zone data version: tzdata2013i
Embedded time zone data version: tzdata2013i

root@host:~# java -jar /opt/java/tzupdater-2.0.0-2015a/tzupdater.jar -v -l http://www.iana.org/time-zones/repository/releases/tzdata2015g.tar.gz
java.home: /opt/java/jre1.7.0_60
java.vendor: Oracle Corporation
java.version: 1.7.0_60
Downloaded file to /tmp/tz.tmp_2/tzdata.tar.gz
JRE tzdata version: tzdata2013i
tzupdater tool would update with tzdata version: tzdata2015g
Downloaded file to /tmp/tz.tmp_2/sha512hash
Extracting files... done.
Renaming /opt/java/jre1.7.0_60/lib/zi to /opt/java/jre1.7.0_60/lib/zi.tzdata2013i
Renaming /opt/java/jre1.7.0_60/lib/zi.tzdata2015g_1 to /opt/java/jre1.7.0_60/lib/zi
Validating for : tzdata2015g
Validation complete
JRE updated to version : tzdata2015g

When the workaround described is applied and we answer the prompt "Do you want to update the timzone tables on this RTMT to match to that of the server now?" with "No" log file should look like this:

[...]

2016-XX-XX XX:13:26,212 [SplashThread] INFO  rtmt.control - doMeat(): after ServerVersionChecker.validMLALogin(): ServerVersionChecker.gStatusCode=200, bValidLogin=true
2016-XX-XX XX:13:26,397 [SplashThread] INFO  rtmt.control - [INFO]Timezone ServerVersion:tzdata2013i
2016-XX-XX XX:13:26,397 [SplashThread] INFO  rtmt.control - [INFO]Timezone ClientVersion:tzdata2014b
2016-XX-XX XX:13:26,397 [SplashThread] INFO  rtmt.control - [INFO]Timezone isUpdateRequired:true
2016-XX-XX XX:13:26,398 [SplashThread] INFO  rtmt.control - [INFO] No tzUpdateFlagfile prompting the user.
2016-XX-XX XX:13:34,638 [SplashThread] INFO  rtmt.control - [INFO] update happened correctly, or user chose not to update tztables.
2016-XX-XX XX:13:34,687 [SplashThread] INFO  rtmt.control - ModuleManager: Loading module file: AST-8.92_003
2016-XX-XX XX:13:34,884 [SplashThread] INFO  rtmt.control - ======>Log On Module :8.92(003)
2016-XX-XX XX:13:34,884 [SplashThread] INFO  rtmt.control - ======>Launch Module begin
2016-XX-XX XX:13:34,887 [SplashThread] INFO  rtmt.control - Control:: received setTraceLevel signal, new level=0
2016-XX-XX XX:13:34,890 [SplashThread] INFO  rtmt.control - Control:: initial trace State for Module is DebugLevel=FATAL
2016-XX-XX XX:13:34,898 [SplashThread] INFO  rtmt.control - In launchModule, gModuleVerStr=8.92(003), gLocalControllerVerStr=8.92(003)
2016-XX-XX XX:13:35,402 [SplashThread] FATAL rtmt.applet -  @ PluginManager.findInstancesOf ==>  : Cache miss!com.cisco.ccm.serviceability.rtmt.plugins.RTMTPlugin
2016-XX-XX XX:13:35,713 [SplashThread] FATAL rtmt.applet -  @ PluginManager.initPlugins ==>  : Searching for new plugins.
2016-XX-XX XX:13:39,780 [SplashThread] FATAL rtmt.applet -  @ SoapServiceHandler:: initServiceList ==>  : Inside initServiceList
2016-XX-XX XX:13:44,589 [SplashThread] INFO  rtmt.control - ======>Launch Module end
2016-XX-XX XX:13:44,589 [SplashThread] INFO  rtmt.control - ======>Start Module begin
2016-XX-XX XX:13:44,589 [SplashThread] INFO  rtmt.control - ======>Start Module end
2016-XX-XX XX:13:44,589 [SplashThread] INFO  rtmt.control - doMeat(): returning true
2016-XX-XX XX:13:44,650 [SplashThread] INFO  rtmt.control - In WaitRunner Thread: Setting frame to visible
2016-XX-XX XX:13:44,650 [SplashThread] INFO  rtmt.control - In WaitRunner Thread:  Before Calling closeRunner
2016-XX-XX XX:13:44,670 [AWT-EventQueue-0] INFO  rtmt.control - In CloseRunner Thread: Calling stopSplash
2016-XX-XX XX:13:44,670 [AWT-EventQueue-0] INFO  rtmt.control - In stopSplash : disposing splashWindow
2016-XX-XX XX:13:44,678 [AWT-EventQueue-0] INFO  rtmt.control - In CloseRunner Thread: After Calling stopSplash
2016-XX-XX XX:13:44,678 [SplashThread] INFO  rtmt.control - In WaitRunner Thread:  Done Calling closeRunner

In case we confirm time zone update RTMT will hang up with the following log messages (last three message continously repeat):

2016-XX-XX XX:48:51,025 [SplashThread] INFO  rtmt.control - doMeat(): after ServerVersionChecker.validMLALogin(): ServerVersionChecker.gStatusCode=200, bValidLogin=true
2016-XX-XX XX:48:51,141 [SplashThread] INFO  rtmt.control - [INFO]Timezone ServerVersion:tzdata2013i
2016-XX-XX XX:48:51,142 [SplashThread] INFO  rtmt.control - [INFO]Timezone ClientVersion:tzdata2015g
2016-XX-XX XX:48:51,142 [SplashThread] INFO  rtmt.control - [INFO]Timezone isUpdateRequired:true
2016-XX-XX XX:48:51,142 [SplashThread] INFO  rtmt.control - [INFO] No tzUpdateFlagfile prompting the user.
2016-XX-XX XX:48:53,155 [SplashThread] INFO  rtmt.control - [INFO] trying to download the tzupdater.jar
2016-XX-XX XX:48:53,156 [SplashThread] ERROR rtmt.control - [ERROR]:TzDataManager:downloadTzUpdaterJar:Ex:/opt/cisco/jrtmt/tzupdater.jar (Permission denied)
2016-XX-XX XX:48:53,157 [SplashThread] FATAL rtmt.control - SystemUtil: IOException: java.io.IOException: Permission denied
2016-XX-XX XX:48:53,157 [SplashThread] ERROR rtmt.control - LOCKING: IOException while getting lock on: /opt/cisco/jrtmt
2016-XX-XX XX:48:53,157 [SplashThread] ERROR rtmt.control - LOCKING: Could not get lock on: /opt/cisco/jrtmt
2016-XX-XX XX:48:53,408 [SplashThread] FATAL rtmt.control - SystemUtil: IOException: java.io.IOException: Permission denied
2016-XX-XX XX:48:53,408 [SplashThread] ERROR rtmt.control - LOCKING: IOException while getting lock on: /opt/cisco/jrtmt
2016-XX-XX XX:48:53,408 [SplashThread] ERROR rtmt.control - LOCKING: Could not get lock on: /opt/cisco/jrtmt

So let us try to run /opt/cisco/jrtmt/JRtmt as root. RTMT terminates automatically while log file shows:

2016-XX-XX XX:55:27,298 [SplashThread] INFO  rtmt.control - doMeat(): after ServerVersionChecker.validMLALogin(): ServerVersionChecker.gStatusCode=200, bValidLogin=true
2016-XX-XX XX:55:27,412 [SplashThread] INFO  rtmt.control - [INFO]Timezone ServerVersion:tzdata2013i
2016-XX-XX XX:55:27,413 [SplashThread] INFO  rtmt.control - [INFO]Timezone ClientVersion:tzdata2015g
2016-XX-XX XX:55:27,413 [SplashThread] INFO  rtmt.control - [INFO]Timezone isUpdateRequired:true
2016-XX-XX XX:55:27,413 [SplashThread] INFO  rtmt.control - [INFO] No tzUpdateFlagfile prompting the user.
2016-XX-XX XX:55:29,663 [SplashThread] INFO  rtmt.control - [INFO] trying to download the tzupdater.jar
2016-XX-XX XX:55:29,869 [SplashThread] INFO  rtmt.control - [INFO]tzupdater.jar downloaded to:/opt/cisco/jrtmt/tzupdater.jar
2016-XX-XX XX:55:29,943 [SplashThread] INFO  rtmt.control - [INFO]:TzDataManager:placeFlagFile: create tzUpdateFlagFile successfully.
2016-XX-XX XX:55:29,944 [SplashThread] INFO  rtmt.control - [INFO] tzupdater.jar downloaded need restart, shutting down.
2016-XX-XX XX:55:29,944 [SplashThread] INFO  rtmt.control - ======>Exit System

After running RTMT once more time zone data version tzdata2013i will be installed and window "Timezone data version mismatch" does not appear anymore:

user@host:~$ java -jar /opt/java/tzupdater-2.0.0-2015a/tzupdater.jar -V
tzupdater version 2.0.0-b03
JRE tzdata version: tzdata2013i
tzupdater tool would update with tzdata version: tzdata2015a

Probably updating RTMT i.e. upgrading CUCM will solve this issue with that "RTMT failed to initialize" message. Of course, you can also return to JRE 1.7 which is easily done under Linux using alternatives:

user@host:~$ update-alternatives --get-selections | grep /opt/java
java                           manual   /opt/java/jre1.8.0_65/bin/java
javaws                         manual   /opt/java/jre1.8.0_65/bin/javaws
mozilla-javaplugin.so          manual   /opt/java/jre1.8.0_65/lib/amd64/libnpjp2.so

root@host:~$ update-alternatives --config java
There are 4 choices for the alternative java (providing /usr/bin/java).

  Selection    Path                                            Priority   Status
------------------------------------------------------------
  0            /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java   1071      auto mode
  1            /opt/java/jre1.7.0_60/bin/java                   110       manual mode
  2            /opt/java/jre1.8.0_25/bin/java                   115       manual mode
* 3            /opt/java/jre1.8.0_65/bin/java                   120       manual mode
  4            /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java   1071      manual mode

The problem yielding the message "RTMT failed to initialize" has been observed and traced on a Debian GNU/Linux 8 system with Cisco Real-Time Monitoring Tool 8.91 and Java Runtime Environment 1.8.0 Update 65. When installing RTMT from CUCM Plugin CcmServRtmtPlugin.bin I changed install directory from /opt/Cisco/Unified-Serviceability/JRtmt (default) to /opt/cisco/jrtmt. RTMT installs Java(TM) SE Runtime Environment (build 1.6.0_26-b03) to /opt/cisco/jrtmt/jre during setup.

user@host:~$ uname -a

Linux debian 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u1 (2015-12-14) x86_64 GNU/Linux

[Deepak Rawat]

Since you are using Linux OS to run RTMT, I hope you have downloaded the correct plugin i.e.,     Cisco Unified Real-Time Monitoring Tool - Linux and not Cisco Unified Real-Time Monitoring Tool - Windows. Also, can you try to install the RTMT version for windows and check if that works or not.

Regards

Deepak

[Joshua Warcop]

Just create the directory and file

sudo mkdir /usr/lib/jvm/java-8-oracle/jre/lib/zi

sudo touch /usr/lib/jvm/java-8-oracle/jre/lib/zi/ZoneInfoMappings

Run RTMT

/opt/Cisco/Unified-Rtmt/JRtmt1151/run.sh

Click "No"