CUCM 11.5 - RTMT (11.5) Won't Show Calls - Session Trace > Real Time Data

[voip7372]

I have a case open with Cisco TAC as well, but so far there's no resolution.  In the meantime, I thought I'd ask about it here as well.

I recently upgraded to CUCM 11.5 (from 10.5).  Everything is working well except for the fact that I can't see our calls in RTMT 11.5 under Session Trace > Real Time Data.  Occasionally a few scattered calls will appear, but I'v made many test calls over and over and I never see my various test calls (completed calls to various devices and entities - outside calls, inside calls, etc).  

All servers have been rebooted and I verified all the appropriate services were running.  Cisco already had me restart some services and then also reboot the servers.  Didn't help.  Well, actually...after restarting Cisco AMC Service and Cisco RTMT Reporter Servlet, it worked but the next day it wasn't working again, so Cisco's next suggestion was to reboot the servers (he said he noticed the same problem with RTMT and CUCM 11.5 in his lab and after rebooting, it was working...but the reboot didn't help in my case).

I downloaded RTMT from the CUCM page for the plugins.  I installed RTMT on Windows 10 with UAC (user account control) turned OFF and I ran the installer as administrator.  (I did the same thing with Windows 7 and have the same problem).  I open and run RTMT as administrator as well. 

My computer is in the same time zone as the CUCM server I'm connecting to (but I've also tried looking for the calls a few hours ahead and back just to be sure it wasn't a time zone issue, but it's not).

I can see the calls Trace & Log Central > Collect Files.  I download the files onto my computer and I can open those and see the raw data for the calls I made.  I can also see the calls if I go to Trace & Log Central > Real Time Trace > View Real Time Data.  I can also view the calls Trace & Log Central > Remote Browse.  So the call data really is there, but for some reason RTMT just won't show it to me in Session Trace > Real Time Data like it did before we upgraded.  Like I said, occasionally some will WILL show up there, but only a very small amount and it's usually not the test calls I made or any other calls I know were made during busy periods of the day.  When something does show up, I can never see the SIP details if I open the call info and then click on one of the entries I see in the call ladder (for example, to see the details of the Invite message)...there's no detail there.  But again, I CAN see all this detail if I manually download the logs using the methods I mentioned previously. 

Any ideas?

[Deepak Mehta]

You may want to check ntp status on each one and make sure all have correct time and  are in sync with pub.

Also check dbreplication status to see if any error's.

[voip7372]

I just checked and our Pub is synced to our NTP server and the two Subs are synced to the Pub.

Also, dbreplication is good.

Info below.  (I hid our real IP addresses for the purpose of posting here)

PUBLISHER:
admin:utils ntp status
ntpd (pid 8470) is running...

remote refid st t when poll reach delay offset jitter
==============================================================================
*XXX.XXX.132.195 128.105.38.11 3 u 552 1024 377 0.915 0.088 0.220

synchronised to NTP server (XXX.XXX.132.195) at stratum 4
time correct to within 86 ms
polling server every 1024 s

Current time in UTC is : Fri Jun 24 19:08:33 UTC 2016
Current time in America/New_York is : Fri Jun 24 15:08:33 EDT 2016

SUBSCRIBER 1
admin:utils ntp status
ntpd (pid 8566) is running...

remote refid st t when poll reach delay offset jitter
==============================================================================
*XXX.XXX.5.37 XXX.XXX.132.195 4 u 324 1024 377 0.388 -0.271 0.246

synchronised to NTP server (XXX.XXX.5.37) at stratum 5
time correct to within 102 ms
polling server every 1024 s

Current time in UTC is : Fri Jun 24 19:08:36 UTC 2016
Current time in America/New_York is : Fri Jun 24 15:08:36 EDT 2016

SUBSCRIBER 2
admin:utils ntp status
ntpd (pid 474) is running...

remote refid st t when poll reach delay offset jitter
==============================================================================
*XXX.XXX.5.37 XXX.XXX.132.195 4 u 856 1024 377 18.885 -0.491 0.592

synchronised to NTP server (XXX.XXX.5.37) at stratum 5
time correct to within 147 ms
polling server every 1024 s

Current time in UTC is : Fri Jun 24 19:08:38 UTC 2016
Current time in America/Chicago is : Fri Jun 24 14:08:38 CDT 2016

PUBLISHER
admin:utils dbreplication runtimestate

Server Time: Fri Jun 24 15:06:20 EDT 2016

Cluster Replication State: BROADCAST SYNC ended at: 2016-06-18-11-05
Sync Result: SYNC COMPLETED on 700 tables out of 700
Sync Status: All Tables are in sync
Use CLI to see detail: 'file view activelog cm/trace/dbl/20160618_104458_dbl_repl_output_Broadcast.log'

DB Version: ccm11_5_1_10000_6

Repltimeout set to: 300s
PROCESS option set to: 1

Cluster Detailed View from US-CUCM-PUB (3 Servers):

PING DB/RPC/ REPL. Replication REPLICATION SETUP
SERVER-NAME IP ADDRESS (msec) DbMon? QUEUE Group ID (RTMT) & Details
----------- ---------- ------ ------- ----- ----------- ------------------
US-CUCM-PUB XXX.XXX.5.37 0.037 Y/Y/Y 0 (g_2) (2) Setup Completed
US-CUCM-SUB1 XXX.XXX.5.38 0.255 Y/Y/Y 0 (g_4) (2) Setup Completed
US-CUCM-SUB2 XXX.XXX.252.180 19.172 Y/Y/Y 0 (g_3) (2) Setup Completed

[voip7372]

By the way, I can ALSO see the call if I go to the CDR Analysis and Reporting web page for our server and then search there:

CDR > Search > By User/Phone Number/SIP URL

The call appears there and looks fine, but I never see it in RTMT.

[Deepak Mehta]

You are running way ahead my friend, lol.. i am  till stuggling with 9.1.2  and 10.5

Most of the time i have seen issue with NTP sync or timezone not matching for call search.

You have confirmed you can see some traces so i assume SIP session traces are enabled All looks good to me.Probably TAC should have a better answer or other experts can comment .thanks

[voip7372]

Thanks.  I'm hoping I didn't just discover a new bug because I really don't want to be updating CUCM again any time soon :-).  Hopefully it's just something with RTMT or a Windows or Java settings.  I don't know...it worked fine just two weeks ago on CUCM 10.5.2 before I upgraded.

[Casey Arnold]

Oh Yay!  Less than 36 hours since the upgrade and already finding bugs. 

Oh well.....

Thanks for the prompt response.  At least I will quit searching for an answer.

Casey Arnold

[voip7372]

I know the feeling.  I just saw this new version available and I'm trying to see if it contains the fix for RTMT and the personal address book error.  

Description:	US Export Restricted - full encryption capabilities (non-bootable) - For upgrades from 11.x only. Upgrades from 10.x or earlier versions must be requested via PUT (www.cisco.com/upgrade) or purchased to obtain necessary licenses.
Release:	11.5(1)SU1
Release Date:	18/Aug/2016
File Name:	UCSInstall_UCOS_11.5.1.11900-26.sgn.iso
Size:	5169.77 MB (5420900352 bytes)

[Tobias Bachetta]

Hi everyone,

I also ran into this bug and updated to 11900-26. I can see the calls again, but they are not showing the "Detailed Sip Message", can anyone confirm this?

"Sip Call Processing Trace" is enabled on all servers. And it worked fine with 11.0, so there is still a bug left?

[voip7372]

I asked the Cisco TAC guy I was working with if the latest release you mentioned contains the RTMT fix and he said it did NOT contain the fix, so I haven't updated to that version.  The other Cisco TAC person I worked with on another bug (PD Error when searching the personal address book) said that newer release also does NOT contain a fix for the PD error bug.  

I'm still waiting for a version that has the fix for BOTH of these bugs I found with the 11.5.1.10006-1 release...

- No calls appear in RTMT real time session trace (but I can use the 'collect files' process and TranslatorX to work around this for now, but it's inconvenient and a pain in the rear).

- PD error when searching the personal address book.

The fact that the PD Error bug has appeared in a few other earlier releases of CUCM has me very disappointed.  I mean really...thanks for bringing it back in a new release.  Did someone copy/paste the code for that part from an earlier release that has the bug?  

Anyway, I'm frustrated and disappointed.  It literally takes all day (on a weekend) to properly update our servers and I spent all that time to update to 11.5 only to find out one of our executives that uses the personal address book all the time now has a bug that prevents him from using the address book search and I have no convenient way to QUICKLY trace calls thanks tot he RTMT bug.  I'm not a happy camper at this point.  I'm too busy to be updating our servers over and over to fix bugs.  Cisco did provide an engineering special release for me to fix the PD error bug, but the RTMT bug remains.  

[John Watkins]

For those interested this is BugID CSCvb09481

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvb09481

[voip7372]

FYI - Cisco TAC did confirm this is a bug and multiple customers have the same problem with RTMT.   There's also a Personal Address Book search error in this release.  If you search the personal address book for a name and see the 'Cisco PD Error', that's why you're seeing the error...it's a bug in version 11.5.1.10006-1

Also, I just noticed that version 11.5.1.11900-26 is available for download now.

[mtuzun]

I can confirm that this bug is still there in version 11.5.1.12900-21, even though it is listed as one of the fixed releases.

[voip7372]

I'm on version 11.5.1.12900-21 now and it works.  If you haven't uninstalled and reinstalled RTMT using the file you get from the upgraded CUCM server, you should do that and it should be ok then.

[mtuzun]

I indeed uninstalled and reinstalled the RTMT, but the issue is still there. Calls show up when collected using trace collection, but real-time date do not show them. By the way, this seems to affect only calls processed by one of our subscriber servers. NTP is good, CDR is enabled, dbreplication is also good. I may have to report to TAC.