RTMT- Realtime trace logs

[Mohaned]

I have an issue with the RTMT not showing the call logs for the publisher server under Voice/Video >> Session trace logs view >> RealTime data.

The logs are working fine for the subscribers' servers.

Do you have any recommendations on what I can check to have the calls show for the publisher?

[Maren Mahoney]

Is the Cisco CallManager Service running on your Publisher (which is unusual except in very small clusters)? If not, there will be no call logs because only Call Processing Nodes (nodes running the CallManager service) process calls.

Maren

[Mohaned]

Yes, the Cisco CallManager Service is running on the Publisher. the RTMT is showing the number of calls on the Publisher under call activity but when I tried to get the call trace it's not showing the calls on the Pub.
Is there anything that could be the reason for this behavior?

[Maren Mahoney]

Okay, so as you know, the order of the call processing nodes in the CM Group determines which node will process incoming calls. As a test, find a device that is registered to the Publisher (you can see that on the Device > Phone listing) and make a call to it. Then go into RTMT into Real Time Data and look for the call. Then use the button at the bottom to look at the diagram for it. Does it actually use the Publisher for call processing?

Another question: Do the other nodes show call data? If not, it may be the Trace settings for the CallManager service that are causing the system to not capture the data. This is unlikely unless you are using an older version of CUCM, but it is possible.

Maren

[Mohaned]

Thanks for your information, As per my attached screenshots the calls used the Publisher (10.73.100.10) as the call processing but they are not showing in the real-time data, however, the calls activity shows the calls using the Publisher.

The sub-servers show the call data correctly.

I'm using CUCM 12.5, please check the attached screenshot for more clarification.

[Maren Mahoney]

Okay. So if I understand you correctly, RTMT call activity is showing the the Pub is processing calls but the Call Flow via Real Time Data only shows calls processed by Subscribers. For example, if you place (and then end) a call to the Publisher you are saying that such a call does not appear in Real Time Data. Do I understand the underlying issue correctly?

If that is the case, we will need to look at a trace file to see what is going on. Please place a call between two endpoints that are both registered to the Publisher, and then pull the trace files for all nodes for the one minute during which that call took place. (Use the Query Wizard and specify the one minute timeframe and also filter on the calling party number in the SIP INVITE to minimize the number of files gathered.)

Please attach the trace file(s) along with the calling number/MAC and the called number/MAC and the time of the call.

This is an interesting problem!

Maren

[Mohaned]

Yes, you're correct.

Please find the attached trace files during the test call between two endpoints that are both registered to the Publisher, the Pub server IP 10.73.100.10 (cucm-pot.egyptpost.org)

calling number/MAC (7663 / 2C73A0024B2E)
called number/MAC (7667 / 2C73A0024B1B)

Call time: 14:40

please let me know if any additional information is needed.

[Maren Mahoney]

What I see in the trace that you gathered is that you set the time filter and text filter (the 7667 filter) accurately, but that there were still no SIP call processing messages in any of the SDL files for any server - not just the Publisher.

I opened the Call Logs in RTMT (use "Open from Local Disk" to do this) and it shows the call referenced:

When I pull the diagram for the call and click on Message Details for the INVITE I see the following:

CUCM generates Call Logs regardless of the Trace level, and the Call Logs are what populates the Real Time Data. But the existence of Call Logs does not imply that CUCM is also retaining the call setup messages - and in your case there are no call setup messages saved on any server.. What this tells me is that the trace levels are set incorrectly for the CallManager service. If I had to guess, they are set at "Error".

In the second pic above I have drawn a box around the instructions where you see "There are no SIP messages to display". Take a look at #2 there to see to what level you should set your CallManager service trace and where/how to do that.

Let me know if you have questions.

Maren

[Mohaned]

Many thanks, Maren for your help.

I have checked trace levels on all nodes and made sure to have it set to "Detailed", I have restarted the Cisco Trace Collection Servlet and Cisco Trace Collection Service on the nodes.

I made test calls and am still unable to see the SIP message details for the calls on the Sub servers, any recommendations on what else can be checked?

For the calls made on the Pub server, I'm still unable to see them under session trace logs view >> Real Time Data, so what can I check to make it visible on the Real Time Data?

[Maren Mahoney]

If your trace levels for the Cisco CallManager service are set to Detailed and all of the call processing-related checkboxes are checked in the service configuration, then you have a serious problem because the trace files do not contain the messages.

The "SIP Message Details" rely on the existence of the SIP messages in the trace files, and as they don't appear in the ones you provided that would explain why you can't see them via RTMT.

As for not seeing calls running through the Publisher in Real Time Data, that list is populated by the Call Logs and not the SIP messages, and the Call Logs are collected regardless of the status of the trace level. And as I saw the two calls flowing through the Publisher in the files you provided they should be appearing in your Real Time Data as well. However, if your PC and the CUCM are in different timezones, I have seen Real Time Data misbehave. So if that is the case, it might be worthwhile to play with the timezone setting in the Real Time Data to see if you can find the data you need. 

The first thing I'd ask are screen grabs of the configuration of the CallManager service trace showing both the SDI page and the SDL page to make sure there isn't something weird we are missing.

Nothing else comes to mind after that. If everything checks out, you may want to reach out to TAC.

Maren

[Mohaned]

I confirmed that the time zone is the same on the PC and RTMT, I will check with the TAC why the Publisher calls not showing in the Real Time Data.

Please find the attached screenshots from the CallManager service trace showing both the SDI page and the SDL page

[Maren Mahoney]

That all looks fine. I'll be interested to know what TAC says.

One last thing: I'm a big believer in "When in doot, reboot!" (Which I say in my best Canadian accent....) If you have a maintenance window coming up, a cluster reboot can't hurt. But you've already restarted any services I would think are involved, so a reboot would be a "in case there are weird gremlins" action. 

Maren

[Mohaned]

Appreciate your help!

Cisco TAC has checked internally and observed that we are hitting on bug CSCvq37426

They provided the below workarounds:
1) Use another timezone with the same offset, that doesn't use DST.

2) Upgrade to a version with the fix

I used another timezone with the same offset, that doesn't use DST and I was able to see the calls on the Pub server in Real Time Data and was able to see the detailed SIP messages.

[Maren Mahoney]

Thank you for sharing your solution as breadcrumbs for folks who follow! -- Maren