cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
10111
Views
190
Helpful
48
Replies

Ask Me Anything- CUCM Troubleshooting: Best Practices for Reading Trace Files

Cisco Moderador
Community Manager
Community Manager

This topic is a chance to discuss more about how to read Cisco Unified Communications trace files. In this session, Cisco Designated VIP Maren Mahoney will answer questions about how to examine, locate and extract information from CUCM trace files. While most questions about trace files are about call setup and troubleshooting, questions about other trace files (DirSync, ILS, CTI Manager, etc.) are welcome and encouraged.

In addition, Maren will provide the best tips and tricks to help you understand and fix different issues such as: what trace fill to poll, how to add router/CUBE debugs to a trace file for telephony troubleshooting and how to follow a single call through multiple trace files. As well as the recommended tools for parsing a trace file.

To participate in this event, please use the Join the Discussion : Cisco Ask the Expertbutton below to ask your questions

Ask questions from Monday 7th to Friday 19th of October, 2019

Featured expert
maren.jpgMaren Mahoney has been in the information system industry for more than 25 years with roles in employee development, technical support & helpdesk administration, network administration, management and engineering, networking courseware development and instruction. She is a Senior Technical Instructor at Sunset Learning Institute and teaches a range of technologies but specializes in Unified Communications. Before joining Sunset Learning Institute (SLI), Maren worked for Cisco Systems as a Network Consulting Engineer. She also worked for several Cisco Reseller Partners in engineering and technical instructor roles. Maren is an official Cisco Certified Systems Instructor (CCSI). She holds a bachelor’s degree in Mathematics and Russian studies and plans one day to gain a Master’s Degree in Mathematics. Maren holds different certifications in Routing, Switching, Data Center and a CCIE in Collaboration (#50569).

Maren was recognized as a Cisco Designated VIP in 2019 for her contributions to the Cisco Community in the IP Telephony category.

Maren might not be able to answer each question due to the volume expected during this event. Remember that you can continue the conversation on the Collaboration, Voice and Video Community

**Helpful votes Encourage Participation! **
Please be sure to rate the Answers to Questions

48 Replies 48

Souley,

Other than the date/time stamp on the file, there's no way I know of to know what dates are contained in what files for the Audit Logs. If you have access to RTMT to view the audit logs, you may have an easier time downloading and viewing the contents of the files.

Maren

dwrice000
Level 1
Level 1

Maren,

 

I recently did a UCCX project and had an issue during testing before going live where calling a Trigger (CTI Route Point) internally worked fine but calling in from the PSTN did not.  The called dropped and CUCM sent a SIP 404 back to the CUBE.  I opened a TAC case on UCCX and they redirected it to the CUCM team. The issue ended up being PT/CSS related but in I wasn't 100% sure why.  I basically just made the change they recommended.  I was hoping you could explain the call flow on an inbound PSTN call through CUCM to an agent answering on UCCX.  In my scenario it was all SIP (ITSP, CUBE and agent phone) except for the UCCX components which are all SCCP.

 

In the customer's existing design, they take in all 10 digits from the CUBE via SIP trunk.  The Inbound CSS on the SIP Trunk had a single partition which contained a series of translation patterns that translated the 10 digit DID to the 4 digit user DNs.  The CSS on the Translation Pattern in this case was the same as the one for the Internal phone that could call successfully call the CTI Route Point.  The "fix" was to add the partition of the CTI Ports created by the UCCX Call Control Group into the Inbound CSS used by the SIP Trunk.  So, my guess is that at some point in the call setup, the SIP Trunk was given a REINVITE or UPDATE to setup a new call from it directly to the DN on the CTI Port.  However, due to the CTI Port's partition not being in the SIP Trunk's Inbound CSS, the call failed.

Interestingly in this case, the DNA showed that dialing the DN of the CTI Route Point from the SIP Trunk inbound should have routed successfully.

 

Thanks!

dwrice000,

You are right that the call destined for UCCX is a redirected call rather than a forwarded call, so the CSS of the inbound trunk comes into play. I don't have that level of detail for UCCX off the top of my head, so let me mock this up in my lab so I can give you a full account of the traffic flow. I'll post a fuller answer for you this weekend.

Maren

Paul Austin
Level 4
Level 4

Hello, One problem we seem to have often is one way voice and call drops on answer, this can be often be very inconsistent especially when the call is eventually handled by UCCX for example. The call drops on answer are usually a codec mismatch and no transcoder resource but the one way voice is quite tricky to find even more so as we are now using CUBE's rather than gateways with ISDN services.

Some great advice would be most welcome.

Paul

 

Yes, no-way audio and especially one-way is the perennial question that plagues us as voice engineers.

Truthfully, I think these issues are very easy to troubleshoot...providing you have access to all of the gear in path of the call. And if you don't, these sorts of problems become very difficult to troubleshoot.

Call drop on pickup, as you mention, is relatively easy to troubleshoot as it is most often a codec negotiation issue or lack of transcoder resources. But you are asking about one-way audio or no-way audio, so let's take a look at that.

The first thing to check (when possible) is if both sides are transmitting. Devices can believe they should not send traffic by one of the devices being told "send-only" or "receive-only" as part of the signaling. To check for this, look at the final SDP sent to the phone (and also out the CUBE towards the provider) for "a=sendonly" or "a=recvonly".

If that is not the case (and it usually isn't) and both sides are transmitting, the next thing to check is which side did not get the audio. For SIP phones look for the 200OK message responding to the BYE message. This will have a pair of lines RTP-RxStat and RTP-TxStat that shows how many RTP packets were sent and received by the endpoint. SCCP phones will send CUCM a StationInit: (<tcp-handle>) ConnectionStatisticsRes message that will contain the same information. This will indicate which side is sending and receiving, and which side is only receiving. Note that the signaling being sent by the SBC to the provider may be as close as you'll get to "that endpoint" in an external call.

This will tell you that a) the call was successfully negotiated meaning that b) the problem is a network issue.

For network issues, most often you are looking at a firewall (or other security device) that is blocking or dumping the RTP traffic.

  • Are the ports for RTP (generally 16384-32767, plus the next odd port for RTCP) open in both directions?
  • Are the firewalls voice-aware (doing payload inspection and dynamically opening/closing ports based on signaling)?
  • Is NAT in use (making the voice-aware firewalls confused)? Focus on the SBC and track the call using the Session-ID or Cisco-GUID.
  • Are you using SIP Inspection with PAT at your CUBE? If so, are the originator (o=) and connection (c=) IP addresses both changed correctly?
  • Are the two devices registered to different CUCM clusters and connecting through two different firewalls?
  • Unlikely, but you never know: Do you have one-way misdirected traffic due to a network change? (routing or VLANs, for instance)

I am not a security expert and usually engage the firewall team to help me gather logs for the transactions. Comparing those to the expected values based on the CUCM trace files (and SBC debug/log) should point the way.

Another signaling issue: Are you using an MTP and is that connection being formed properly?

When I'm trying to track down these problems, I start at the side that is not transmitting and view the signaling beginning with the ingress point and check it to make sure it has all of the correct elements. Then I go to the most likely source of the problem (firewall or SBC) and look at the inbound and outbound signaling packets for that call (based on session-id or Cisco-Guid) for consistency with the ingress signaling.

Another place to look when working with UCCX is tracking the call through CUCM-to-UCCX-queue and then the handoff to the agent. Again, tracking a call using the Session-ID or Cisco-GUID should help you find the signaling related to a single call. I'll be posting a response to a similar question in this thread later today.

And what I've written here is only what is most common. Remember that the Internet is your friend! If you encounter traces/logs that look like the problem but you are unable to interpret the information, please post on the Collaboration Community and we will help.

I hope this has been helpful. I tried to offer pointers for where to look for the issues rather than specifics as this topic is big. Let me know if you have a follow-up question.

Maren

selvakumar475
Level 1
Level 1

How to investigate RTP flow of call through Call manger and SDL traces.

1.By using logs,how to confirm the RTP flow for this call is established between two end points or not.

2.How to identify One way audio issue.

3.How to identify RTP issue from service provider side or Cucm end user side.

 

selvakumar,

I just posted an answer to the one-way/no-way audio issue in the question just before yours. In my response I also addressed how to tell if the endpoints are both sending and receiving RTP packets, and if not whether the problem is on the CUCM side or the provider side.

Please give that a read and let me know if you have additional questions.

Maren

Bruno Rangel
Spotlight
Spotlight

Hello

Which the best/Good Tool to read Trace Files... And which one that you use to Search any issue or information?

Cheers
Bruno Rangel
Please remember to rate helpful responses using the star bellow and identify helpful or correct answers

Bruno,

I use TranslatorX to parse trace files. You can download it at: http://www.translatorx.org/ 

Once you have your trace files, what exactly you are looking for will depend on the type of problem.

When you need to dig into the raw trace files themselves, you will want to use Notepad++: https://notepad-plus-plus.org/ 

I hope I am answering the question you are asking. If not, let me know.

Maren

megahertz56
Level 1
Level 1

HI ,I also got problem related to DX80 .In the DX80 the time showing is wrong .I have also added the NTP Ip address .

The NTP server is working .Is there any utility in which i can diagnose the time issue .

The DX80 is using version 9.5 and call manager using is 11.5

Kindly suggest me

 

megahertz,

DX80's have a known issue with NTP and CUCM. The working fix is to create a Phone NTP Reference set to Unicast, and put that in a special Date Time Group and special Device Pool for your DX80 units.

Give that a try. If that doesn't work, follow up with the Cisco Collaboration Community.

https://community.cisco.com/t5/collaboration-voice-and-video/ct-p/4691-collaboration-voice-video 

Maren

yutaito553410
Level 1
Level 1

I recently built CUCM 12.5.1.11900-146, but immediately after that LowActivePartitionAvailableDiskSpace was reported.

 

Checking with "show status" on the CLI, the usage rate of ActivePatition reached 98%.

 

Is there an effective way to reduce the active partition disk space usage?

 

Kindly suggest me.

yutaito,

If you have just installed CUCM and used the correct OVA, you should not receive a LowActivePartitionAvailableDiskSpace error.

If this was a fresh install, did you use the 1000 size OVA? If so, reinstall using a larger OVA.

If this was an upgrade, does the disk size of the original active partition match the OVA specification for the new version?

Disk usage on the active partition can go to 95% or so and it's really fine. That's the storage for the system and the configuration and shouldn't grow much. Yours sitting at 98% isn't ideal.

Note that log files or trace files etc., are on the common partition. There are techniques and utilities to clear space on that partition. But there is no way I know of (others feel free to jump in!) to reduce the utilization of the Active Partition, nor any way to increase the size of the Active Partition without reinstalling.

Maren

Thank you for reply  Maren.

The CUCM built this time is a freshly installed CUCM using the OVA template 1000.
I would like to reinstall the suggestion, but it is difficult because the user has already started using it.
From what you're replying, I'm assuming it won't be a fundamental solution. Can you tell me the techniques and utilities to clear the partition space?

yutaito,

Here is a link to a very thorough discussion of that topic. The techniques I know about relate to the Common partition where logs files, trace files, and CDRs are stored. From what I know, there is little you can do to clear space on the Active partition.

CUCM - How to Lower Disk Usage on Partitions 

Note that even with the high utilization (although 98% is higher than normal), as long as you don't build more devices than the OVA is designed for you should be okay.

Maren

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: