This topic is a chance to discuss more about how to read Cisco Unified Communications trace files. In this session, Cisco Designated VIP Maren Mahoney will answer questions about how to examine, locate and extract information from CUCM trace files. While most questions about trace files are about call setup and troubleshooting, questions about other trace files (DirSync, ILS, CTI Manager, etc.) are welcome and encouraged.
In addition, Maren will provide the best tips and tricks to help you understand and fix different issues such as: what trace fill to poll, how to add router/CUBE debugs to a trace file for telephony troubleshooting and how to follow a single call through multiple trace files. As well as the recommended tools for parsing a trace file.
To participate in this event, please use the button below to ask your questions
Ask questions from Monday 7th to Friday 19th of October, 2019
Maren was recognized as a Cisco Designated VIP in 2019 for her contributions to the Cisco Community in the IP Telephony category.
**Helpful votes Encourage Participation! **
Please be sure to rate the Answers to Questions
I use the Cisco 3905, 7841, 7861, and 8851 models. When I access them via web page I have console logs and core dumps and debugs. Where does one start to begin troubleshooting a phone having issues ? What are some items to look for in these logs that may hint the phone is having some issues ?
Before poking around in phone logs, if you have a phone that won't boot I would start with tracing the phone boot process, most of which can be verified on the phone itself:
But if you are asking about the phone logs, then I would imagine that you have checked out the items that list already. (But it's a good list for folks that are new.)
Things to check in the phone console log:
I hope this was helpful. I really like your question and I may do a Tech Talk on it later on.
What are the best practices you generally recommend for tracing a call in the call manager logs and specific scenarios like call drops or media resources issues?
Also, incase of multiple server hops like call traversing through multiple call managers, what is the best way to track a call between different cluster?
Let me take the second part of your question first. @Anthony Holloway posted links in his reply to two videos by Patrick Kinane that show how to track a call between two phones on the same node, and then between two phones on two different nodes in the same cluster. (And I'll include the links below as well.) Patrick's technique for tracking a call is the best I've ever seen and taught me a thing or two.
As for tracking a call between clusters, start by identifying the Call-ID and Session-ID in the outbound INVITE message from the first cluster. Use that information to locate the same INVITE Message as it arrives inbound on the second cluster. From there, you can use the techniques Patrick shows to track the call through the trace files in the second cluster.
Of course in a production environment a single call can span more than one file. In the case where tracing a call spans more than one trace file, open both of those files (or, heaven forbid, all three files) and use the "Find All in All Open Documents" feature in Notepad++ to locate the relevant lines.
Here are the links to the two videos:
Now for the first part of your question. For dropped calls, it will depend on the reason for the drop.
A 404 Not Found error means, of course, that there is either a digit manipulation issue or a permissions issue (PT/CSS in the case of CUCM). For those errors, I don't generally use trace files to troubleshoot. Rather, I use the Dialed Number Analyzer to see what CUCM is doing with the call internally, and the Dialed Number Analyzer for CUBE TAC tool to analyze my CUBE or router's config.
A 503 Service Unavailable error is the second most common thing you'll see in SIP calls. For those you'll occasionally have an IP address misconfigured in CUCM or a dial-peer, but most often this will indicate codec negotiation problem. To identify (or at least get a first approximation of) the cause of the error, look for the Q.850 cause code in the error message. A "47" is "Resource unavailable – unspecified", but generally indicates a codec problem. To start troubleshooting that: If it is an internal call, look at the region settings. If that doesn't reveal the problem, dig into the trace. In the trace file, watch the SDP exchange during call setup to get an idea of what the devices' capabilities are and what they are requesting. Hopefully the answer will pop out. If not...
I did a Tech Talk for Sunset Learning that walked through a trace file showing how CUCM processes a call that requires a media resource, specifically a conference bridge. It shows CUCM processing region/codec information, the process for allocating the media resource, and the process for having the call legs redirect through the media resource. Knowing what is "supposed" to happen will help you identify what isn't working. That video is currently in a student-only video library. I will get it moved to the public library and post it in another response to this message once it is available.
Please feel free to ask additional questions. And thanks for this one!
Here is a link to the video I did on Media Resources. It show DSPs registering, and then follows a call allocating a transcoder. It will show you what to look for in a normal call. Comparing that to the call you are troubleshooting should help you identify where the media resources allocation is failing.
If you have more questions or would like a further discussion of this, let me know.
Wow! As a fellow VIP I commend you for putting yourself out there like this. Good luck! I'll be sure to follow the discussion.
Thanks, Anthony. You are right that this does not feel like a small thing. Your good wishes are appreciated!
I want to ask about most painful issue an admin can face (at least for me) and that is voice quality issues within cluster and for PSTN calls. I know there are many questions found and there are many different answers to this question, but I would like to know what tools do you recommend and approach you recommend to trace logs for these issues and tackle them effectively.
I had faced this issue couple of times and they turned out to be network related issues; but it becomes quite challenging to prove that the issue needs to be looked upon from network perspective.
I'm sorry for the delay in answering your question, but I wanted to ponder how to respond. As you pointed out, a lot of community discussions, blog posts, and white papers have been written on this topic so I didn't know what I could add.
In general, there isn't much in the trace files or phone console logs that can identify call quality issues since those files are primarily internal-processing related rather than payload-delivery related. That said, Call Management Records (CMRs) can be analyzed for packet-loss and delay statistics. If you can show that a particular piece of gear or a single site is common to a bunch of poor-quality calls that can help you convince the network folks that the problem may be theirs.
For phone/headset problems, Wireshark is my go-to tool for analyzing the traffic flowing in and out of the phone. (Bluetooth is a little harder, of course.) If you think the problem is QoS-related, an analysis tool like NetFlow can help prove that. But that usually takes buy-in from the network team in the first place.
In the end, I don't know that I have any special magic to share on this question so I doubt my answer very satisfying. Your question reminds me of discussions surrounding faxing over IP - everyone hates it, it's a pain to troubleshoot, and no two problems are the same.
You can see a list of the Application Audit Logs using the CLI command:
file list activelog audit/AuditApp/*
Once you have the list of files, you can view one with:
file view activelog audit/AuditApp/<name-of-file>
You can also set up transfer of these files to another server with the information here:
I hope this answers the question you are asking. Let me know if there are more answers I can provide.
Awesome thanks Maren i really appreciated it. I was able to see the audit logs.
Can I see the audit logs from a specific date and time ? for instance October 1st to October 3rd ?