Ask Me Anything- CUCM Troubleshooting: Best Practices for Reading Trace Files

[Cisco Moderador]

This topic is a chance to discuss more about how to read Cisco Unified Communications trace files. In this session, Cisco Designated VIP Maren Mahoney will answer questions about how to examine, locate and extract information from CUCM trace files. While most questions about trace files are about call setup and troubleshooting, questions about other trace files (DirSync, ILS, CTI Manager, etc.) are welcome and encouraged.

In addition, Maren will provide the best tips and tricks to help you understand and fix different issues such as: what trace fill to poll, how to add router/CUBE debugs to a trace file for telephony troubleshooting and how to follow a single call through multiple trace files. As well as the recommended tools for parsing a trace file.

To participate in this event, please use the button below to ask your questions

Ask questions from Monday 7th to Friday 19th of October, 2019

Featured expert

Maren Mahoney has been in the information system industry for more than 25 years with roles in employee development, technical support & helpdesk administration, network administration, management and engineering, networking courseware development and instruction. She is a Senior Technical Instructor at Sunset Learning Institute and teaches a range of technologies but specializes in Unified Communications. Before joining Sunset Learning Institute (SLI), Maren worked for Cisco Systems as a Network Consulting Engineer. She also worked for several Cisco Reseller Partners in engineering and technical instructor roles. Maren is an official Cisco Certified Systems Instructor (CCSI). She holds a bachelor’s degree in Mathematics and Russian studies and plans one day to gain a Master’s Degree in Mathematics. Maren holds different certifications in Routing, Switching, Data Center and a CCIE in Collaboration (#50569).

Maren was recognized as a Cisco Designated VIP in 2019 for her contributions to the Cisco Community in the IP Telephony category.

Maren might not be able to answer each question due to the volume expected during this event. Remember that you can continue the conversation on the Collaboration, Voice and Video Community

Find further events on https://community.cisco.com/t5/custom/page/page-id/Events?categoryId=technology-support

**Helpful votes Encourage Participation! **
Please be sure to rate the Answers to Questions

[maxmeisel]

Hi Maren,

to identify which logs I have to gather from CUCM nodes I use the CDR information for "Orig Node Id" and "Dest Node Id".

To know which server is which I query the DB for processnode. Unfortunately the information from DB does not exactly match the information from CDR information.

Here is an example output of the SQL query of a mega cluster, the "== [digit]" information was manually added by me:

admin:run sql select name,description,nodeid from processnode
name description nodeid
================== ============== ======
EnterpriseWideData 1
node145.server.example CUP Publisher 12
node162.server.example 10.10.10.62 17 == 10
node129.server.example 10.0.10.29 20 == 13
node116.server.example 10.0.10.16 3 == 2
node146.server.example CUP Subscriber 15
node161.server.example 10.10.10.61 10 == 9
node118.server.example 10.0.10.18 5 == 4
node218.server.example 10.10.10.18 7 == 6
node246.server.example CUP Subscriber 13
node141.server.example 10.0.10.41 22 == 15
node140.server.example 10.0.10.40 21 == 14
node115.server.example 10.10.10.15 25 == 18
node114.server.example 10.10.10.14 24 == 17
node216.server.example 10.10.10.16 2 == 1
node124.server.example 10.10.10.24 26 == 19
node142.server.example 10.0.10.42 19 == 12
node245.server.example CUP Subscriber 14
node117.server.example 10.0.10.17 4 == 3
node113.server.example 10.10.10.13 23 == 16
node217.server.example 10.10.10.17 6 == 5
node128.server.example 10.0.10.28 18 == 11

Basically from the DB information I have to substract 1, to get the information I have in CDR to identify the server that participated in call processing. With the mega cluster I noticed that this somtetimes have to be -7, I do see a pattern but I don't understand it.

With CUCM version 11 the server folder name got additional information, so I do see after collecting the files if the node ID matches the information on CDR, e.g. node218.server.example_6

I would like to understand this better, especially why there is an offset in DB to CDR information of -1 or -7, I would understand if there is an offset by 1 as of if you start counting with 0 or 1, but the -7 I don't understand especially as it is mixed with -1 as far as I can tell.

Why are there offsets?

Is there even a more simple way to identify the correct server(s)?

Thanks,

Max

[Maren Mahoney]

Max,

Your question made my math brain say "Squee!" to figure out the pattern. I'm guessing here (because I can find no specific documentation on this), but it fits:

Did you by chance build and later decommission two CUCM servers? And possibly an IMP publisher that you decommissioned and rebuilt, and later an IMP subscriber that was built and decommissioned?

According to the Data Dictionary, the SQL-nodeid is the order in which any node in the cluster was built at any time. If I am right, the Orig Node Id/Dest Node Id is the order in which the CUCM servers were built at any time.

The first nine (CUCM Pub + 8 Subs) would be the SQL-nodeid minus one pattern. After CUCM nodes in the cluster were built and decommissioned and IMP nodes were built and some decommissioned, the numbering picks up where it "left off".

Here's what I mean:

Does that fit the evolution of your cluster?

Maren

[maxmeisel]

Hello Maren,

Chapeau! I totally missed to sort the output, but your finding is great, it makes a lot of sense to me. Unfortunately I can't fill in the full information gabs, but I can confirm the two decomissioned CUCM nodes and also there was a decomissioned IM&P node as well (should be the second one), the last missing node I couldn't dig up any information, but from your explanation it must have been an IM&P node. So basically with an historical grown cluster it might be harder to get the correct node association the way I use.

So first of all thanks a lot for your information! :)

Do you now a better way to match the CDR information (nodeID) to the according nodes?

Cheers,

Max

[Maren Mahoney]

Max,

I don't know of an easier way to associate the SQL-nodeid with the CDR-nodeid other than to do some painful thing (like you've already done with this cluster) and keep track. It might be worthwhile to post the question on the Unified Communications Infrastructure community thread to see if someone knows a trick that I do not. I'd be interested to know the answer as well!

Maren

[Maren Mahoney]

One last thing: Per the Data Dictionary, CUCM issues a SQL-nodeid once the name of the new server is configured in the CUCM Publisher. That SQL-nodeid is never reused. So the 'missing' gap may have been something as simple as configuring a new server on the System > Server page, realizing you misspelled it, deleting it, and re-adding it with the correct name. The SQL-nodeid would tick up twice during that process.

Maren

[SBCounty_TechTree]

Hello, we are running CM 11.0.1.23900-5.  Is there a reason why 8841's built with two lines and max number of calls = 2 and the busy trigger = 2 is cannot transfer from the 2nd line?  Example- I call my main line 80747, then I call the main from another phone and it rolls to my 2nd 4680747. I answer that call and it place main line on hold and accepts new call on 2nd line.  If I try to transfer the call from 4680747 I get the message "Unable to transfer.  Maximum number of calls have been reached"  But if I set my max number of calls to 2 and busy trigger to 1, the transfer works.  Same with 2/1, 3/2, 4/1, 4/2.  It is only on 2/2 that it does not work.

Thank you for your time.

[Paul Austin]

That's correct. The max number if calls is really how many voice channels you can have activated. Each call is 1 channel so if you have max calls =2 then that's the max number of call legs you can handle.

[SBCounty_TechTree]

Thank you for the response.

We understand that part of it, the only thing we're having a hard time wrapping our heads around is when the busy trigger is set to 2 the call that rolls over to the 2nd line is unable to transfer .  When it is set to one, transfers work fine, or of course when the max calls are increased. Only the few 2/2 phones that our TSS group built are having these issues. It was a weird one to troubleshoot.

[Maren Mahoney]

SBCounty_TechTree,

So that I understand: You have two lines on an 8841 phone, 80747 and 4680747. The first line, 80747, is set to Max Calls=2 and Busy Trigger=1 with Call Forward Busy set to roll to 4680747. Is that part correct?

And for 4680747, you have Max Calls=2 and Busy Trigger=2 but if you have one call active it won't use the second channel for transfer? It is indeed a strange behavior, especially if it is restricted to one phone model.

Questions:

Does the second number have Single Number Reach Enabled? MLPP enabled? Call recording (manual or automatic) enabled? Is it a shared line appearance?

In any of those cases, an extra channel is required to allow for those features.

I'd also be interested to know what happens if you set the Max/BusyTrigger to 3/3 if the same behavior occurs.

If none of the above questions applies, and if the 3/3 setting does not exhibit the same behavior, I'd have to look at a trace of the call rolling to the second line and then the transfer attempt to see if we can figure out what's going on.

Another thing to try would be to upgrade the firmware on one phone to see if the behavior changes. I don't see a bug associated with this behavior, but stranger things have happened.

Please let me know what you find out! This is an interesting puzzle.

Maren

P.S.: Is this San Bernardino County in CA?

[SBCounty_TechTree]

Yes this is San Bernardino CA. I was in your classes when you came here. I was happy to see your name on the AMA. Gilbert Sanchez says hello by the way.



Ok, so I've been testing this on my desk phone which is the 80747 number and its rollover 4780747. Gilbert and I have been going through this today and we have it now, or better to say I have it now. I was under the assumption that when the main line is called and another call comes in, that it automatically sends that 2nd call to the 2nd line. I had the explanation that the busy trigger is stacking the calls on the main line and if 2/2 is set and if 2 calls come in there is no channel left to transfer the call, hence the max calls reached error. So I was told that since most of the County of San Bernardino was on Nortel before and that was common practice for the need of the second line. But now in Cisco we can virtually stack up to 200 calls on one line and there is really no need for that 2nd line anymore in most cases. So there is talk of us doing away with that 2nd line as it creates confusion and unnecessary trouble tickets and just stack the calls. This site was the Guinee pig and there needs to be more education to the staff.



I was always taught to assume nothing due to the "making an ass of u and me" and I fell into it. Thank you for taking the time out to answer this question. I hope we get to see you teach us again real soon.



Take care,


[http://www.sbcounty.gov/main/images/sbcountylogo.gif]


Nathaniel Ramirez
Information Services Department
System Analyst II
-----------------------------------------------------------------------------------------------------
Phone: 909-522-2091
Office: 909-388-0747
Nathaniel.ramirez@isd.sbcounty.gov
www.SBCounty.gov

Our job is to create a county in which those who reside and invest can prosper and achieve well-being.

Confidentiality Notice: This communication contains legal privileged and confidential information sent solely for the use of the intended recipient. If you are not the intended recipient of this communication you are not authorized to use it in any manner, except to immediately destroy it and notify the sender.

[Maren Mahoney]

Nathaniel,

Hello to you and Gilbert and the rest of the folks at SB-ISD! I hope I get to come out to teach another class for you folks some day. You are among my favorite groups to teach.

I was wondering why you had the rollover line but I'm always reluctant to say, "Why are you doing that?" to folks who ask questions as they usually have a good reason. Yes, I would encourage you to get rid of the second line and increase the max-calls/busy-trigger for the main line. Some things to consider when you do:

• The max calls is per device, not just per line. So if you have four lines on one phone, you are still limited to 200 total channels across the four lines.
• If a line is a shared line appearance, adjust the busy-trigger so that more than one transfer (or other call activity) can go on at the same time. Basically, keep in mind the behavior across the set of phones on which the line will appear.
• Certain phone models support fewer than 200 calls, but those tend to be either older phones or low-end phones (think kitchen or lobby type phones).
• I encourage you to train users on how to manage multiple calls, especially on lines that have more than four or so channels. (Also investigate Join!)

As always, please feel free to reach out if you have more questions. Have a great day!

Maren

[SBCounty_TechTree]

The rollover has to do with "old dogs and new tricks." Since SB County was converting from a Nortel system and that was "How things were done," the confusion of the masses was "where is my 2nd line?" So we continue to place it there to appease these people.



Thank you for your time, and I hope you take care!



Thank you,

[Mohamed Ali D.A]

Hi,

i would like to monitor or collected the logs about Confiuration changes when, what and by Who?

Is ther any way to save the logs outside the CUCM?

[Maren Mahoney]

Mohamed,

What you are asking about is called the Audit Log. The Audit Log is collected by default and can be accessed via RTMT.

Here is information about the Audit Logs, what is collected, and what (few) configuration parameters there are for the Audit Logs (It's for v11.5, but applies to all recent versions of CUCM including v12.x):
Cisco Unified Serviceability Administration Guide - Chapter: Audit Logs 

To view the Audit Logs, access RTMT and navigate to: Trace and Log Central > Audit Logs

I hope this helps. Let me know if you have additional questions.

Maren

[NetGhost33]

Good morning,

I establish a point-to-point call between the TelePresence SX80 and the TelePresence C20 or C40, they can't able to see us nor hear us, but, when the second call is added we are able to establish a complete call with the first one. Why is this happening?    

[NetGhost33]

Good morning,

I establish a point-to-point call between the TelePresence SX80 and the TelePresence C20 or C40, they can't able to see us nor hear us, but, when the second call is added we are able to establish a complete call with the first one. Why is this happening?