We have a Multi-Services Voice Debug Lookup Tool for different troubleshooting scenarios located here: www.cisco.com/en/US/tech/tk1077/technologies_tech_note09186a0080207ec6.shtml
This document indicates the different combination of traces that are required by TAC in different scenarios so that customers can provide complete information to the CSE at the beginning of the case itself. This saves time and allows us to solve the cases faster as majority (if not all) of the right information is given to us, and we do not need to ask for it again. This document only mentions the traces required from the call manager side of the issue. If other applications such as Attendant Console, UCCX, gateways are involved, we will need the respective traces from there as well.
There is a Cisco Support Forums document that indicates the locations of the traces for different services in call manager. This can be used in case the RTMT tool is not functioning as expected, and when solving the main problem is a priority. This document is located here: supportforums.cisco.com/docs/DOC-16943
NOTE 1: Please make sure that the traces for the services in question (e.g. Cisco Call Manager, Cisco Extension Mobility, Cisco Dirsync) are at the highest level possible during the occurence of issue or when re-creating the issue. Traces not set to the highest level are incomplete.
I have specified a few scenarios below that are commonly faced by the customer. This can be used as a reference.
NOTE 2: The required traces have to be collected from all servers in the cluster to account for SDL communication between the servers for call processing or media resources.
I am attaching a few images here of the traces that are usually collected from the "Collect Files" option in RTMT.
PAGE I
PAGE II
1.CALL ROUTING
a. Detailed Cisco Call Manager Traces
Please make sure that the SIP Call Processing Traces and SIP Stack traces are also checked in case you have any SIP devices/trunks involved in the call flow. SIP Stack traces should be enabled when the call volume is not very high as it takes up more CPU cycles, leaving less available for call processing.
SoftKey traces are also important and can be turned on.
We don't need to change the level of SDL traces, unless explicitly asked for.
b. Event Viewer - Application and Event Viewer - System logs.
c. Timestamps of call failures, Calling and Called numbers.
d. Detailed Call Flow (Devices involved in the call flow & Protocols).
2.HIGH CPU/MEMORY UTILIZATION
Please collect all of the below traces for a particular time period before the problem began till after the problem went away. For instance, if we started observing high CPU or memory usage at 4 P.M. and the problem went away by 5 P.M., then we would collect traces for a time interval of 3:30 P.M. to 5:30 P.M. This time interval can vary from one issue to another, and also based on the customer set up.
a. Detailed Cisco Call Manager traces.
b. Event Viewer - Application and Event Viewer - System logs.
c. Cisco RISDC and Perfmon logs.
d. Cisco AMC service.
e. Cisco Tomcat and Tomcat Security logs.
f. Outputs from CLI: 'show status', 'show process using-most cpu/memory', 'show process load'.
g. Output of 'utils diagnose test'.
3. EXTENSION MOBILITY LOGINS
a. Detailed Cisco Call Manager traces.
b. Cisco Extension Mobility.
c. Cisco Extension Mobility Application.
d. Cisco Tomcat and Tomcat Security logs.
4. AGENT LOGINS - UCCX
a. Detailed Cisco Call Manager traces.
b. Cisco Extension Mobility.
c. Cisco Extension Mobility Application.
d. Cisco Tomcat and Tomcat Security logs.
e. Cisco AXL Web Service.
f. Detailed Cisco CTI Manager logs.
5.LDAP SYNCHRONIZATION/AUTHENTICATION
a. Cisco Dirsync logs.
b. Cisco Tomcat and Tomcat Security logs.
c. Packet Captures on the call manager server when the sync is initialized also help.
6.DRF BACKUP
a. Cisco DRF Master.
b. Cisco DRF Local.
c. Failure logs from the DRF 'Current Status' page.
7. PHONE REGISTRATION
a. Detailed Cisco Call Manager traces.
b. Packet captures from both sides - CUCM server and IP Phone or the switch port to which the IP Phone is connected.
c. Event Viewer - Application and Event Viewer - System logs.
d. Phone Status Messages.
e. Phone Console Logs.
8.IPMA
a. Detailed Cisco Call Manager traces.
b. Detailed Cisco CTI Manager traces.
c. Cisco IP Manager Assistant.
d. JTAPI logs from the client side - PC.
9. CAR/CDR
a. Cisco CAR Scheduler.
b. Cisco CAR Web Agent.
c. Cisco CDR Agent.
d. Cisco CDR Repository Manager
10. SECURITY - Such as IP Phone rejected due to security errors/ TLS Connections not setting up.
a. Detailed Cisco Call Manager traces.
b. Cisco CTL Provider.
c. Cisco Trust Verification Service.
d. Cisco Certificate Authority Proxy Function.
11. MEDIA RESOURCES - MOH not working/ DTMF issues/ No Ringback/ Conference drops/ MTP or XCoder Allocation failures
a. Detailed Cisco Call Manager traces.
b. Cisco IP Voice Media Streaming App.
c. Event Viewer - Application and Event Viewer - System logs.
12.COREDUMP
a. utils core active list.
b. utils core active analyze <coredump name>.
c. Detailed Cisco Call Manager traces for a particular period of time before the core was created.
d. Event Viewer - Application and Event Viewer - System logs.
e. Cisco RISDC and Perfmon Logs for a particular period of time before the core was created.
. f. Outputs of 'show status', 'utils diagnose test', 'show process load', 'show process using-most memory/cpu'
13. GUI DISPLAYS WRONG STATUS INFORMATION
a. Cisco CCMAdmin Web Service.
b. Cisco Common User Interface.
c. Cisco Database Layer Monitor.
d. Cisco Database Notification Service.
e. Cisco RisBean Library.
f. Cisco Tomcat.
14. TOMCAT ISSUES
a. Cisco Tomcat and Tomcat Security logs.
b. Output of 'utils diagnose test'. This may generate a heap dump, which you can then collect using 'file get activelog tomcat/logs/*'
c. Detailed Cisco Call Manager traces.
