cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
20518
Views
124
Helpful
16
Comments
pkinane
Cisco Employee
Cisco Employee

 

-------------

Please rate helpful content (i.e. videos, documents, comments) so quality content shows at the top of people's search results. Also, please select the correct answer(s) if any comment(s) answers your question otherwise the question remains on the support forums as unanswered.

-------------

 

 

 


Introduction:

Everyone has some overlap in the way they read traces; however, there are tips and tricks that each person can bring to the table. With that being said, this series displays my way of reading the traces.


Things about the series

This is not meant to be all inclusive. This is meant to show some of the things I do when reading CallManager service traces. The goal of this is to help people be more comfortable with looking at traces. Then they can practice on their own, or try to find things on their own when they have an issue.

 

I will not cover much with SCCP phones.

 

This will be geared more towards the interleaved SDI/SDL traces (version 9 and higher... if you are on version 8, there will be a lot that still applies; however, you should think about upgrading)


Tools I use

RTMT

extract now

This 2.5 minute video discusses how I setup extract now.

notepad ++

translator X (only when the occasional need arises, and this should be used once you are a little more familiar with traces)

CDR cause codes

Convert to IP address

Call signaling diagrams

SIP

SCCP, MGCP, and Gatekeeper

H.323

Version 10 SRND for media resources


Things to understand

Every call has a few things in common:

 

1: The call is extended to CUCM

2: CUCM will do digit analysis

3: If a pattern or DN is matched, CUCM will try to select a device (trunk, gateway, route point, phone (via Directory Number and partition combination), etc...)

4: If CUCM successfully selects a device, the call will be extended to the device

 

You also want to understand a little about the processes.

 

The process listed on the right is talking to the process listed on the left. Here we see the SIPStationCdfc process talking to the SIPCdpc process.

 

|SIPCdpc(3,100,83,3)              |SIPStationCdfc(3,100,75,2)

 

There are two major types of processes in the SDL device layer:

 

1: Edge Processes (Talk with the outside world from the perspective of CUCM)
2: Control Processes (Internal processes that control the devices)

 

These are 5 of the major edge processes.

  1. SipHandler
    1. SIP phones and SIP Trunks will communicate with the CUCM using the SipHandler internal process
    2. SipHandler creates an intermediate process named SipStationInit (there is only one per CUCM),
    3. SipStationInit creates SipStationD instances for each SIP Phones.
    4. For every SIP Trunk SipHandler will create an instance of SipD   
  2. StationInit
    1. Communicates with all the Skinny devices: Phones, Voicemail ports, Media Resources (CFBs, XCDRs, Annunciators, MOD, etc...)
    2. StationInit creates one StationD per Skinny device.
  3. H225Handler
    1. Will Communicate with all the H323 Gateways and Trunks that point to the CUCM
    2. StationHandler creates an instance of H225D per H323 device   
  4. MgcpHandler
    1. Will talk to all of the MGCP gateways regardless of protocol (T1 CAS/CCS, PRI, FXS, FXO, etc...)
    2. For each T1 CAS, FXS, or FXO MgcpHandler will create an instance of MgcpTrunkD to manage the port
  5. MgcpBhHandler
    1. Will only talk to the MGCP Pri interfaces because MgcpBhHandler processes the ISDN backhauled messages from the gateway as these messages are terminated at the CUCM. When the ISDN messages hit the GW, the GW convirts the Time Division Multiplexing (TDM) messages to IP Packets (i.e. the legacy voice setup message will be put into a TCP packet which will be put into an IP packet), the gateway will open a TCP socket with CUCM, the IP packet will be sent to CUCM, CUCM will read the message off the socket when it terminates at the CUCM, CUCM will then reply with an MGCPPri call proceeding message by putting the message inside a TCP packet which is put into an IP packet, the IP packet will be sent to the gateway using the TCP socket, the GW will then translate the message into TDM and send it to the PSTN
    2. The regular MGCP messages like CRCX, MDCX, etc. go to the MgcpHandler
    3. For each Pri interface MgcpBhHandler will create an instance of MgcpPn9D

 

There are a few things you will want to find to help you better follow the call:

 

1: The CIs (these are identifiers that are unique to a call, but more specifically they are unique to a call leg within the call)

2: The CDCCs (Call Dependent Call Control process)

3: Call dependent processes (processes created specifically for the call you are analyzing). These are not as important as knowing the CIs and CDCCs; however, they can be very useful at times.

 

Here are some of the call dependent processes:

 

Cdcc

LineCdpc

SIPStationCdfc

SIPCdpc

SIPInterface

Forwarding

PickupMonitoring

MatrixControl

MediaExchange

MediaManager

Transferring

Recording

RouteListCdrc

HuntListCdrc

QueueControlCdrc

CallPark

H225Cdpc

MGCPpn9cuser

MediaResourceCdpc

H245Interface

AgenaInterface

MGCPInterface

 

When the processes are created it looks similar to this (I modified it for ease of viewing).

 

 


Before looking at the logs

There are a few things you want to know before reviewing the traces. These will help find the right call and better understand what the sequence of events are:

The time stamp of the call

The calling Number

The called number

The call flow

What happened on the call (transfer, pauses, sounds, no audio, etc...)

 

It is not exactly necessary; however, it is very helpful if you know the mac addresses and IP addresses of the phones and the IP addresses of servers and routers that may be in the call flow. The output of "show network cluster" gets the IP addresses and host names of the CUCM servers in the cluster (along with other helpful information that doesn't directly relate to call routing). This information is what helps create a more detailed call flow which is something I talk about here: How To Identify A Call Flow In CUCM


Good things to review

Troubleshooting IP Telephony book

The links on the support forums that deal with reading traces

H323 Call

SIP Call

Detailed Doc

Yet another

RTMT Features

How to gather traces from the CLI

Trace lookup per scenario

If there is a router in the call flow: How to debug the gateway

  CUCM Troubleshooting Methods, SIP Concepts and Troubleshooting Tools

SCCP Call States


Things I will discuss in this series:

How to gather traces from CUCM:

 

Make sure they are detailed, select all nodes, relative range if you can reproduce the issue, relative range if you don't know how to reproduce the issue but you are getting an RTMT alert about an issue, no issue reported by users

 

This is the video for RTMT

 

Getting the most out of this (2.5 minute video)

Phone to phone same node (SCCP - the only one I will go out of my way to make with Skinny phones)

Phone to phone same node (SIP - most the rest will be made using SIP phones)

Phone to phone different node

Phone to phone, hold, resume

Phone to phone, call transfer to another phone

Inbound call from H.323 gateway to phone

Inbound call From MGCP gateway to phone

 

The list below are place holders for future videos:

Call Recording

Call Pickup

Hunt Pilot

MWI SCCP

MWI SIP

Mobile Connect (SNR)


-------------

Please rate helpful content (i.e. videos, documents, comments) so quality content shows at the top of people's search results. Also, please select the correct answer(s) if any comment(s) answers your question otherwise the question remains on the support forums as unanswered.

-------------

Comments
Deepak Mehta
VIP Alumni
VIP Alumni

Very good initiative,+5.

Looking forward to the video series.

pkinane
Cisco Employee
Cisco Employee

Thank you, Deepak.

I have most of it already set up in the lab, just need to make the time to pull traces for all the different call examples, make the recordings, and do the editing. Hopefully it won't take forever. I will be updating the document as I complete each call example.

brianw2
Cisco Employee
Cisco Employee

Thanks for creating this video series. This is going to help me tremendously. +5

Charles Armstrong
Cisco Employee
Cisco Employee

I look forward to seeing this effort come to fruition.  Excellent initiative.

hardiksp
Level 1
Level 1

Excellent details and very helpful in troubleshooting...+5 

Deepak Rawat
Cisco Employee
Cisco Employee

[+5] mate, great quality stuff.

Regards

Deepak

pkinane
Cisco Employee
Cisco Employee

Thank you, Deepak!

I am happy to see people are able to make use of this.

Ciscollab_Amit
Level 4
Level 4

+5 my friend... You are doing a great job.. This would really help a lot of us. Looking for other ones'' :)

Much appreciated!!

jonathan.salter
Level 3
Level 3

All of a sudden I am unable to see the videos....any ideas?

Ritesh Desai
Spotlight
Spotlight

@pkinane

Excellent, great and much worthy stuff to learn troubleshooting. Thanks again.

 

thanks & regards,

Ritesh Desai.

pkinane
Cisco Employee
Cisco Employee
Thank you, Ritesh!


Thank you so much for these videos tutorials. I have been using your guides in order to deeply understand call traces in my lab as well. However, you did not mentioned that you were using CUCM Traces (User Defined Language) in Notepad ++ which I find very cool. I searched without any luck to find that CUCM Traces XML file in order to load as a language in Notepad ++. Would you be so keen to share that? Regards, JK.
pkinane
Cisco Employee
Cisco Employee

Thank you @Johnny Nkunda Kabundi.

 

The NP++ language can be downloaded from this video: https://supportforums.cisco.com/t5/collaboration-voice-and-video/sip-hold-and-resume-with-multicast-moh-basic-trace-overview/ba-p/3103507

 

I also have the call information and logs on each video as well. You may download and use the logs if you don't have much time to test in your lab.

Thanks a lot. It was very informative

rahulsahus540
Level 1
Level 1

All of sudden video are not available now..any idea? it was available till month of May 2020

giving an error "SAS  error- invalid authorization response

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: