Information Hub for IP Phone Troubleshooting

pkinane · ‎12-02-2016

Introduction
Phone Registration/De-registration
Phone VPN
PhoneSecurity

Introduction

This document serves as a hub for data needed when troubleshooting Cisco IP Phone issues with phones that are registering to Cisco Unified Communications Manager (CUCM).

Phone Registration/De-registration

It is important to understand the phone registration process.

Questions to ask/answer

For phones that show unregistered, are they able to make and receive calls? If yes, did you check the registration status from the webpage of the other CUCM nodes? Did you check the status of the phone in RIS DC?

Note: If the phones are able to make and receive calls use the command below on each node to see the status of the phone in RIS DC.

show risdb query phone

If the phone shows registered on RIS DC and shows unregistered on the GUI then there is an issue between RIS DC and the GUI.

How many phones are impacted, and how many phones do you have total?
If only a subset of phones are affected, what do they have in common (i.e. model, protocol, firmware version, on the same switch/blade, at the same site…)?
Are you using 802.1x?

http://www.cisco.com/c/en/us/td/docs/solutions/Enterprise/Security/TrustSec_1-99/IP_Tele/IP_Telephony_DIG.html

Does the phone have a shared line?
Does the issue happen at the same time of day each time it happens?
Are you running any security checks (i.e. port scanners)?
Do you have any firewalls between the phone and the CUCM?
Are you doing SIP inspection on any devices in the path between the phone and the CUCM?
How many phones are in the same subnet, how many IP addresses do you have available for lease to that subnet?
Are you using SIP over TCP or UDP?
Are the phones using a secure, or a non-secure, profile? If the phones have a secure profile, did they have an LSC installed before applying the secure profile to the phone's configuration (this is necessary)?
Is anyone logged into the problem phone(s) using extension mobility? If yes, does the protocol (SCCP/SIP) of the device profile match that of the phone and does the same behavior exist after logging out?

Data To Gather From The Phone

Document the message on the phone's screen when the issue occurs. It is typical for a message to be displayed on the phone's screen so be sure to ask what the user's are seeing.
Get a packet capture (pcap) from the phone

Note: For SIP phones be sure to look at the expires field in the SIP register message found in the pcap. The default value for this field when the message is sent from the phone to it's primary CCM is 120 seconds. When the phone is sending a "keep alive" register message to it's secondary CCM server the expires field is 0.

Tip: The things below will require web access to be enabled on the phone and you may need to enable certain debugs. Even if a phone isn't registering you can sometimes change the settings on the phone. It is worth trying to enable webaccess, span to pc port,and SSH Access. If the issue is CTL or ITL related you will not be able to change the settings on the phone.

Document the debug messages on the phone
Check for cores on the phone and download them if they are there. Be sure to also gather the output of show show core-dump from the CLI of the phone if cores were found on the web interface of the phone.
Gather the CDP Neighbor information from the Network page located in the Network statistics section.
Get the console logs from the phone. If the phone supports the Problem Report Tool (PRT) feature it is recommended to use the PRT.

Data To Gather and Review From The Switch For The Phone

The running configuration using show run.
The output of show proc cpu hist.
The output of show log.

If 802.1x is in use, gather the output from the following commands:

Replace $if_name$ with the name of the interface (i.e. FastEthernet0/1)

sh cdp n $if_name$
sh run interface $if_name$
sh authentication session interface $if_name$
sh ip access-list $if_name$
sh tcam interface $if_name$ acl in ip
sh tcam counts detail
sh dot1x interface $if_name$
sh ip device tracking interface $if_name$
sh fm interface $if_name$

Data To Gather and Review From CUCM

The directory number (DN) of the phone. If there is no DN and the phone uses the Session Initiation Protocol (SIP), the phone will not register.
The phone's configuration file
pcaps from the CUCM servers. It might be best to get a pcap from all the CUCM servers to be sure you don't get a pcap from the wrong server.
Use the Real Time Monitoring Tool (RTMT) to collect logs and the pcap from the CUCM servers. Be sure to select all servers when collecting the logs. Depending on the environment/symptoms you may want to collect some or all of the following log types: Cisco CallManager, Cisco Certificate Authority Proxy Function, Cisco Tftp, Cisco Trust Verification Service, Event Viewer-Application Log, Event Viewer-System Log, and Packet Capture Logs.
The output of show itl and show ctl from all TFTP servers in the CUCM cluster
The output of these commands from the CUCM publisher:

Determine if the cluster is in mixed-mode:

run sql select paramname,paramvalue from processconfig where paramname='ClusterSecurityMode'

Determine if the rollback parameter is set to true:

run sql select paramname,paramvalue from processconfig where paramname='RollBackToPreGrayback'

Determine if database replication is healthy:

utils dbreplication runtimestate

Note: If the cluster is not in mixed-mode, the output will look like this:

admin:run sql select paramname,paramvalue from processconfig where paramname='ClusterSecurityMode'
paramname paramvalue
=================== ==========
ClusterSecurityMode 0

Note: If the rollback parameter is set to false, the output will look like this:

admin:run sql select paramname,paramvalue from processconfig where paramname='RollBackToPreGrayback'
paramname paramvalue
===================== ==========
RollBackToPreGrayback F

Tip: For an explanation of the output from utils dbreplication runtimestate review this document.

Things to search for in the phone logs

Fail
Err
Error
Exception
newUnregReason=
Lastoutofservice
Fallback
socket error=
opvvlan
JAVA-sipio-
REGISTERED
Network_detect_change_task
tftpAddr1=
Locked out:
VPN.: (NOTE: Make sure you are searching with regex for this one or the "." will be analyzed as a literal instead of a special character)

Things to search for in the CUCM logs

The MAC address of the phone
The ip address of the phone

Tip: If you see error messages you can find the explanation of the reason code in the Error and System Messages Documents. It is also useful to know the last out of service values.

Additional Reading

Phone VPN

These documents are great resources for understanding and troubleshooting phone VPN issues.

Phone Security

An important thing to note is the you can install an LSC on a phone without being in mixed mode (without a CTL file). This is because of the ITL. The only reason to use mixed-mode in CUCM version 8 and higher is to do encrypted signaling/media.

To do encrypted signaling or encrypted media the CUCM cluster will need to be in mixed mode and the phone will need to have an LSC on it. These two documents discuss putting the cluster into mixed mode (the second also discusses LSC install):

How to retrieve certificates from Cisco IP phones
IP Phone Security and CTL
Migrating IP Phones Between Clusters with CUCM 8 and ITL Files
Communications Manager Security By Default and ITL Operation and Troubleshooting
Cisco IP Phone Certificates and Secure Communications
This is for Jabber but closely related to this section Encrypted Media With Jabber On Windows 7