IP Phones are rebooting with reason code 13 and 16

[Suresh Hudda]

Hi,

One of my customer is running with CUCM 6.1.2 and they are facing intermittent issue with IP Pones. Actually 50-60 % IP phones are rebooting twice or thrice in a day, If someone is on call on an IP Phone then it shows “CUCM is down” and it reboots and it also reboots when it is idle. In logs we are getting reason code 13 and 16.

We have opened a TAC for it and TAC said this is a Network related issue but Network team has also not found any issue within network devices and there is no issue with other Cisco and non-Cisco applications all are working fine except IP phones.

Now I'm trying to narrow down the issue with Network team but your inputs/suggestions would be highly appreciated because it will help me greatly.

Rgds,

Suresh Kumar

[Terry Cheema]

Hard to say anything conclusive from the information. You should check the Call Manager event logs. They will give you an indication where the issue lies. If there's a failure within the call manager service etc or if there's any network related issue.

Terry

[saif musa]

Hi,
It sounds like network issue. Try to test connectivity between ip phones and CUCM. If its was ok then try to figure out port connects ip phones to the switch from switch side to use duplex auto and speed auto.

Hope that will help


Sent from Cisco Technical Support Android App

[Suresh Hudda]

Hi Saif,

Thank you !! I will check and change the speed and duplex settings  to auto at switch end.

We are trying to narrow down the issue, so we have connected the IP phone in the same switch where CUCM nodes are connected and observing the phone.

Suresh.

[yahsiel2004]

Verify that you're not having a duplicate IP issue, with CUCM.

[Suresh Hudda]

Hi Yahsie,

Thanks for the reply !!

we are not suspecting duplicate IP issue, as Phones  (40-50% of 8000 IP Phones) reboots in half an hour duration alltogether and issue resolved automatically.

Do you have any Idea, how to rule out duplicate IP issue, we will apply.

Regards,

Suresh

[Suresh Hudda]

Hi All,

Still I’m struggling to find the exact cause of IP phone’s reboot issue. During troubleshooting, I have observed one pretty strange behavior of keepalive message travelling between IP phone and CUCM. When I’m taking packet capture (Wireshark sniffer) of switch port where IP Phone has connected (user's desk). Actually IP Phone is not sending keepalive message to it’s secondary Callmanager, instead of keepalive message it sends SYN packet to it’s secondary callmanager and secondaory callmanager sends RST message back. But IP Phone is sending Keepalive messages to it’s primary callmanager properly.

510          10:38:10.257971     10.10.12.93             10.20.20.14             TCP          60            52487 > cisco-sccp [SYN] Seq=237967391 Win=8192 Len=0 MSS=1340

511          10:38:10.258790     10.20.20.14             10.10.12.93             TCP          60            cisco-sccp > 52487 [RST, ACK] Seq=1498793753 Ack=237967392 Win=0 Len=0

If I connect this IP Phone to local switch where callmanagers are connected then it is sending keepalive messages to it’s primary and secondary callmanager both as below.

6041 11:28:20.859149000 10.20.20.50 10.20.20.14 SKINNY 66 KeepAliveMessage

6042 11:28:20.859399000 10.20.20.14 10.20.20.50 TCP 60 cisco-sccp > 49153 [ACK] Seq=2917402400 Ack=3359513143 Win=5840 Len=0

6043 11:28:21.748790000 10.20.20.50 10.20.20.13 SKINNY 66 KeepAliveMessage

6044 11:28:21.749236000 10.20.20.13 10.20.20.50 SKINNY 66 KeepAliveAckMessage

6045 11:28:21.755157000 10.20.20.50 10.20.20.13 TCP 60 49152 > cisco-sccp [ACK] Seq=2124995648 Ack=2817545904 Win=8192 Len=0

IP address Info is as belows:

10.10.12.93 IP Phone which is connected at user’s desk.

10.20.30.50 IP Phone which is connected at local switch.

10.20.20.13 Primary callmanager.

10.20.20.14 Secondary callmanager.

Could anyone of you please help me to understand this behavior, thanks in advance.

Regards,

Suresh

[Suresh Hudda]

'Thanks a lot' all of you for your response on this discussion :-)

Just want to update final findings of this issue with you all, actually In customer network there was a DP (Defense Pro- which used to monitor packets) was re-masking the packets or I would say it was changing the header of packets which was causing this problem. Now all is well after bypassing DP:-) :-)