IP Phones Randomly rebooting 7942, 7971, 7941

[danaledford]

I have been getting reports of phone rebooting and after futther analysis they seem to be loosing power as the ports are being shutdown? Anybody seen similar issues?  Wanted to post before I call TAC, any help would be nice.

Enviornment:

IP Phone Models: 7941-42 7971

IP Phone Load 9-1-1SR1S

Switch 3560X, 3560G

Switch IOS: 12.2(58) SE2

interface GigabitEthernet0/6

switchport access vlan 40

switchport mode access

switchport voice vlan 41

srr-queue bandwidth share 1 30 35 5

priority-queue out

mls qos trust device cisco-phone

mls qos trust cos

auto qos voip cisco-phone

spanning-tree portfast

service-policy input AUTOQOS-SRND4-CISCOPHONE-POLICY

SHOW LOG

000735: Dec 27 2011 09:51:31.421 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/6, changed state to down

000736: Dec 27 2011 09:51:32.420 PST: %LINK-3-UPDOWN: Interface GigabitEthernet0/6, changed state to down

000737: Dec 27 2011 09:51:34.969 PST: %LINK-3-UPDOWN: Interface GigabitEthernet0/6, changed state to up

000738: Dec 27 2011 09:51:35.976 PST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/6, changed state to up

000739: Dec 27 2011 09:51:35.976 PST: %SWITCH_QOS_TB-5-TRUST_DEVICE_LOST: cisco-phone no longer detected on port Gi0/6, operational port trust state is now untrusted.

000740: Dec 27 2011 09:51:46.855 PST: %SWITCH_QOS_TB-5-TRUST_DEVICE_DETECTED: cisco-phone detected on port Gi0/6, port's configured trust state is now operational.

[Leo Laohoo]

Were you able to get the resolution for your issue? I am getting the same issue in my environment too.

What is the exact model of your Cisco switch?

[anthony.king]

I started having this problem as well.  At first, I thought it was the IOS because the 3560G switch it was happening on had pretty old software.  I upgraded the switch and it seemed to have resolved it but I just had the phones on that switch go down again today.  Interestingly, I also had a different switch do this same thing.

Here is something else that is interesting.  We have had a CME with about 20 phones running for several years on a 3560G-24P with no problems.  We recently upgraded ourselves and our parent company to a BE-6000 (they had a Nortel with TDM phones).  Our CME was running in conjunction with the BE-6000 as we pre-deployed phones.  As soon as we put the new 7945 phones out, that switch started having problems.  Nothing had changed with that switch, our CME or our 7945 phones for years and that switch had absolutely no problems.  The new phones we put out had newer firmware so I'm leaning towards the phone firmware and/or CUCM doing something odd.  Also, the phones that were still working on our CME went down when the new phones went down, so it looks like something system wide on the switch, not localized to one port.

[Mick Doran]

We have the issue of phones resetting randomly in a CMBE6000 system with only 48 phones connected. There are 4 locations, 3 connected via EPL and 1 location via point to point T1.

All locations, including the main site are seeing sporadic phone resetting. All the phones are either 7942 or 7962 with SCCP42.9-1-1SR1S app load.

CCM version is 8.5

Over the past month I have initiated two TAC calls. The following has been done with no resolution to the problem:

• Increase the 'Station Keepalive Interval' to 300 ms from the default 30 ms
  
• Erased all of the QoS settings on every swtich and router and utilized Cisco AutoQoS
  
• Checked cabling to the CCM/Phones, etc.
  
• Set all the phones to Delayed according to the following:
  

Geometric TCP

#

The Cisco Unified IP Phone firmware 7.2(1) introduced a Geometric TCP mechanism to permit IP Phones to measure the round-trip delay between the IP Phone and Unified CM, then adapt the keepalive timeout value. This provided a very accurate failover mechanism when the network delay is consistent.

However, if the network delay is inconsistent, this mechanism may cause the IP Phones to inaccurately attempt failover. The Cisco Unified IP Phone firmware 8.4(2) introduces the ability for the Network Administrator to disable this behavior, if necessary, through the Detect Unified CM Connection Failure parameter defined on the IP Phone device configuration. The default value is Normal; this Geometric TCP mechanism can be disabled if the parameter is set to Delayed.

Has anyone resolved this issue?

[Kapil Atrish]

One of my client is facing similar issues. There servers are across the WAN in a DC. In my case, one of the remote site is loosing phones intermittently. Other remote sites are working fine at the same time.

Myself and TAC has spent lots of time troubleshooting this and finally pinpointed bad WAN connection at that site but customer is confident that there are no issues in the WAN.

I've also done similar troubleshooting what you mentioned above except disabling "Geometric TCP" which I've done last night.

If the issue continues, my next steps will be to run wireshark at remote site and at DC. If I see pakcet loss/unexpectedly higher delays, I can go back to customer and prove that the issue lies somewhere in the network and not in voice gears.

Thanks,

Kapil

[Mick Doran]

We have as well.  We had another customer that was indeed having WAN issues and once resolved the phones resetting or bouncing between fallback SRST and Call Manager stopped.

In this case I, Point-to-Point T1, we have tested and worked with the Carrier to eliminate the ciruits as the potential problem.

I too have had several TAC cases and spent many hours on this with no resolution or next steps from TAC.

We are of the same opinion that it will take some wireshark captures to truly see what is going on.  I plan to put a system with wireshark on the same switch as the call manager and do port mirroring to monitor the communication to the Call Manager for a 24-48 hour period. 

If you do the same I would recommend that you set filters on Wireshark to only look for the keep alive/SCCP ports or you will be inundated with a lot data.

I am curious as to what you will find.

[srichardson]

Did anyone on this thread find a solution or root cause?  I'm having similar issues.  Thank you.

[Leo Laohoo]

Can you create a different thread?

[Guru Mysoruu]

Hi,

Meet to had the same issue,fortunately I had called the fiber testing guy.

when he tested there was a fiber link issue which was joined,when it was cut.

Surprisingly,in the first test with general characterstics it was ok.but when we tested with specific characterstics,we were able to identify.

Regards,

Gurudath

[Aaron H]

Iam having the same issues but using Nortel 1000 phones

Several times a day the phone will reboot after which it will stay up for 10-20 minutes and then reboot again. When the phones power cycle the switches do not log a change in the port status, nor do the switches log and issues related to power.
This is happen on two different switches, a 4507R+E and a stack of 3 3850's, in different areas of our building. I have upgraded to IOS versions on both, cat4500-entservicesk9-mz.150-2.SG11 for the 4500 and cat3k_caa-universalk9.16.12.08 for the 3850's after being advised of the CSCvg74474 software bug.
https://urldefense.com/v3/__https:/bst.cloudapps.cisco.com/bugsearch/bug/CSCvg74474__;!!IECxBNJ3jonIpDRP4g!PTOn-jY4ZzcUSYcDWKxMBbjiX7lX7M5LvZflXkCpoPkFNLicS9AF0MkNxCUy6uh2oJBemdwP5IM5BJ6Je1FukZ2n7g$

The thinking at this point is that there is a bottleneck in traffic going to the voice infrastructure. All phone traffic goes through a 1 Gig link to reach the voice servers.  The voice servers are onsite, not over a WAN.