8851 Phones rebooting randomly

[Vishal Bhardwaj]

Hi there,

For Customer ABC we deployed 8851 Phones for their Contact center ( around 80). They are using Finesse/UCCE. CUCM ver is 9.1.2. 

Issue: Randomly any 8851 phone will reboot. It will go through the whole power cycle and Agent will have to login again to Extension Mobility and Finesse.

Steps taken (Issue still not resolved as of now):

1) Customer had 3560 series switches. At that time there were almost 15 to 20 phone resets were happening in a day.

2) Changed Switches to 3750G. Reset reduced to almost around 2 a day. (Seems like a cabling issue at this point)

3) Cabling tests were done by third party company using Fluke. Results came green and OK.

4) Patch cables from Phone to Jack was replaced with a Cat6 but no difference.

5) Speed and Duplex on phone and Switch was hard-coded to 1000 Full but again no difference.

6) Firmware upgraded on all phones as suggested by TAC but since then it's almost 6 phones a day which are rebooting. Possibly rollback for that either today or tomorrow.

Next Steps scheduled to be taken:

1) Put a Sniffer/Pcap to capture all the IP traffic and go on from there.

2) Hard-code the Speed and Duplex to 100 and FULL.

3) Possibly put few 7900 series phone to see if it's SIP causing the issue.

The issue is still not resolved so if anyone is experiencing similar situation or had faced in past, Please share your thoughts. I did read a few forums where people faced similar issue and for some it got resolved by replacing switches or by changing port security timer setting on Switch. We haven't tried port security timer yet but will update if we ended up doing it.

I will update this if issue gets resolved or about the steps taken, meanwhile please share any information you think can help out here.

Thanks

[Manish Gogna]

Hi Vishal,

Has the issue been narrowed down to any of the following ( where it is experienced the most )?

1) Specific switch or floor

2) Specific Device Pool or CM Group

3) Is there anything connected to these IP phones ( like a PC / KEM etc )

4) If only this phone model is affected then phone debugs might also help if u end up opening a TAC case

HTH

Manish

[Vishal Bhardwaj]

Hi Manish,

TAC Case has been opened and in fact right now it's with Developer's Business Unit. pcap is running from CUCM to Phone and Phone to CUCM. Also on the switch side. There have been couple of incidents and debugs have been provided to TAC. Let's see what the output comes from Developers.

I will update it once i have some information back from TAC.

Regards,

Vishal

[Vishal Bhardwaj]

Just providing an update here.

After lots of troubleshooting and based on findings from Developers it is discovered that Firmware of USB port 8851 was causing the issue. Since last week or so Agents are using RJ11 headsets or analog headsets and there has been no issue whatsoever. So it clearly was a USB related issue.

Cisco has sent details to their Partner who develops that firmware to USB and once it's fixed we will try again with the USB headsets. 

I will keep it updated if there's anything new. 

[Adam_Lynch]

I am having the same problem. We changed out switches, cables, and phones. Your update makes sense to me. Only the phones that have wireless "USB" headsets are rebooting. I have a TAC case open at the moment I will mention your findings. Thank you very much for posting. This was a big help!

Cheers

Adam

[Vishal Bhardwaj]

Excellent, i am glad i could help. Also, Cisco just now put a new firmware which is supposed to fix this issue. So try that as we are going to try it as well.

[Adam_Lynch]

I see FR 10.3(1), it does not mention any fixes in the release notes. Is this the firmware you are referring to?

[MARTIN WEDDLE]

This issue does not appear to be resolved with firmware 10-3-1-20. Or at least, I have a couple of these phones on CUCM 10.5 running 10-3-1-20 firmware and they still reboot randomly. 

[jeliasoncisco]

Hi Martin,

We have had similar issues and some successes. Question, are any of the phones using a Power Cube 4 power adapter or are they using PoE? We are finding that during minor power sags, the phones can get corrupted. One thing we have done is a full factory default on them and they pull the firmware from the server again fixing the problems. For ones with Power Adapters we have put them on AC Line Conditioners like the APC LE600 and problems went away. Also if you are using PoE, best to ensure that there is a high quality UPS on them. The full factory default that re-writes the flash is when you pull the power on the phone and press the key while plugging in the phone. The full instructions are in the 8851 user manual. I recommend you do this as your own risk as the firmware needs to be available on your TFTP server to restore. Please let me know more about your issue and how it goes. Thanks.

[fenilantony]

did u try new firmware?

[Vishal Bhardwaj]

Yes, 10.3.1.20 new firmware is stable and we haven't had any reset issue since. You can try it as well.

[Waldo Roca Martin]

Hi I have the exactly same problem on a 8851 with usb headphones and BEKEM module. In fact there are other phones with the same configuration but not usb headphones and they are working well.

I have just updated the firmware to 10.3.1.20 version. Have you had any issue since your updated the firmware?

This is the bug, it is not officially fixed.

https://tools.cisco.com/bugsearch/bug/CSCuu19519

Thank you

Best Regards

[jeliasoncisco]

Hello. Did you try the new firmware cisco released? Did it fix the phone reboot due to USB issue?

Please advise the version number so we can ask them for the same version. Thank you.

[btancon01]

I am also having the same issue except instead of wireless headsets we have 8851's with KEM's attached rebooting

[jeliasoncisco]

Hello

The CP-BEKEM module connects to the 8851 via the USB spine connector. I can imagine that anything that is affecting the USB firmware on the 8851 would also cause a reboot with the BEKEM module connected. I too have had the receptionists with BEKEM modules complain about random reboots of their phone.

Cisco, any ideas? Thanks.