My org has a single CUCM cluster with around 10k phones registered across 100+ buildings. We've been semi-regularly experiencing an issue that we've had trouble tracking down. Due to the nature of our deployment and the intermittent/difficult to reproduce nature of the issue, we've had a hard time providing TAC with requested info, so I thought I'd float this to the community to see if anyone else has experienced this and/or has any thoughts on root cause.
In troubleshooting, we've described the issue as follows: 8851 or 8865 phones with sidecars (36- or 28-button) show unregistered in CUCM, and debug display on the phones' web UIs show alternating timeout and failback errors every two minutes and change (SIP Station KeepAlive Interval = 120). Altering config so that only half of the available sidecar buttons are configured restores stable connectivity every single time.
As an example, a user has an 8851 with a CP-8800-A-KEM, 26 BLFs configured on the sidecar, and is experiencing this issue. I super copy the config to a dummy MAC to preserve it, then edit the phone in CUCM to remove the configuration for 12 of the buttons (14/28 used now); They're still assigned as BLF buttons, just no Destination/Directory Number/Label set. I apply the config change, the next time the phone registers, it stays registered. I edit the config again, add a single BLF back to the sidecar (15/28 used), the phone immediately unregisters and goes back to a timeout/failback loop until I remove the config from that 15th button. The order/position doesn't matter, I could put half on page 1 and half on page 2, or all on page 2, no change in the phone's ability to stay registered until >14 buttons are configured.
TAC has suggested network issues, but we're not seeing how that could be the case. E.g., I've got one remote site right now with two buildings. Building A has five 8851s with sidecars, three of which are currently experiencing this issue. Building B has six 8851s with sidecars, only one of which is experiencing this issue. Both buildings share a WAN connection, and in some cases affected and unaffected phones are connected to the same switches. All sidecars are CP-8800-A-KEM.
I'd really appreciate any thoughts anyone has on this! This issue has persisted through multiple phone firmware upgrades and a UCM upgrade from 12.x to 14.x.
