Phones suddenly did not register and gets rejected

[JH51928]

Hello,

I'm having issues with the phones in the same cluster. It gets stuck to registering and gets rejected after the phone reboots.

We successfully upgraded our CUCM from 11.5 to 12.5 a week ago.
CUCM cluster filled up the active partition to 98% after the upgrade and technical assistance center advised us to rebuild using 7500 users OVA so we can have 110 GB space. 3 days ago I rebuilt the secondary subscriber using DRS restore, a day after I rebuilt the primary subscriber using DRS restore , then a few hours later, we found out that all phones are getting rejected. When we run show ITL , it says ITL was last modified on Thursday, Aug 13 22:50. I'm not sure but that could be the time I was rebuilding the primary subscriber.

All phones are stuck registering
*I looked at the RTMT, device search>phones and it says rejected, db config error. 

*In the publisher syslog, it shows "ITL are updated or installed" on the phones.

*In the primary subscriber, it shows "An endpoint attempted to register but not compete registration".  It also has Reason 3 = DatabaseConfigurationError 
*Phones are getting ip address from DHCP server
*Call Manager and TFTP service on our publisher, primary subscriber, and secondary subscriber are running.
*We tried deleting the ITL on phones, it didn't work, same issue
*We tried rebooting the servers in the cluster, it didn't work, same issue
*We tried replacing the ITL file by running "utils itl reset localkey" it didn't work, same issue.

Please help, this has been communicated with TAC yesterday morning already but they did not get the chance to get back to me.  

[Maren Mahoney]

That error indicates that as far as CUCM is concerned, the device does not exist in the database and since autoconfiguration is not enabled, the device is being rejected.

System Error Messages for Cisco Unified Communications Manager 11.5(1)  (Same message in 12.5)

So either the phones did not restore properly, or did not restore at all, or the database is corrupted.

• First: What is the status of the database replication after your restore?
• Second: Check the log for the restore, were there any errors?
• Third: Try building a new phone from scratch to see if it will register. That will tell you if you have a system-wide failure or if it is just a problem with the restored phones.

Let us know what you find and we will proceed from there.

Maren

[JH51928]

Thank you for your response Maren, 

• First: What is the status of the database replication after your restore?

I did checked the database replication after performing DRS restore and the results for all the all the servers are good (2). I also run utils dbreplication status to checked the tables, they were no issues and were able to update data.   

• Second: Check the log for the restore, were there any errors?

The restore status said "success". Is there any particular log I could check?   

• Third: Try building a new phone from scratch to see if it will register. That will tell you if you have a system-wide failure or if it is just a problem with the restored phones.

I have not try this yet. I will try this 

[JH51928]

Third: Try building a new phone from scratch to see if it will register. That will tell you if you have a system-wide failure or if it is just a problem with the restored phones.

Did not work, same issue

I have attached the phone console logs.

I found a reason code in there but I could not find anything on the internet about it.

1868 NOT Aug 16 16:55:11.835403 (561-1039) JAVA-SIPCC-SIP_REG: ccsip_handle_ev_4xx: SIP_REG_FAILURE_POSSIBLE_UCM_CONFIG_ISSUE - Received 4xx in ccm mode. Setting registration_reject to TRUE.
1869 NOT Aug 16 16:55:11.837326 (561-1038) JAVA-SIPCC-SIP_CC_PROV: ccsnap_gen_deviceEvent: event type : NOTIFYPROMPT
1870 ERR Aug 16 16:55:11.845170 (561-1039) JAVA-%REG send failure: REGISTER
1871 ERR Aug 16 16:55:11.845383 (561-1003) JAVA-System P7-display MQThread|cip.app.AppFrame:startNotification - Notification: REG send failure: REGISTERcan't be display in status prompt
1872 NOT Aug 16 16:55:11.860857 (561-1039) JAVA-SIPCC-PLAT_API: setUnregReason: setting unreg reason to=24

[Maren Mahoney · Accepted solution]

The phone trace file shows it connecting to the TFTP server at 10.6.130.11 and downloading a config file. But after download, the file cannot be parsed by the phone. In the messages.1 file, look at Line 1358 and again at Line 1682:

MQThread|cip.xml.XmlParser: - XML Parser Exception: Too few 'member' elements in '/device/devicePool/callManagerGroup/members': occurs=0 minOccurs=1 (line=47)

If I'm reading it right, the phone has been told about a CallManager group, but the group has no members. That may have something to do with the restore. So here's what I suggest:

• For kicks, check to make sure that the CallManager service is running on your call processing nodes. (I'm sure it is, but check anyway.)
• Check your CMGs to make sure they have members, and have the right members.
• If it looks ok, remove the servers from the CMG, save and reset. Then re-add the servers to the CMG, save and reset.

If that's not it, I think you may need to talk to TAC.

Maren

[AbuAkif Riyaz]

Hi,

1. All Phones or Some model?

2. Which Phone model you have this logs?

3. Cluster type Non-Secure or Mixed?

4. Upgraded CUCM exact version

5. If possible can you please collect Packet Capture from IP Phone & TFTP Node (Primary in Network VLAN configuration) and CUCM Node (Primary in Phone Device pool)

    A> For Phone please do Switch port Mirroring from Cisco Switch

    B> For CUCM go to (TFTP & CUCM node)

         utils network capture eth0 file packets count 100000 size all host ip <phone IP>        

6. Can you please check the Configuration File for the Phone from TFTP

    A> http://x.x.x.x:6970/SEP<Phone MAC>.cnf.xml

[JH51928 · Accepted solution]

Hi Maren, 

I checked the tftp logs too and yes the phones get connected to TFTP server but it gets rejected when connecting to call manager because the device name, mac, and user id were all empty. 

I  believed the DRS restore was not fully restored or the db got corrupted.

Hi Riyas,

1. All Phones or Some model?

ALL

2. Which Phone model you have this logs?

ALL

3. Cluster type Non-Secure or Mixed?

Non secure

4. Upgraded CUCM exact version

12.5.1.12900-115

5. If possible can you please collect Packet Capture from IP Phone & TFTP Node (Primary in Network VLAN configuration) and CUCM Node (Primary in Phone Device pool)

    A> For Phone please do Switch port Mirroring from Cisco Switch

    B> For CUCM go to (TFTP & CUCM node)

         utils network capture eth0 file packets count 100000 size all host ip <phone IP>       

No problem with network connectivity  

6. Can you please check the Configuration File for the Phone from TFTP

Phones had no problem connecting to TFTP. Phones were able to download the config file

Anyway, thanks for all your help. The issue was critical and time constraints, I really wanted to troubleshoot more but I figured that the database might be corrupted. What I did was to just restore the publisher from backup so it will get the db that has no issue, and that fixed the problem.