RTMT errors Informix Could not insert new row - duplicate value in a UN

[bkhanal1971]

I am getting tons of RTMT alerts as given below. Is there anyway I can find the phone/mac that is causing this? We use extension mobility in our entire company consisting of about 100 remote sites. My CUCM version is 11.0

MatchedEvent : Jun 20 16:04:39 cucmsrv1 local7 1 ccm: 1741924: cucmsrv1.company.com: Jun 20 2019 20:04:39.457 UTC : %UC_CALLMANAGER-1-DBLException: %[ErrorCode=-239][ExceptionString=Error fetching data from datasource: [Informix][Informix ODBC Driver][Informix]Could not insert new row - duplicate value in a UN][AppID=Cisco CallManager][ClusterID=StandAloneCluster][NodeID=cucmsrv1]: An error occurred while performing database activities AppID : Cisco Syslog Agent ClusterID :
NodeID : cucmsrv1
TimeStamp : Thu Jun 20 16:04:39 EDT 2019

[Maren Mahoney]

The ErrorCode=-239 is often related to auto-registration issues when CUCM thinks the device already exists. The error itself will generate shortly after the errant device contacts CUCM. Can you post the CallManager Service trace file (or a significant portion leading up to one of these errors) here and we can take a look?

It's also worth a quick check to make sure your databases are in sync. On the Publisher, issue the utils dbreplication status command. Then wait 15 minutes or so and issue the utils dbreplication runtimestate command. If the runtimestate is not yet complete (it's pretty obvious when you look at the output) leave it run some more and check again.

Maren

[bkhanal1971]

Hi Maren,

You are right it is caused by one of the errant phone trying to auto register. Can you please explain how I can collect Call manager traces? I can't find such service name in RTMT.

Thank you, Khanal

[Maren Mahoney]

To collect trace files, the most straightforward way is using the RealTime Monitoring Tool (RTMT). You can download the RTMT application from the CUCM interface under Applications > PlugIns. Here is a link to a document that shows how to use the tool to collect traces. You will want to collect the trace file for the Cisco CallManager service and also the Cisco Tftp service.

How to Collect Traces for CUCM 9.x or Later

Maren

[bkhanal1971]

Here is the trace file for call manager and tftp services. I set the trace duration to last 5 minutes since those rtmt alerts are generated every minutes. Please let me know if need to set the trace duration to a bit longer period.

Thank you,

Khanal

[Maren Mahoney]

There seems to be something wrong with the file you uploaded. It won't open saying it's an "invalid file". Can you post the two trace files individually?

Maren

[bkhanal1971]

Please let me know if you have trouble opening the logs i uploaded. Thank you so much for your assistance.

Thank you,

[Maren Mahoney]

I can open the CUCMLOG files, but there are 28 CUCM logs covering a hour's worth of time and that's too many for me to plow through.

The other log you sent, the Informix log, is not the trace from the Cisco Tftp service. And that trace (the trace of the Cisco Tftp service) is the key one. It will show a phone requesting the XMLDefault.cnf.xml file and that will be the phone we are looking for.

Maren

[bkhanal1971]

Sorry about that. I have captured new logs and this time i have reduced the size of call manager service logs. I don't know how I can filter out unnecessary logs. Thank you so much.

Khanal

[Maren Mahoney]

I could not download or open the RTMT4.zip, but the RTMT3.zip gave the information we are looking for.

It seems you have three devices that are repeatedly trying to auto-register:

• 84802DD4FA6D - 172.26.28.68
• 0012D912231B - 172.25.207.197
• 84802DD4FDC1 - 172.26.4.71

Locate those phones and either configure them in CUCM or take them offline and your problem should be solved. Let us know how it goes.

Maren

[Maren Mahoney]

After a little more digging I discovered that the following three devices are also requesting the XMLDefault file repeatedly, although less frequently.

• 00082F1A4F66 - 172.25.68.199
  BCC49396AE3F - 172.25.188.187
  0012D9122F75 - 172.25.207.197

Maren

[bkhanal1971]

Hi Maren, Thank you so much for taking your time to assist me. Somehow, those alerts stopped since Friday. I have no clue how. Maybe someone in the remote site swapped the faulty phone. Anyway, now I know that i need to look at the tftp traces to find out the phones trying to download default xml file repeatedly without success. Can you tell me what should I be looking in the call manager traces?

Thank you, Khanal

[Maren Mahoney · Accepted solution]

When phones reach out to the TFTP server during the boot/registration process, they ask the TFTP server for the file SIP<MAC>.cnf.xml. When you build a phone in CUCM it pushes this file (with the MAC address in the name) to the TFTP server.

If the phone has not yet been built on the CUCM server, this file will not exist so when the phone requests it the TFTP server will reply with a file not found. The phone will then attempt the auto-registration process by requesting the file XMLDefault.cnf.xml. The phone will do this whether auto-registration is configured on CUCM or not. If that fails, the phone will reboot and repeat the process.

So, what I looked for in the TFTP trace file were repeated requests for the XMLDefault.cmf.xml file. These would be phones that are cycling over and over through the boot/registration process.

If I had not seen the XMLDefault.cnf.xml file be requested repeatedly I was going to look in the Cisco CallManager trace file for the registrations themselves. I would have been looking at the SIP messages for REGISTER messages for anomalies.

I hope that clarifies!

Maren

[bkhanal1971]

It absolutely helps! Thank you very very much!!!!