High CPU Usage (90%) on 2 CUCM Subscribers.

[ShidlovichMor4576]

Hello, 

2 Subscribers are reaching 90% CPU Usage, when I looked in the RTMT I saw that the Database Layer Monitor (DBmon) Process got 49% CPU usage on Sub1 and 45% CPU usage on Sub2.  I ran utils DBreplication runtimestate command and seems like there is no problem with the replication between the 3 servers (Pub, Sub1, Sub2). 

I pulled out Database Layer Monitor process Logs from the RTMT and will post the logs here, the Log doesn't contain too much Information, if there are other logs I should be looking at - would be grateful for any suggestions.

Thanks in advance.

[Nathan Gageby]

Have you compared the ova sizing to the supported phone counts?

[ShidlovichMor4576]

Hi Nathan,

Yes, also this is not a new environment. these servers are up and functioning for at least 2 years, never encountered this issue before.

[Nathan Gageby]

How is the free space on the partitions? What is the total phone count registered to each server?

[ShidlovichMor4576]

Hi Nathan,

There is enough free space on the partitions, total phone count registered is 3600, about 1000 to each server when each server can deal with up to 7500 phones.

[Mohammed al Baqari]

Hi,

Do you need to keep this service parameter on.? It's off by default. This
service parameter is called 'tables out-of-sync detection'. It keeps
checking the tables every 1.5 min. I don't know how many endpoints you have
but this can be an issue with large number of endpoints (including phones,
gateways, etc). Also, it can be an issue if you have DB table flapping or
you cluster over WAN.

Generally, I recommend to check with tac on this.

***** please remember to rate useful posts

[ShidlovichMor4576]

Hi again Mohammed and thank you,

The "Table out-of-sync detection" service is off, it was never on as far as I know.

I have around 3600 registered phones.

[ShidlovichMor4576]

Restart of the Cisco Database Layer Monitor service resolved the issue, Cisco Tac was opened, trying to figure out the root cause of this event.