UCCX Total Processes And Threads Exceeded

[Gordon Ross]

We recently upgraded to UCCX 10.6(1) and we've started receiving the following RTMT alerts quite frequently - mainly in the morning:

From Tue Nov 17 08:32:45 GMT 2015 to Tue Nov 17 08:35:15 GMT 2015 on node uccx-a01,
there are 1 TotalProcessesAndThreadsExceededThresholdStart alarm(s) and 1 TotalProcessesAndThreadsExceededThresholdEnd alarm(s) received.  
On Tue Nov 17 08:35:01 GMT 2015, the last TotalProcessesAndThreadsExceededThresholdStart alarm generated:  
NumberOfProcesses : 200
NumberOfThreads : 2812
Reason : Total processes and threads have exceeded the maximum tasks [3000]
ProcessWithMostInstances : Process [cmoninit] has instances [15]
ProcessWithMostThreads : Process [cuic_reporting] has Thread Count [792]
AppID : Cisco RIS Data Collector
ClusterID : StandAloneCluster
NodeID : uccx-a01
TimeStamp : Tue Nov 17 08:35:01 GMT 2015

I can't see anything in BugTrack and Google only shows someone asking about a similar problem but with no resolution.

Any ideas, or is a call to TAC in order?

GTG

[Manish Gogna]

Hi Gordon,

You can use the workaround from the following bug

https://tools.cisco.com/bugsearch/bug/CSCuq84179/?referring_site=ss

Manish

[Deepak Rawat]

Adding to the defect which Manish currently pointed out, please increase the count to 4000 since you are using UCCX 10.6(1) and the number of suggested Processes and Threads count in this particular version is 4000 instead of 3500. The reason for this is the fact that number of CUIC threads (cuic_reporting) which add to the Processes And Threads Exceeded count most of the times had been increased from version 10.5 to 10.6, hence just increasing the count to 3500 can still trigger the alert.

Even on Cisco Unified CCX Serviceability, you will see the suggested value as 4000 only.

Regards

Deepak

[Jonathan Schulenberg]

I have a customer who is seeing this alarm many times a day. They are running 10.6(1)SU2 with the Maximum Number of Processes and Threads set to 4000.

Date: Oct 14 19:12:58
Machine Name: [redacted hostname]
Severity: Critical
App ID: Cisco AMC Service
Message: : 633: [redacted FQDN]: Oct 14 2016 11:12:58 PM.856 UTC : %UC_RTMT-2-RTMT_ALERT: %[AlertName=TotalProcessesAndThreadsExceededThreshold][AlertDetail= From Fri Oct 14 18:19:58 EDT 2016 to Fri Oct 14 19:12:58 EDT 2016 on node [redacted hostname],#012 there are 1 TotalProcessesAndThreadsExceededThresholdStart alarm(s) and 0 TotalProcessesAndThreadsExceededThresholdEnd alarm(s) received. #012 On Fri Oct 14 19:12:40 EDT 2016, the last TotalProcessesAndThreadsExceededThresholdStart alarm generated: #012 NumberOfProcesses : 247#012NumberOfThreads : 3766#012Reason : Total processes and threads have exceeded the maximum tasks [4000]#012ProcessWithMostInstances : Process [ext4-dio-unwrit] has instances [20]#012ProcessWithMostThreads : Process [cuic_reporting] has Thread Count [1211]#012AppID : Cisco RIS Data Collector#012ClusterID : StandAloneCluster#012NodeID : [redacted hostname]#012 TimeStamp : Fri Oct 14 19:12:40 EDT 2016][ClusterID=][NodeID=[redacted hostname]]: RTMT Alert

Is there official guidance of what to do about this? For example, is this indicative of a real problem that should be investigated with TAC or is it simply higher than typical CUIC usage? I have not been successful finding a reference to this in any of the product documentation or a TechNote on the topic.

[cowherjohn]

We have this same alert and are concerned, I have an open TAC Case, but wondered if anything was ever found and cannot determine whether it was.  We are just upgraded to Cisco Unified CCX System version: 10.6.1.11002-15.

From Wed Nov 15 02:30:31 EST 2017 to Wed Nov 15 06:57:31 EST 2017 on node SDIPCCXPRD01;
there are 1 TotalProcessesAndThreadsExceededThresholdStart alarm(s) and 1 TotalProcessesAndThreadsExceededThresholdEnd alarm(s) received.

On Wed Nov 15 06:57:01 EST 2017; the last TotalProcessesAndThreadsExceededThresholdStart alarm generated:
NumberOfProcesses : 228 NumberOfThreads : 3773 Reason : Total processes and threads have exceeded the maximum tasks [4000]

ProcessWithMostInstances : Process [cmoninit] has instances [15] ProcessWithMostThreads : Process [cuic_reporting] has Thread Count [1085]

AppID : Cisco RIS Data Collector ClusterID : StandAloneCluster NodeID : sdipccxprd01  TimeStamp : Wed Nov 15 06:57:01 EST 2017

Thanks

[Muhammad Rehan Naseem]

Hello cowherjohn,

We are getting same error, currently we are on Cisco Unified CCX System version: 10.6.1.11002-15.

Did you manged to find reason and fix ?

Thanks

Rehan.

[chad.andrews]

Per TAC Support on UCCX 10.6:

The Total number of processes and threads is a parameter which needs to be decided and set based on the features/applications being added to the system in each release. You can modify this value up to "5000" on your current UCCX version.

The default limit of 3000 which was set for the earlier release had to be increased based on the requirements and the features added in the releases. In your current version the default limit to 4000. However, the value that can be set for the parameter is 5000 in order to avoid the alerts.

To change the value:

-Navigate to UCCX Serviceability --> Tools --> Service Parameters

-Select the server and select the service as 'Cisco RIS Data Collector'

-Change the value of Maximum Number of Processes and Threads to 5000.

-Perform this on both nodes

*After hours* =

-Restart the UCCX Engine on both nodes= First on publisher, then subscriber.

[Stuart C]

Hi, did you get to the bottom of this issue, I have the same issue as all the others on this post it seems.  Value on parameter is set to 3000 where as the suggested is 4000....