UCCE/Router -A 10.5.2 Private Link network outage

[Ramamoorthy Shanmugam]

Dear All,

 We are running UCCE 10.5.2 with comprehensive call flow with Rogger 4000 Agent deployment  model. In Rogger A side Private link has been broken on  everyday night at 12:30 AM and during the time below process are getting restarted.

Router : rtr, rts and dbagent ,

Logger : clgr, hlgr .

In the same data centre  we have CVP VRU ,  CUCM and MR PG and  for  those components Private link there is no drops and errors .

We suspected may be the below reason after few days we changed Logger DB  scheduled auto Purge time has been changed to 1:30 AM .but there is no luck .please help me to resolve this issues .

Antivirus disabled

Daily DB auto purge time also changed to 1:30 AM

Rogger -A

Node: ICM\ipcc\LoggerA, process: hlgr, successfully reinitialized after restart.

Node: ICM\ipcc\LoggerA, restarting process: hlgr, after having delayed restart for 10 seconds.

Node: ICM\ipcc\LoggerA, process: clgr, successfully reinitialized after restart.

Node: ICM\ipcc\LoggerA, restarting process: clgr, after having delayed restart for 10 seconds.

Node: ICM\ipcc\LoggerA, process: hlgr, is down after running for 816 seconds. It will restart after delaying 10 seconds for related operations to complete.

Node: ICM\ipcc\LoggerA, process: hlgr, exited cleanly and requested that it be restarted by the Node Manager.

Terminating process: hlgr.

Node: ICM\ipcc\LoggerA, error checking health of process: hlgr.

Node: ICM\ipcc\LoggerA, process: clgr, is down after running for 813 seconds. It will restart after delaying 10 seconds for related operations to complete.

Client: hlgr, stopping due to error.

Client: clgr, stopping due to error.

Requesting MDS termination due to error.

Exited and requested that the Node Manager reboot the system: the Logger has stopped processing time messages.

Process: clgr at the central site side: A is down.

Process: hlgr at the central site side: A is down.

Process: rtr at the central site side: A is down.

Process: basv at the central site side: A is down.

Process: tsyr at the central site side: A is down.

Process: csfs at the central site side: A is down.

DBWorker process on side: A is down.

Process: dbw at the central site side: A is down.

Process: rcv at the central site side: A is down.

Thanks & Regards,

S. Ramamoorthy 

[Omar Deen]

What do your purge jobs say in SQL Mgmt Studio? Is the actual code for the purge pointed at the correct registry? Let's check the TCD purge since it's one of the larger tables

Check the SQL Jobs:

Open SQL Mgmt Studio > SQL Server Agent > Jobs > right click <instance>_sideA_PurgeTCD and select View History

Check the stored procedure

Open SQL Mgmt Studio > expand <instance>_sideA database > Programmability > Stored Procedures > right click dbo.PurgeTCDTable > Modify. You should see something that says 

@key=

what the key is equaling to is a registry path. Does that path actually exist? You should confirm this. Also... don't actually modify anything unless the path is actually wrong. This is what it should look like...

@key='SOFTWARE\Cisco Systems, Inc.\ICM\<instance>\LoggerA\Recovery\CurrentVersion\Purge\Retain\CallDetail\Termination',

Obviously, the <instance> portion will be unique to your environment. 

If the logger is has reached it's max file size limit, the logger service will go down, which would kill off the router service... you cannot route calls unless the logger is up. When the router process first starts, it reads all of the config info from the logger to memory. 

You might want to check the sizing of your databases as well... Open SQL Mgmt Studio > right click <instance>_sideA > Properties > Files... is the initial size sufficient? For 14 days, a 40GB size is usually good enough. 

[Ramamoorthy Shanmugam]

Hi Omar, 

 Thanks for your reply , I have verified all the parameter and looks like fine . Please find attached snapshot kindly check those logs and revert back .

I have changed Daily Autopurge Schedule Settings in logger and same snapshot is attached in this forum.

If you need any logs or clarification please respond to me .

Ram.

[Ramamoorthy Shanmugam]

Dear All,

 Cisco TAC also did not help for this case we verified Microsoft , SQL and system performance but no luck. Finally we rebuild the Rogger-A system and now the Private link is stable.

We are suspecting may be some issues in OS or SQL level .

Ramamoorthy Shanmugam