OPC exit errors on ICM PG's

[ccarmichael]

We have been receiving OPC exit errors as shown below in node manager on the PG's at one of our sites. These cause a disconnect of about 30seconds where calls are not routed. Hotfixes have been unable to correct this issue.

Has anyone experienced this and had success resolving the problem.

Thanks

-nm Trace: Wait returned for process opc pid 168 handle 214 wait handle 214

-nm ICR\cig\PG11A node process opc exited with unexpected exit code 0xc0000005.

-nm ICR\cig\PG11A node process opc exited after 363150 seconds. Process restart will be delayed for a minimum of 1 seconds.

-nm Trace: Received MDS_MSG_OUT_OF_SERVICE

-nm Trace: Wait returned for process pim1 pid 19a handle 17c wait handle 17c

-nm ICR\cig\PG11A node process pim1 detected failure and requested that it be restarted by the Node Manager.

[2r-reeder]

You need to pull opc, pim, and mds logs for that same time to see exactly why everything is dying. My gut says buffers are getting busted or you have an IP Network issue, but those are TOTAL guesses. Good luck!

[sandgupta]

Looks like OPC process crashed. Is there any Drwatson log? What is the PIM type?

OPC is a critical process on PG and if it exits, it may bring PG in rolling reboot (stopshut). You might want to open TAC case with all the logs.

Thanks,

Sandeep

[ccarmichael]

These are ACD Pim's and there are no DrWatson Logs on the boxes. This problem is very evasive. It has occured at multiple sites at different times and we have had hotfixes which resolved the problem at one site but will cause issues at another. This has made it necessary for us to run multiple hotfix levels throughout the environment. At this time only one site is being affected by these errors but it bounces several times a day. Is anyone else having this problem and also having problems with hotfixes from site to site trying to fix it? MIS is configured on these boxes and IVR PG's are also co-located at the site. Thanks

[lenny247]

Usually translation post-routing is affected by a flip in our environment. As we use redundant PGs, the fail-over is quite seemless and pre-routes work as normal.

In our environement, we have had multiple idle PGs (as opposed to active ones) located in different cities, reboot at the exact same time. Network or application seem to be only possiblity. No official word from Cisco.

[Birendra Hansda]

Hi

Even I encountered a similar event with ICM 7.2.7 but it was with pim.

The nm logs show

Wait returned for process pim1 pid 6f8 handle 2fc wait handle 2fc

pg65B-nm Process pim1 on ICM\fedex\PG65B went down for unknown reason. Exit code 0xc0000005. It will be automatically restarted.
pg65B-nm Process pim1 on ICM\fedex\PG65B is down after running for 1618931 seconds. It will restart after delaying 10 seconds for related operations to complete.

Is there any way to trace what pid 6f8 and 2fc are?

Thanks

Birendra Hansda