ISE MnT Node purge running for 6 days straight

[Leroy Plock]

Hi all,

Our ISE 1.1.4 patch 2 MnT node appears to be stuck in DB purge. I am getting e-mail alerts that say "Hourly purge skipped as purge is already running." Also, when I try to run a backup of the MnT node I receive the message. "Cannot submit full backup when data purging is in process."

We had received the "maximum open cursors exceeded" error. When that I happened I re-synchronized the deployment which re-started the services on the MnT node. This cleared the open cursors error but left us where we're at now. I was hoping it would clear itself with time, but it hasn't.

All I can think to do is restart the ISE services on the MnT node, but I'm a little worried about what might happen if I do that in the middle of a purge. Of course we don't have a recent backup of MnT (see above) and I would not like to lose the historical data.

I haven't opened a TAC yet, will if no answers here. We can't patch or upgrade above 1.1.4 patch 2 because we're waiting on a fix for an unrelated bug.

Any ideas appreciated, thank you.

[Naresh Ginjupalli]

Hi Leroy Plock,

Let me explain you the root cause of experiencing the "maximum open cursors exceeded" .

In ISE 1.1.4 Patch 1 we have introduced a new hourly purge Mechanism. Due to this feature we are experiencing this open cursors issue.

The issue 'ORA-01000: maximum open cursors exceeded' is caused because of the feature of HOURLY PURGE introduced in ISE 1.1.4 Patch 1.

On each hour a purge process is triggered and a connection to the MNT database is opened. As per the  logic this opened connection should be closed right immediately after the transaction of purge process is completed. But this opened connection is not being closed and thus in a day 24 connections are kept opened.

In oracle we have set the count of 1500 open cursors and so the database will not open the cursor count beyond this value.

With the above said 24 connections opened every day this 1500 cursors will be consumed within 62 days (1500/24) and this error will then populate.

If we restart the MNT node once in a month these cursors will get freed up and then will not see this open cursor issue. This defect is addressed in ISE 1.1.4 Patch 4.

The defect for this issue is CSCuh70984

[Naresh Ginjupalli · Accepted solution]

Hi Leroy Plock,

Let me explain you the root cause of the experiencing the "Hourly purge skipped as purge is already running."

While the hourly purge process is triggered the MNT node has experienced the open cursors issue and thus as there are no threads to open/update a communication of the purge complete process to the purge logic, the hourly purge is struck in the running state.

The can be resolved by changing the already running purge process to completed status. By doing this the next hourly purge process will get triggered automatically.

The alarm you are seeing that purge process is already running is the side effect of the above said defect CSCuh70984.

Please open a SR with TAC and they will assist you in addressing the fix for the purge already running alarms.

The reason for asking to open a TAC case is that , the fix involves in running the database SQL queries against your MNT database and once this is done , you need to restart the services on the MNT node.

[Leroy Plock]

OK thanks, we'll call TAC.