topic Re: Database Server not-running Cisco ISE in Network Access Control

Database Server not-running Cisco ISE

oum-odom — Thu, 30 Oct 2025 02:31:19 GMT

Dear Cisco ISE lover,

Recently we were testing lab to re-issue related to Cisco ISE "Database Server not-running",
while on user on Secure client UI "Posture Failed due to server issues"

We configure the Distributed Deployment 2 nodes PAN and SAN.
ISE v3.1 P10
Both PSN are active-active during issue

+ Node 1 (PAN)
Database Server not-running
- oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:509)
- root: notice:[application:operation:isehourlycron.sh] ISE Database is not running, aborting cleanup…

+ Node 2 (SAN)
Database Server running

So, we used the workaround solution to resolve by stop and start application ise on issue node (PAN).
Then everything was okay.
While we open case with TAC, they suspect this was memory exhaustion.
By the ways, we back date to report of healthy report on ISE. there was not any peak performance especially with memory.

Please kindly share us your experience and provide the suggest ensuring never happen again in future.
Thank you,

Re: Database Server not-running Cisco ISE

Kasun Bandara — Thu, 30 Oct 2025 03:18:47 GMT

@oum-odom hi, as TAC explained this can be due to memory exhaustion or memory leak bugs. i were not able to find specific bug in bug tool for exact 3.1 patch 10. but there is few for other versions. you can check the bug list here.

Bug Search Tool

additionally, i am personally recommending to test latest ISE major version if you are planning to test and deploy in production. (i am not sure about the current use case at its on LAB). 3.1 and 3.2 versions are to be go End of support in 2027 Nov and development will stop 2026 Nov.

Re: Database Server not-running Cisco ISE

oum-odom — Fri, 31 Oct 2025 03:09:44 GMT

Thank @Kasun Bandara for your comment as TAC can't define the root cause yet while required to wait until issue again and Generate heap & Thread dump.

It doesn't make sound while we need to wait until issue happen.
First, we think it was the bug, bug was resolved by exiting v3.1 P10
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwm48867

Re: Database Server not-running Cisco ISE

Marcelo Morais — Mon, 03 Nov 2025 04:18:30 GMT

Hi @oum-odom

both commands are workarounds for Memory Leaking:

ise/admin# reload

and

ise/admin# application stop ise
ise/admin# application star ise

also use the following command to check for Memory Leaking:

ise/admin# tech top
 top - 12:11:18 up 26 days, 16:28, 2 users, load average: 4.01, 4.53, 4.92
 Threads: 1534 total, 1 running, 1533 sleeping, 0 stopped, 0 zombie
 %Cpu(s): 14.5 us, 2.5 sy, 0.0 ni, 82.4 id, 0.1 wa, 0.0 hi, 0.5 si, 0.0 st
 KiB Mem : 98832816 total, 708812 free, 11291044 used, 86832960 buff/cache
 KiB Swap: 8191996 total, 8180988 free, 11008 used. 73835392 avail Mem 
 ...

What HW are you using (SNS 36xx, SNS 37xx, ...) ? Disk size ?

Hope this helps !

Re: Database Server not-running Cisco ISE

Aref Alsouqi — Mon, 03 Nov 2025 10:54:23 GMT

If I were you I would just go ahead and spin up a new node and replace it with the one that is giving that database issue. I think that would be a cleaner and faster approach to sort it out.

Re: Database Server not-running Cisco ISE

oum-odom — Tue, 18 Nov 2025 04:40:00 GMT

@Marcelo Morais appreciate your support.
After working with tac, this is the bug CSCwr50566

Re: Database Server not-running Cisco ISE

oum-odom — Tue, 18 Nov 2025 04:43:31 GMT

appreciate your support @Aref Alsouqi
After working with tac, this is the bug CSCwr50566 also suggest to upgrade to 3.4 latest patch.

Re: Database Server not-running Cisco ISE

Marcelo Morais — Tue, 18 Nov 2025 11:44:27 GMT

@oum-odom ,

thanks for the feedback !

What HW are you using (SNS 36xx, SNS 37xx, ...) ? Disk size ?

ISE 3.3 P8 was released, but reading the CSCwr50566 Database server not running after oracle crash, it looks like that 3.3 P8 still have the issue because:

" .. this issue is not applicable to 3.4 where we already have updated Oracle version running ... "

Can you confirm this with the TAC ?

Re: Database Server not-running Cisco ISE

vmandri — Thu, 22 Jan 2026 19:36:18 GMT

I had a pair of fresh new 3855 ISE servers. They came with ISE3.3.x so i wanted to upgrade to 3.4 and then to patch 3. I upgraded from 3.3 to 3.4 without any issues services came up normally all good. then i did the patch 3 upgrade after the restart the services never came up only Database Listener was running everything else stopped or disabled, application start ise failed, rebooting nothing worked. So, i blamed the patch for the failure so i issued the command remove patch 3, after some time that failed too. i wasn't to worry as it was a fresh server i could always reimage or reset the application. I did a final reboot and what do you know the services came up everything ok. I would pass this a lucky accident, but the weird thing is that literally the exact same thing happened with the second server. the same upgrade procedure the same problem and the same weird solution. That was so weird i had to share it.

Re: Database Server not-running Cisco ISE

Marcelo Morais — Fri, 23 Jan 2026 01:06:18 GMT

@vmandri ,

thanks for the feedback !

Remember that

SNS 38xx supports ISE 3.3 P7+, ISE 3.4 P4+ and ISE 3.5+ (ISE - What we need to know about SNS / VM)
ISE 3.4 P4 solves the following important issues:

CSCwo01630 ISE AD (lwsmd) stuck at "Updating" and consuming 90-100% CPU even after CSCwh72754

CSCwp06928 Changing Network Device settings makes the RADIUS shared secret change

CSCwp01928 NAD Port ID column is blank in the Context Visibility table

ISE 3.4 P4 has the following important issues open:

CSCwr96966 Endpoint ownership lookup causing latency under certain scenarios

CSCws76788 ConfD failed to start on 3.4 P4 making the node unavailable

Hope this helps !