topic Re: CSCwk06817 - Known OS Bug causing swap to increase on ISE nodes in Network Access Control

CSCwk06817 - Known OS Bug causing swap to increase on ISE nodes

amunozmo — Thu, 17 Jul 2025 02:59:30 GMT

Linear Swap Memory growth on two ISE nodes cluster having all personas on 3715 appliances. High swap and RAM memory usage, memory leaks in jsvc and Java admin processes, the critical point here for this discussion is that this issue is still happening on ISE version 3.3 patch 6 when the known fixed Release is 3.3 patch 4 according to this bug https://bst.cisco.com/bugsearch/bug/CSCwk06817

How is possible to still have linear Swap Memory growth on ISE nodes when the defect was fixed on ISE 3.3p4? TAC confirmed evidence of defect CSCwo82042 (LSD MAC handling) in logs but there is no evidence of what is causing swap memory leak and growth.

Re: CSCwk06817 - Known OS Bug causing swap to increase on ISE nodes

Greg Gibbs — Thu, 17 Jul 2025 06:28:41 GMT

This is not something the Community can help with. You would need to work with TAC to investigate and determine if a new bug needs to be opened.

Re: CSCwk06817 - Known OS Bug causing swap to increase on ISE nodes

robertwilkinson1 — Thu, 17 Jul 2025 12:56:56 GMT

I am having the same issue on 3.3p6 and I can watch the increase in memory and swap climb slowly over time. it looks to be approx 1% increase give or take each day. Initially I did a reload on the entire deployment just to make sure it wasnt something funky going on. but since it is still increasing slowly. im going to wait until it hits near 70% usage before creating a new TAC on it.

Re: CSCwk06817 - Known OS Bug causing swap to increase on ISE nodes

amunozmo — Thu, 17 Jul 2025 13:52:47 GMT

Hi Greg,

Thank you very much for your prompt response and thoughtful comments.

While I fully understand and appreciate your perspective, and in fact, we've been actively working with the TAC to address this issue, I started this conversation within the community to help raise awareness and encourage a broader, constructive dialogue. The idea was to understand whether others in the community are also observing a progressive increase in swap memory usage over time, despite the workaround that was expected to resolve this in version 3.3 Patch 4, and my customer is now in version 3.3 Patch 6.

I believe this could be a valuable opportunity to share insights and to have a different approach that may benefit customers globally. After all, if someone in the community has identified the root cause and is willing to share a solution, it would greatly support others facing similar challenges.

Thank you once again an i would love to hear your thoughts on this approach.

Re: CSCwk06817 - Known OS Bug causing swap to increase on ISE nodes

Marcelo Morais — Fri, 18 Jul 2025 06:08:41 GMT

Hi @amunozmo ,

I have Customers with ISE 3.3 P4 and ISE 3.3 P6, using RADIUS only, with 3755 and 3795 (VMs) and I have not noticed the memory leak related issues that we had in ISE 3.3 P2.

Are you able to test your scenario on a 3755 VM ?

Best regards.

Re: CSCwk06817 - Known OS Bug causing swap to increase on ISE nodes

scastrilonospina — Mon, 21 Jul 2025 19:50:05 GMT

Hi,

We have the Same problem 3 months ago, and we are working with TAC and the only solution today is ADD 2 nodes more to separate PAN and MnT, we will update to 3.4 patch 2 to test, if the issues get solved i will tell you. Today we are running ISE 3.3p5, and we have to reload each 15 day aprox.

Re: CSCwk06817 - Known OS Bug causing swap to increase on ISE nodes

JuanVelez — Mon, 21 Jul 2025 20:15:39 GMT

I've been experiencing this same issue consistently. I've been monitoring system processes daily using the "tech top" command and have noticed memory usage increasing by approximately 100 MB per day. The jsvc process appears to be the primary contributor to this memory drain.

Over time, I've opened multiple support cases regarding this problem. The recommended solutions have included disabling certain features or applying various patches, but none have resolved the issue. The most recent explanation I received pointed to system sizing. We're currently running two SNS-3715 nodes handling RADIUS, TACACS, and Posture Assessment. No self-provisioning, no CWA, no pxGrid, no SGT, no API integrations. While TAC believes the hardware is insufficient, the official sizing guide indicates that our deployment should be well within supported limits, even oversized for our needs.

We’re currently on version 3.3p5. I plan to test 3.4p2 soon to see if the problem is addressed in that release.

Has anyone else encountered a similar memory problem with jsvc or found a lasting fix?

Re: CSCwk06817 - Known OS Bug causing swap to increase on ISE nodes

Marcelo Morais — Tue, 22 Jul 2025 00:21:44 GMT

Hi @JuanVelez ,

please take a look at: ISE - What we need to know about SNS / VM, pay special attention to the Particularities topic.

Are you "running the SNS-3715" as an Appliance or a VM ?

Hope this helps !!!

Re: CSCwk06817 - Known OS Bug causing swap to increase on ISE nodes

Marcelo Morais — Tue, 22 Jul 2025 00:23:39 GMT

Hi @scastrilonospina ,

what is your Hardware ? Appliance or VM ?

Note: I also recommend you the: ISE - What we need to know about SNS / VM.

Hope this helps !!!

Re: CSCwk06817 - Known OS Bug causing swap to increase on ISE nodes

JuanVelez — Tue, 22 Jul 2025 02:13:32 GMT

Hi @Marcelo Morais, thanks a lot for replying!

We're running ISE on two physical SNS-3715 appliances (no virtual machines involved).

I reviewed the section you referenced: "SNS (Secure Network Server) Appliance > Particularities", and compared it with the official "Performance and Scalability Guide for Cisco Identity Services Engine", which I often read thanks to this issue. According to that guide, our deployment falls under the "small" category. We handle around 2000 to 3000 concurrent sessions at peak, and according to the sizing tables, a shared PSN on an SNS-3715 is rated to support up to 25000 sessions. Both our nodes run PAN/SAN, PSN, and MnT personas.

In theory, we're well under capacity — operating at <20% of the rated limit — yet we’re seeing continuous memory growth, mostly tied to the jsvc process.

So, this leads me to some broader questions:

Under what real-world conditions does a two-node architecture for small deployments with shared personas actually remain stable long-term?
Are there memory behavior differences between physical appliances and virtual machines that are not reflected in the official documentation?
Could there be hardware or firmware issues in the SNS‑3715 platform contributing to memory leaks?
Could disk I/O issues on the SNS‑3715 appliances (e.g., logs, database operations, backups) gradually increase memory consumption over time?
Are there specific patches or configuration adjustments for the jsvc process recommended for physical appliances, similar to those applied in virtual machine deployments?

Even though we appear to be within specs, we’re still encountering this issue. I really appreciate the pointer — and I'd welcome any further insights from you or others in the community

Re: CSCwk06817 - Known OS Bug causing swap to increase on ISE nodes

Marcelo Morais — Tue, 22 Jul 2025 15:55:28 GMT

Hi @JuanVelez ,

keep in mind the following, described in the Particularities part of the ISE - What we need to know about SNS / VM.

" ... Additional hardware resources such as RAM, CPU, or HDD cannot be added to an SNS Appliance, but additional Power Supplies can be ordered separately for SNS 3615 and SNS 3715 ... "

" ... The SNS 3x15 acting as PAN/MnT are highly recommended for either RADIUS only or TACACS+ only workloads. If a Deployment requires both RADIUS and TACACS+ at scaled workloads, it is recommended to use SNS 3x55 or higher models ... "

In other words ... it is easier to make some changes to a VM than to an Appliance, for testing purpose (in the past, we managed to find a very annoying issue that became a Cisco Field Notice through VM testing, please take a look at ISE - Slow Replication).

About your questions ...

Yes, a Small Deployments with Two-Node Architecture can remain stable long-term, but let's take a look at your case with RADIUS & TACACS+.
IMHO, no, as far as differences in the Memory behavior between Appliances and VMs.
Memory Leaks are primary a Software issue, but keep your Hardware update is important (please take a look at the Update part of the ISE - What we need to know about SNS / VM).
Yes, actions like generate logs, backups, Support Bundle, and others, increases Memory usage.
Let's talk 1st about configuration:
1. At Operations > Reports > Reports > Endpoints and Users > Authentication Summary, what is your Total AuthC per Day (at Authentications by Day and Quick Link) ?
2. At Administration > System > Settings > Profiling, your MFC Profiling and AI Rules and Endpoint Analytics Settings are disabled ?
3. At Operations > System 360 > Settings, can you disable Monitoring and Log Analytics just for testing ?
4. At Administration > System > Maintenance > Operational Data Purging, what about your Database Utilization ?

Hope this helps !!!

Re: CSCwk06817 - Known OS Bug causing swap to increase on ISE nodes

Marcelo Morais — Wed, 23 Jul 2025 15:19:00 GMT

Hi @amunozmo , @JuanVelez and @scastrilonospina ,

keeping in mind the CSCwk06817 Known OS Bug causing swap to increase on ISE Nodes .

When you run the tech top command, what is the result of your SNS-3715 ?

Examples of a 3795-VM-PAN and 3755-VM-PSN:

3795-VM-PAN/admin# tech top
 Invoking tech top. Press Control-C to interrupt.
 top - 11:41:47 up 11 days, 16:44, 1 user, load average: 0.54, 0.71, 0.82
 Tasks: 819 total, 1 running, 818 sleeping, 0 stopped, 0 zombie
 %Cpu(s): 1.4 us, 0.4 sy, 0.0 ni, 97.9 id, 0.1 wa, 0.0 hi, 0.1 si, 0.0 st
 MiB Mem : 257403.3 total, 161230.0 free, 45260.3 used, 50913.0 buff/cache
 MiB Swap:   8001.0 total,   8001.0 free,     0.0 used. 190306.5 avail Mem 
 ...

3755-VM-PSN/admin# tech top
 Invoking tech top. Press Control-C to interrupt.
 top - 11:43:57 up 11 days, 15:29, 1 user, load average: 1.51, 1.63, 1.61
 Tasks: 878 total, 2 running, 876 sleeping, 0 stopped, 0 zombie
 %Cpu(s): 7.9 us, 3.1 sy, 0.0 ni, 88.4 id, 0.0 wa, 0.2 hi, 0.3 si, 0.0 st
 MiB Mem : 96124.0 total, 46568.2 free, 30508.5 used, 19047.3 buff/cache
 MiB Swap:  7999.9 total,  7999.9 free,     0.0 used. 58621.5 avail Mem
 ...

Best regards.

Re: CSCwk06817 - Known OS Bug causing swap to increase on ISE nodes

scastrilonospina — Thu, 24 Jul 2025 14:05:27 GMT

Hi Marcelo,

Appliance

Re: CSCwk06817 - Known OS Bug causing swap to increase on ISE nodes

Marcelo Morais — Thu, 24 Jul 2025 18:21:29 GMT

Hi @scastrilonospina ,

thanks.

What is the output of the tech top command before reloading the Node ?

Re: CSCwk06817 - Known OS Bug causing swap to increase on ISE nodes

MatthewA — Wed, 30 Jul 2025 21:10:32 GMT

My 3715 pair is having this same problem, on 3.3 patch 7 now. SWAP memory stays stable as long as I have SXP and LogAnalytics disabled. I keep tech top running in case it starts spiraling again but TAC effectively said the only fix is 3.4 (patch 2 I think?). Not really willing to do yet another upgrade to fix this issue (been working with TAC since 1/2/25 on it) unless I know it will work this time. Holding it stable right now with those 2 features disabled. Have you come up with a solution to yours?

Also seems to be specific to device pairs running all 3 personas but haven't been able to test with adding VMs yet to see if the problem continues.

Re: CSCwk06817 - Known OS Bug causing swap to increase on ISE nodes

Marcelo Morais — Wed, 30 Jul 2025 22:08:09 GMT

Hi @MatthewA ,

at Administration > System > Settings > Profiling, are your MFC Profiling and AI Rules also disabled ?

If the the answer is No, could you please disable it and check if there is any improvement ?

Hope this helps !!!

Re: CSCwk06817 - Known OS Bug causing swap to increase on ISE nodes

MatthewA — Thu, 31 Jul 2025 13:38:01 GMT

Good morning Marcelo!

Thanks so much for the response! I have disabled MFC Profiling and AI Rules (didn't realize it was on and don't use it anyway). No immediate change in tech top but I will be working on it all day today and can watch. I will enable SXP again later to see if SWAP starts to spiral again.

Re: CSCwk06817 - Known OS Bug causing swap to increase on ISE nodes

Marcelo Morais — Thu, 31 Jul 2025 15:06:02 GMT

@MatthewA ,

excellent ...

MFC Profiling and AI Rules had issues In ISE 3.3 P2, which were solved in ISE 3.3 P4.

It's worth a shot to disable it and test it (even if you are on ISE 3.3 P7).

Best regards

Re: CSCwk06817 - Known OS Bug causing swap to increase on ISE nodes

MatthewA — Thu, 31 Jul 2025 20:18:03 GMT

So far it doesn't seem to be part of the problem. No change in memory usage outside of a small bump (couple hundred mbs on average). I re-enabled SXP on one of my servers to see if SWAP continued to spiral and immediately saw my available mem tank and SWAP start rising.

I appreciate checking that though! This has made expanding into TrustSec very difficult.

*for time context that tech top was taken roughly 15min after enabling SXP and has risen an additional 150mbs in typing this so I am re-disabling SXP*

Extra notes that might help others:

TAC and I originally believed the bug to be related to Log Analytics, disabling this stabalized the environment until enabling SXP caused the same behavior.

While typical SWAP usage will delete and release, the SWAP increase seen from enabling these 2 services does not recover on its own and require a restart to clean out.

Disabling/Enabling SXP provides an instant response on SWAP usage (within a minute to see effects)

Re: CSCwk06817 - Known OS Bug causing swap to increase on ISE nodes

Marcelo Morais — Mon, 04 Aug 2025 14:55:13 GMT

Hi @MatthewA ,

you said that " ... Disabling/Enabling SXP provides an instant response on SWAP usage (within a minute to see effects) ... ".

This issue occurs even if you do not have any SXP Configuration (at Work Centers > Trustsec > SXP Devices) ?

Best regards