CSCvn91978 - jsvc process taking up high amount of swap space

[zstamm]

I have an ISE instance that is using quite a bit of swap space, and it is causing disk latency on it's host machine due to large amounts of drive access.  It looks like CSCvn91978 to me, but I can't be sure. 

Is this a normal amount of swap usage for ISE?  If not, what is?  What other troubleshooting steps should I take?

top - 17:20:16 up 26 days, 2:46, 2 users, load average: 12.07, 12.03, 11.93
Tasks: 683 total, 2 running, 679 sleeping, 2 stopped, 0 zombie
%Cpu(s): 40.0 us, 16.7 sy, 0.0 ni, 36.7 id, 5.8 wa, 0.3 hi, 0.4 si, 0.0 st
MiB Mem : 15830.8 total, 446.9 free, 10019.1 used, 5364.8 buff/cache
MiB Swap: 8000.0 total, 2306.0 free, 5694.0 used. 3648.0 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND SWAP
24089 iseadmi+ 20 0 13.0g 1.5g 43140 S 711.6 9.5 238581:13 jsvc 1.1g
30402 iseca20 0 6796716 448124 8296 S 0.0 2.8 60:09.59 jsvc455208
24344 isemtlo+ 20 0 8402416 3.1g 433652 S 6.0 19.9 1397:51 java 285740
34242 iseadmi+ 20 0 9002824 136076 8524 S 0.3 0.8 295:55.24 java 269204
73892 iseadmi+ 20 0 10.7g 136436 8440 S 0.7 0.8 306:49.83 java 259188
39686 iseadmi+ 20 0 9003824 139124 8516 S 0.7 0.9 292:23.80 java 251892
41481 root 20 0 9002796 125876 8324 S 2.3 0.8 300:34.37 java 247880
32442 iseadmi+ 20 0 9069436 124248 8416 S 0.3 0.8 301:10.95 java 236784
38859 iseadmi+ 20 0 10.7g 151612 8732 S 0.3 0.9 295:21.44 java 212232
20297 isesch 20 0 9138852 100380 9176 S 0.3 0.6 34:28.37 java 130968

Cisco Application Deployment Engine OS Release: 3.1
ADE-OS Build Version: 3.1.0.135
ADE-OS System Architecture: x86_64

Copyright (c) 2005-2021 by Cisco Systems, Inc.
All rights reserved.
Hostname: XXXXXXXXXX

Version information of installed applications
---------------------------------------------

Cisco Identity Services Engine
---------------------------------------------
Version : 3.1.0.518
Build Date : Mon Aug 9 20:28:55 2021
Install Date : Tue Sep 13 21:55:41 2022

Cisco Identity Services Engine Patch
---------------------------------------------
Version : 3
Install Date : Thu Sep 15 03:19:18 2022

[Greg Gibbs]

It might be more likely you're hitting this bug - https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwh25160

I don't know that we have a documented 'normal' range for swap.

It looks like you're running patch 3, which is over 2 years old, so your first step should be updating to the latest patch.

If you are still seeing the same symptoms after updating to the latest patch, then you would be best opening a TAC case to investigate further.

[zstamm]

Thank you!  I will give that a try.

[Feds]

Hi @zstamm ,

How did it go?
We have started seeing this behaviour for all three PSN nodes (but not the two PAN/MnT) after the installation of 3.2 Patch 7- previoulsy running 3.2 patch 6, no issues.
Process that holds memory is "jsvc".
TAC hasn't commented yet as CPU/Memory reservation is only partially in place - just before installing patch 7 all ISE VMs had reserved resources but for "small" deployment only.
This deployment also uses PassiveID and the customer has highlighhted the AD accoubnt gets locked out when either of the PSNs are using > 95% of memory.  
Thanks @Greg Gibbs  for pointing out that bug ID.

Thank you
Feds