ISE PSN node dependencies

[LAN team]

Hello,

We're currently facing an issue with a PSN node with a CPU overusage, we have to reload it almost each days.

It should be a hardware issue or bug issue.

But waiting to solve the hardware issue, we want to isolate this node, but when we shut the node or stop application, the other PSN node which belong to the same group node undergo a CPU and memory increase until the first PSN node is fully restarted.

I need to precise, it's not related to an authentication load issue, as it happens even when there's no traffic.

1) In my mind, the sync is done between the PAN and the PSN, then I don't understand what happens. 

2) Is it necessary to isolate a PSN node to remove it from the PSN group ?

Thank you, 

[marce1000]

 - When this happens ,check CPU usage on the remaining PSN with : 

ise/admin# show cpu usage

M. 

[LAN team]

Sorry, not solved, it's a mistake.

Thank you for your answer.

I've just stopped ISE application service on the first PSN.

Here's what happen on the second PSN in the group node :

PA4-RNH-ISEPS02/admin# sh cpu usage

ISE Function                         % CPU Usage           CPU Time     Number o
f threads
--------------------------------------------------------------------------------
-------------
Profiler                                  1.55           4138:13.42
 57
Admin Process JVM Threads                 1.45           3858:28.52
 28
Profiler Database                         0.46           1221:14.90
  4
Database Server                           0.09            252:22.74
147 processes
Quartz Scheduler                          0.06            171:02.83
 29
Docker Daemon                             0.01             19:24.32
 26
Admin Webapp                              0.01             18:54.87
 39
RMI Services                              0.01             13:21.19
514
Certificate Authority Service             0.00             11:55.83
 54
NSF Persistence Layer                     0.00             10:08.14
 51
Database Listener                         0.00              2:26.32
  2
Syslog Processor                          0.00              1:59.71
  4
Guest Services                            0.00              0:19.02
  1
Message Queue                             0.00              0:00.00
  0
BYOD Services                             0.00              0:00.00
  1
Miscellaneous services                    2.22           5927:11.35
569
M&T Session Database                       N/A
M&T Log Processor                          N/A
M&T Log Collector                          N/A
ISE Indexing Engine                        N/A
Identity Mapping Service                   N/A
SXP Engine Service                         N/A
Threat Centric NAC MongoDB Container       N/A
Threat Centric NAC RabbitMQ Container      N/A
Threat Centric NAC Core Engine Container   N/A
Vulnerability Assessment Database          N/A
Vulnerability Assessment Service           N/A
WIFI Setup                                 N/A

PA4-RNH-ISEPS02/admin# sh inventory 

NAME: "SNS-3595-K9 chassis", DESCR: "SNS-3595-K9 chassis"
PID: SNS-3595-K9       , VID: A0  , SN: FCH2134V08C
Total RAM Memory: 65701636 kB
CPU Core Count: 16
CPU 0: Model Info: Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
CPU 1: Model Info: Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
CPU 2: Model Info: Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
CPU 3: Model Info: Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
CPU 4: Model Info: Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
CPU 5: Model Info: Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
CPU 6: Model Info: Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
CPU 7: Model Info: Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
CPU 8: Model Info: Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
CPU 9: Model Info: Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
CPU 10: Model Info: Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
CPU 11: Model Info: Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
CPU 12: Model Info: Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
CPU 13: Model Info: Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
CPU 14: Model Info: Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
CPU 15: Model Info: Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz

PA4-RNH-ISEPS02/admin# tech top
Invoking tech top. Press Control-C to interrupt.

top - 07:48:20 up 11 days, 14:35,  2 users,  load average: 16.93, 17.13, 14.40
Tasks: 422 total,   6 running, 416 sleeping,   0 stopped,   0 zombie
%Cpu(s): 64.3 us,  5.3 sy,  0.0 ni, 29.4 id,  0.0 wa,  0.0 hi,  1.0 si,  0.0 st
KiB Mem : 65701636 total,  9609260 free, 17633960 used, 38458416 buff/cache
KiB Swap:  8191996 total,  8187900 free,     4096 used. 38305064 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                              
  316 iseadmi+  20   0   24.1g   7.9g  64440 S 759.6 12.5  15149:34 jsvc                                                                                                                                 
 7331 iseprof+  20   0  960588 821440    476 R  98.0  1.3   0:05.24 redis-server                                                                                                                         
25719 iseprof+  20   0  960444 822036   1328 R  34.4  1.3   1222:35 redis-server                                                                                                                         
15962 iserabb+  20   0 2101528 336620   3268 S  21.2  0.5   1465:17 beam.smp    

We can see that first process use 759% of CPU : 

316 iseadmi+ 20 0 24.1g 7.9g 64440 S 759.6 12.5 15149:34 jsvc 

But as soon as the process will be fully restared on the first PSN, this process on the second PSN come decrease to normal value.

[Marcelo Morais · Accepted solution]

Hi @LAN team ,

 beyond @marce1000 said ... compare your CPU Type (command bellow) with Cisco ISE HW Platforms:

ise/admin# show inventory
NAME: "ISE-VM-K9 chassis", DESCR: "ISE-VM-K9 chassis"
PID: ISE-VM-K9 , VID: V01 , SN: <ISE SN>
Total RAM Memory: <ISE RAM>  kB
CPU Core Count: 24
CPU 0: Model Info: <Intel Model>
...
CPU 23: Model Info: <Intel Model>
Hard Disk Count(*): 1
Disk 0: Device Name: /dev/sda
...

and use the command bellow to check not only the CPU but also the Load Average (1min, 5min and 15min): 

ise/admin# tech top
top - 11:46:33 up 4 days, 15:08, 2 users, load average: 5.39, 4.57, 4.26
Tasks: 531 total, 1 running, 530 sleeping, 0 stopped, 0 zombie
%Cpu(s): 13.6 us, 2.2 sy, 0.0 ni, 83.5 id, 0.0 wa, 0.0 hi, 0.7 si, 0.0 st
...

Hope this helps !!!