08-29-2016 09:48 AM - edited 03-11-2019 12:02 AM
Hi ALL ,
I have a set up for 12000 active connections approx. with nearly 19000 endpoints in VM based environment . No of nodes as below
1) One Primary admin node ( 20 GB RAM)
2) 3 PSN ( RAM :20 GB each)
3) One MNT (RAM :16 GB )
Above set up is active , it is in production we need to upgrade to mentioned version without downtime . After reviewing all documentation , I found that downtime is applicable with nearly 1.5-2 hrs or atleast 30-40 minutes for PSN and 1 hr for MNT , we are not using profiling , just basic dot1x/MAB
We do have a admin node for secondary but it is not added in deployment , Please clarify best practices and steps to upgrade with/without secondary admin node and applicable downtime
++Important : Since you notice we are running large deployment (12000 connections ) with less resources , we are planning to increase the RAM as well
In my opinion /searches done on webpages , it is better to upgrade the RAM first and then plan upgrade . I have few questions as it is not documents in cisco guidelines
1) In case we upgrade the RAM , is it true that we have to do a fresh installation for ISE VM ( new admin/PSN/MNT) as it may show upgraded RAM but will not use the same with old VM , for disk it is mentioned that we have to do fresh installation in case we plan to upgrade
2) I can see the CPU is nearly 3% but I am regularly getting alerts on load average , is it calculated based on MNT or overall . What is the command to check the peak value of CPU and number of cores allocated , we are having 6 cores on ISE (show inventory)
Thanks in advance
Regards,
Sam
Solved! Go to Solution.
08-30-2016 09:02 PM
08-31-2016 07:10 AM
Hi Sameer,
Let me know if you have any further concerns. Otherwise please close by rating it.
Regards
Gagan
08-29-2016 10:15 AM
Hi Sam,
1) Yes, you have to build VM from the scratch again in order to get ISE to inherit new disk in consideration.
2) The command to check CPU utilization is "tech top". Also can use "tech mpstat" for CPU usage.
*Note: Once running tech top press "1" to see CPU by core or you will end up seeing CPU percentages above 100% for multi-core systems like ISE.
The technical definition of load average is the calculation of the 1,5, and 15 minute average of the number of processes, Running (R) or Runnable (D), per run queue. This means how many process are waiting for CPU time as opposed to CPU precentage which is how much of the CPU is currently being used. Load average can be displayed on ISE with the "tech top" command from the CLI:
jesse-dunkel/admin# tech top
top - 11:15:46 up 10 days, 6 min, 2 users, load average: 0.00, 0.02, 0.00
Tasks: 192 total, 1 running, 191 sleeping, 0 stopped, 0 zombie
Cpu(s): 2.4%us, 0.4%sy, 0.0%ni, 97.1%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 3924380k total, 3800900k used, 123480k free, 314648k buffers
Swap: 8185112k total, 752228k used, 7432884k free, 956256k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2886 oracle 20 0 1439m 57m 52m S 5.3 1.5 3:00.64 oracle
5782 iseadmin 20 0 5369m 740m 13m S 4.0 19.3 195:16.15 jsvc
2514 oracle 20 0 1346m 43m 40m S 0.3 1.1 45:37.60 timestensubd
6006 isemtlog 20 0 3937m 267m 8704 S 0.3 7.0 16:38.09 java
6060 isemtlog 20 0 508m 1408 1276 S 0.3 0.0 35:57.73 Decap_main
14273 oracle 20 0 1441m 48m 42m S 0.3 1.3 0:10.61 oracle
31021 admin 20 0 116m 1548 840 S 0.3 0.0 0:00.02 screen
31054 root 20 0 15028 1376 988 R 0.3 0.0 0:00.08 top
In the example above the 1 minute load average is 0.00, the 5 minute load average is 0.02, and the 15 minute load average is 0.00. This system is has very low utilization so when would this become a problem? This number is a representation of how many process are in each run queue and each CPU has a run queue. A 100% utilized system therefore would mean that each CPU on the system is always handling a process, in the case of a 4 CPU system this would be a load average of 4.00, on and 8 CPU system it would mean an load average of 8.00. It isn't until the load average eclipses the number of CPUs that the in theory the system is overloaded, though this leaves no room for spikes in activity so the generally accepted safe value would be between 70% and 80% of the number of CPUs on the server.
A great analogy for load average is to look at a highway. In our above example, this is a 4 lane highway. Each lane is a CPU queue, if there is bumper to bumper traffic but each lane is still moving at 65mph (posted speed limit) the highway is 100% utilized which would equate to a load average of 4. If another car, think process needing cpu, tries to get onto the highway one of the lanes needs to slow down because there is no room for that car on the highway and the load average has now gone over 100% which would mean the load average is now over 4.
Hope it helps!!!
Regards
Gagan
08-30-2016 11:14 AM
Thanks ,
Please clarify on the upgrade part and if we upgrade RAM do we need to have a fresh installation for VM as documented for Disk in case we need to upgrade
Regards,
Sameer
08-30-2016 11:37 AM
Hi Sameer,
Yes, VM never takes disk addition in the running state. You have to build VM from the scratch again.
Let me know if you have any further questions.
Regards
Gagan
08-30-2016 01:30 PM
Thanks Gagan ,
I assuming the above answer in context of RAM as well , Is there any documentation you can point to which clearly specifies for the same
Secondly , I need to know the upgrade steps recommended with minimum downtime
regards,
Sameer
08-30-2016 02:13 PM
Hi Sameer,
Yes, earlier I mentioned about disk but forgot to mention about RAM.
RAM will take affect but disk not.
Please find document for upgrade
http://www.cisco.com/c/en/us/td/docs/security/ise/1-4/upgrade_guide/b_ise_upgrade_guide_14/b_ise_upgrade_guide_14_chapter_010.html#ID20
Hope it works!!!!
Regards
Gagan
08-30-2016 09:02 PM
08-31-2016 07:10 AM
Hi Sameer,
Let me know if you have any further concerns. Otherwise please close by rating it.
Regards
Gagan
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide