01-15-2018 11:48 PM - edited 03-01-2019 06:21 PM
I have update our Cisco Prime Infrastructure from 3.1 to 3.3 and now facing very slow performance from login to toggle between tabs and pages. If I restart the server everything works fine for about 15 to 30min.
Can somebody help me please?
Cisco Prime Infrastructure
Version : 3.3.0
Build : 22.214.171.124.342
disk: 1% used (192316 of 138271784)
temp. space 2% used (36396 of 2031952)
total memory: 16332120 kB
free memory: 263716 kB
cached: 3377140 kB
swap-cached: 2280 kB
user time: 21043962
kernel time: 8943886
idle time: 33607924
i/o wait time: 22437766
irq time: 116814
01-16-2018 02:04 AM
- Was your original installatation sized or installed according to network sizing parameters :
- Also , if you are using a VM based installation, check resource parameters or alerts of/for the VM in vcenter, pay attention to memory allocated to the VM.
01-16-2018 05:46 AM
thanks for your reply. The original VM based installation was installed according to network sizing parameters from "Standard".
I also checked the VM in vcenter. Attached you can see the CPU and Memory Usage.
Since if installed PI 3.3 the usage is permanently high. With PI 3.1 I only registered high usage when backups where taken.
On the PI GUI i can also see that there a lot of jobs that have not started at sheduled time. Could this be the reason why the CPU and Memory Usage is so high?
01-16-2018 06:52 AM
>installed from Standard
And does this 'comply' with your network size (according to the link send earlier) ?
>jobs not started
I would describe this as a deadlock, where both parameters can influence each other, basically I suspect lack of memory resources. Try doubling available memory (as a test). Check wether these issues persist.
01-16-2018 10:44 AM
I have the same problem after in-place upgrade from 3.1 to 3.3. Logging into the CLI and running "ncs status" just hangs. I can get to the web login page but no further.
01-16-2018 10:41 PM - edited 01-17-2018 02:35 AM
yes the Standard installation comply with our network size. Actually we only use the PI to manage up to 1500 Lightweight APs.
I will try to double the available memory and check if the problem persits.
01-18-2018 02:55 AM - edited 01-18-2018 02:56 AM
I doubled the available memory of the VM. The PI now works better but not as usual. Also, there are still a lot of scheduled jobs that cannot be started and the CPU usage is also very high (see attached file).
Doubling the memory is also only a temporary solution because I need the resources for other projects.
Is there a workaround to fix this problem durable? If not are there any possibilities to downgrade to PI 3.1?
01-18-2018 03:56 AM
>Is there a workaround to fix this problem durable?
Contact CISCO TAC
>If not are there any possibilities to downgrade to PI 3.1?
You cannot downgrade prime, you can if you have a 3.1 (or compatible) backup re-initialize a VM, install virgin 3.1 and restore your backup.
01-24-2018 05:45 AM
You may wish to drop to the linux root shell and find out what occupies prime.
Use top to get the pid's hogging the cpu or consuming all memory and publish the ps -ef | grep <pid>
You never know if it rings a bell for someone.
01-24-2018 01:27 PM
I am facing the same issue, please open a TAC, so they can see that others are having the same.
I have broken my HA, one node running version 3.2MR1 and the other node running 3.3, both using the same restore DB from 3.1.2.
The version with 3.2MR1 workings perfectly fine, and 3.3 runs for about 10-15 minutes and goes to a crawl, so far TAC/BU saying, I am over the interface limit.
01-24-2018 02:31 PM
I am working with TAC currently - they want to tell me it's a disk IOPS issue. TAC ran "ncs run test iops" and it measured at ~30MB/sec when the sizing chart says it should be 200. I have 10 other VMs on this host and none of them have issues. It could be a problem with my iSCSI storage on my NetApp but I am doubtful. I am going to downgrade back to 3.1 and see if I get my performance back, plus run the same IOPS tests to see what the metrics are when it's working properly.
01-24-2018 02:48 PM
01-24-2018 10:29 PM - edited 01-24-2018 10:30 PM
The command ncs run test iops can be run when PI is active, but should be run, when it is not!
TAC doesn't always stop PI to make the test :-)
01-29-2018 12:13 PM
With NCS stopped, I ran the IOPS test several times. Results:
1: 128 MB/s
Those are widely varied results... Next I am going to downgrade to 3.1 which was working fine before. Hopefully it will go back to normal even with my less-than-ideal disk write speeds. Maybe if I can show TAC that the web interface behaves normally under similar write times in 3.1 and chokes in 3.3 they will be more helpful.