10-03-2016 09:51 AM
We are having Out of Memory problems with our Prime Infrastructure 3.1.3 installation, which causes the application to become unresponsive. As a short-term mitigation, I'd like to schedule a weekly restart of the application. Is there a supported way to do this? I don't find any reference in the product documentation.
Thank you very much.
Solved! Go to Solution.
10-03-2016 03:09 PM
Hi Scott,
A graceful restart of the PI services should get the job done to reclaim the memory.
The CLI commands "ncs stop" and "ncs start" do exactly this.
There is no built in (supported) way of scheduling this, however, you should be able to setup a Linux Cron job to run these commands at the desired times during the week.
I would advise giving at least 15 minutes between the call for shutdown and the call for startup, so you can allow enough time for the system to stop properly all services.
Hope this helps.
Luis
10-03-2016 03:09 PM
Hi Scott,
A graceful restart of the PI services should get the job done to reclaim the memory.
The CLI commands "ncs stop" and "ncs start" do exactly this.
There is no built in (supported) way of scheduling this, however, you should be able to setup a Linux Cron job to run these commands at the desired times during the week.
I would advise giving at least 15 minutes between the call for shutdown and the call for startup, so you can allow enough time for the system to stop properly all services.
Hope this helps.
Luis
10-05-2016 07:10 AM
That what I suspected, but was looking for confirmation. Thanks for your feedback.
12-12-2016 02:32 PM
Scott,
Did you ever solve the out of memory problem? I'm running Prime 3.1.4 and while PI has 32GB of RAM allocated, I note from the Admin dashboard that once utilization hits 20GB, the application becomes unresponsive. Alarms take forever to pull up, maps don't populate, etc.
A NCS stop/start fixes it, but then I can see memory use creeping up again until it hits the magic number of 20GB.
12-13-2016 05:13 AM
We are still having the same symptoms of the system locking up on 3.1.4 too, but it isn't because of memory. So the short answer is no. I have been working with a development engineer for several weeks now on determining the root cause and so far it has eluded us.
Wish I had a better answer for you.
12-13-2016 07:36 AM
Scott,
Would you mind sharing your support case #? I'd like to attach it to my request.
Do you see the memory utilization slowly climb and then hit a max that is below the allocated memory? As soon as mine hits 20GB the problems start despite there being 31GB avail. It never uses more than 20GB, and I then see everything slow down e.g. data cleanup goes from minutes to hours, alarms take forever to display (alarm API max climbs to 1.3 million ms). Maps don't render, etc.
12-13-2016 09:24 AM
I am hesitant because it turned out that our issue was not a memory problem. We are still trying to figure out what's going on, but it isn't memory-related. And I'd hate for your engineers to try and untangle the mess that our case has turned into.
12-14-2016 03:12 PM
With the assistance of TAC I fixed our issue. The Prime Admin ->Dashboards ->System Monitoring Dashboard -> Overview reports "Memory Utilization" as the combination of physical memory and swap.
For a "standard" VM deployment that's 16GB RAM and 16GB swap, and the above dashboard reports that as 32GB for total memory.
Prime seems to die a fiery death once it's exhausted physical memory, so if you aren't aware of how much physical memory is allocated, the graph will have you chasing your tail.
Once I allocated more physical RAM Prime seems happy again, but the same graph now claims 40GB of memory (24GB RAM + 16GB swap).
I figured I'd post this in the case that someone else runs into this issue.
08-17-2023 04:26 PM
FWIW, in Cisco Prime Infrastructure (3.10.2), we can see a "systemctl status NCS.service" (uppercase NCS as case-sensitive)
Stopping and starting this appears to do the same thing as a "ncs stop" and "ncs start" via ADE.
It's early days, but progressing weekly scheduled reboots using them in a crontab.
06-19-2024 02:59 AM
We would also like to shedule a restart of NCS, did it work ok to make a crontab start and stop the NCS.service ?
06-19-2024 04:51 AM
Hi @s.hellman - we have had no "lock up" issues with Cisco Prime after implementing a weekly reboot via Crontab. Just logging on now and posting what our crontab now looks like (6PM on each Sunday) ...
[root@OBFUSCATE-01 ~]# uptime
21:47:59 up 3 days, 3:46, 1 user, load average: 1.52, 1.37, 1.42
[root@OBFUSCATE-01 ~]# crontab -l
0 18 * * 0 /etc/cron.d/zz-reboot
[root@OBFUSCATE-01 ~]# cat /etc/cron.d/zz-reboot
#!/bin/sh
/bin/systemctl stop NCS.service
/bin/sync ; /bin/sync ; /sbin/shutdown -r now
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide