Scheduled restart of Prime Infrastructure 3.1.3

[smp]

We are having Out of Memory problems with our Prime Infrastructure 3.1.3 installation, which causes the application to become unresponsive. As a short-term mitigation, I'd like to schedule a weekly restart of the application. Is there a supported way to do this? I don't find any reference in the product documentation.

Thank you very much.

[luijimen · Accepted solution]

Hi Scott,

A graceful restart of the PI services should get the job done to reclaim the memory.

The CLI commands "ncs stop" and "ncs start" do exactly this.

There is no built in (supported) way of scheduling this, however, you should be able to setup a Linux Cron job to run these commands at the desired times during the week.

I would advise giving at least 15 minutes between the call for shutdown and the call for startup, so you can allow enough time for the system to stop properly all services.

Hope this helps.

Luis

[smp]

That what I suspected, but was looking for confirmation. Thanks for your feedback.

[JEFFREY SESSLER]

Scott,

Did you ever solve the out of memory problem? I'm running Prime 3.1.4 and while PI has 32GB of RAM allocated, I note from the Admin dashboard that once utilization hits 20GB, the application becomes unresponsive. Alarms take forever to pull up, maps don't populate, etc.

A NCS stop/start fixes it, but then I can see memory use creeping up again until it hits the magic number of 20GB. 

[smp]

We are still having the same symptoms of the system locking up on 3.1.4 too, but it isn't because of memory. So the short answer is no. I have been working with a development engineer for several weeks now on determining the root cause and so far it has eluded us.

Wish I had a better answer for you.

[JEFFREY SESSLER]

Scott,

Would you mind sharing your support case #? I'd like to attach it to my request.

Do you see the memory utilization slowly climb and then hit a max that is below the allocated memory? As soon as mine hits 20GB the problems start despite there being 31GB avail. It never uses more than 20GB, and I then see everything slow down e.g. data cleanup goes from minutes to hours, alarms take forever to display (alarm API max climbs to 1.3 million ms). Maps don't render, etc. 

[smp]

I am hesitant because it turned out that our issue was not a memory problem. We are still trying to figure out what's going on, but it isn't memory-related. And I'd hate for your engineers to try and untangle the mess that our case has turned into.

[JEFFREY SESSLER]

With the assistance of TAC I fixed our issue. The Prime Admin ->Dashboards ->System Monitoring Dashboard -> Overview reports "Memory Utilization" as the combination of physical memory and swap.

For a "standard" VM deployment that's 16GB RAM and 16GB swap, and the above dashboard reports that as 32GB for total memory.

Prime seems to die a fiery death once it's exhausted physical memory, so if you aren't aware of how much physical memory is allocated, the graph will have you chasing your tail.

Once I allocated more physical RAM Prime seems happy again, but the same graph now claims 40GB of memory (24GB RAM + 16GB swap).

I figured I'd post this in the case that someone else runs into this issue. 

[itservicesinfrastructurenetwork]

FWIW, in Cisco Prime Infrastructure (3.10.2), we can see a "systemctl status NCS.service" (uppercase NCS as case-sensitive)

Stopping and starting this appears to do the same thing as a "ncs stop" and "ncs start" via ADE.

It's early days, but progressing weekly scheduled reboots using them in a crontab.