PI 3.1. Poor performance and high memory usage

gary.gatten · ‎02-06-2017

Hi all,

Experiencing very poor performance and high memory usage om PI 3.1, especially when compared to LMS 4.2. I'm currently doing very little active monitoring and receiving maybe 30 traps per second from a handful of WLC's and MSE.

We are running the virtual appliance with 32GB RAM and 16 CPU's. I THINK it's a "professional" install, but not 100% sure as I did not install it. If there is a way to determine this that would be helpful. CPU is OK, load usually around 4 or 5. Sometimes IO wait is 20% which seems not good... But my main question for now is RAM usage. Java is around 12GB (up tod 16GB) and a bunch of other Oracle processes that are 4GB - so it's using all 32GB and some swap. I only have MAYBE 10% of LMS devices moved over to PRIME, so am concerned something is not right here. LMS is using a fraction of RAM and CPU and runs much quicker.

Wondering if anyone would be willing to share output from top, iotop, iostat, etc. to compare with mine? Also, how many devices you are managing and what you are monitoring with what frequency? How many logs and traps per second or per hour, etc.

Basically just trying to determine if there is something wrong with my specific install or config - or if this is a common thing and just need to throw more hardware at it?

I'll post some of my top, iostat, iotop info as well. Thanks!

marce1000 · ‎02-07-2017

- I had to increase the available mem significantly when upgrading my prime VM from 3.0 to 3.1; you may want to do some testing with higher memory settings until satisfactory performance is achieved.

M.

-- Each morning when I wake up and look into the mirror I always say ' Why am I so brilliant ? '
When the mirror will then always repond to me with ' The only thing that exceeds your brilliance is your beauty! '

akisbouza · ‎03-01-2017

I also face poor perfomance with Prime 3.1.0.0.132.

My installation was Standard with 16 GB RAM.

I have now a total of 24 GB Ram but performance is still very poor.

I am aware that linux uses all available memory so the output of show memory is misleading

prime/admin# show memory
total memory:   24216736 kB
free memory:      490852 kB -----> this is misleading.
cached:          5133016 kB
swap-cached:       57928 kB

So in shell if i enter free - m

ade # free -m
                   total           used       free     shared    buffers     cached
Mem:         23649      23179        469       2465        176       5020
-/+ buffers/cache:    17983       5665
Swap:        15999        662      15337

I see that used memory is 17.98GB my free memory is 5.5 GB which makes sense

However when i use top command i get

top - 11:09:03 up 1 day, 15:01, 1 user, load average: 2.75, 2.47, 3.10
Tasks: 450 total,   2 running, 448 sleeping,   0 stopped,   0 zombie
Cpu(s): 8.5%us, 1.3%sy, 0.0%ni, 90.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 24216736k total, 23951416k used,   265320k free,   178824k buffers
Swap: 16383996k total,   678008k used, 15705988k free, 5121228k cached

PID USER      PR NI VIRT RES SHR S %CPU %MEM    TIME+ COMMAND
15508 oracle    20   0 4723m 883m 617m R 99.4 3.7   3:32.55 oracle_15508_wc
31443 prime     20   0 15.2g 10g 216m S 19.2 45.8   1889:13 java

The most worrying for me is that Java seems to use 15.2 GB Ram.

I talked with TAC and they suggested to upgrade to 3.1.5 maintenance release.

Has anyone resolved this issue?

patoberli · ‎03-01-2017

In my opinion, 3.1.5 made it actually worse. But sure, you can try it out, it does fix a lot of bugs.

Also, the version number you posted doesn't show any installed patches (it remains the same for any 3.1.x release), for this you can either click on the "View installed Updates" on the login mask or under Administration -> Software Update the latest installed Critical Fix.

Brett · ‎04-17-2017

I know 3.1.5 is having memory leak issues. Upgrades 01 and 02 were suppose to fix it, however the issue still persists. Hopefully 3.1.6 solves this issue.

patoberli · ‎02-09-2017

In our case PI 3.1 also seemed to use more memory, but only on first glance. What they probably have changed though, they started to use some SWAP, which they didn't in previous versions. This is, in my opinion, fine for data that gets rarely used.

We use here 8 CPU cores and 16 GB ram, having one WLC which sends up to a few hundreds events per second, but typically it's under 50/s. I'm using Assurance now, also on that machine.

I rebooted the server today, but here is the output of the command top (you can run this via ssh, issue the command 'shell' first).

ade # top
top - 14:31:51 up  1:10,  1 user,  load average: 1.83, 1.36, 1.25
Tasks: 300 total,   1 running, 299 sleeping,   0 stopped,   0 zombie
Cpu(s):  9.5%us,  1.9%sy,  0.0%ni, 87.1%id,  1.4%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  16333448k total, 15756120k used,   577328k free,    17128k buffers
Swap: 16383996k total,    45864k used, 16338132k free,  3742000k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                        
 3764 root      20   0 98.5m 2784 1892 S  9.2  0.0   5:41.89 Xvfb                            
25960 oracle    20   0 3431m 1.0g 1.0g D  9.2  6.4   0:06.01 oracle_25960_wc                 
 5298 root      20   0 2260m 258m 9020 S  5.5  1.6   3:32.37 java                            
 7487 prime     20   0 11.5g 5.7g  57m S  5.5 36.8  29:48.50 java                            
 5295 root      20   0 2400m 345m 9.9m S  3.7  2.2   3:38.01 java                            
 5296 root      20   0 2779m 416m 9020 S  3.7  2.6   3:38.71 java                            
27448 admin     20   0 15164 1296  836 R  1.8  0.0   0:00.05 top                             
    1 root      20   0 19364 1212  920 S  0.0  0.0   0:01.36 init                            
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.01 kthreadd                        
    3 root      RT   0     0    0    0 S  0.0  0.0   0:00.02 migration/0

Before the reboot it was using around 4 GB of Swap. I guess there might be a memory leak in 3.1.5 somewhere. My Swap usage was rising by around 200 MB per day. You can see that under Administration / Dashboards / System Monitoring Dashboard, in the System Memory dashboard.

I've attached a graph to outline the Swap growth. I guess if you server is running for some weeks at a time, this might really become a serious issue :(