cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2127
Views
5
Helpful
12
Replies

ANIServer CPU usage LMS 4.1

dominikhug
Level 1
Level 1

Hi

I recently made a fresh install of LMS 4.1 and added all of our devices (about 400 devices). After configuring all jobs and services everything ran fine.

After a few days a guy from the server team called me and told me that the CPU-usage increased over the days. I made some investigations and found out which process is using CPU ressources. Whenever ANIServer is running and collecting data CPU is running at approx. 100%. First it takes only a few minutes to complete but after some cycles ist takes more and more time to complete, after a week it takes more than 4 hours. After restarting ANIServer process it takes again a few minutes and then the duration increases.

Is this normal behaviour or could it be a bug?

I've attached a screenshot which shows the cpu usage.

Server specs:

Windows Server 2008 R2 64-bit

VMWare ESX

4 CPUs  @ 2.93 GHz

6 GB RAM

Kind regards

Dominik

12 Replies 12

dominikhug
Level 1
Level 1

Can anyone help me?

to find out what is causing the issue it is necessary to get the processes consuming CPU. You can use "Process Explorer" from Sysinternals / MS Technet to get details on each running process:

the sysinternals suite (with a lot more usefull tools):

http://technet.microsoft.com/de-de/sysinternals/bb842062

or Process Explorer alone:

http://technet.microsoft.com/de-de/sysinternals/bb896653

Hi Martin

I made two screenshots:

1     Process Explorer

2     Process Management section from LMS

CPU-usage raised after manually starting data collection.

DataCollection is under control of ANIServer, and it is in deed a CPU and RAM consuming process. This alone should not be a problem.

The first graph you posted is missing the description of the x- and y-axis. So I do not see how often the CPU spikes appear and what the numbers on the right side mean.

You said, your problem ist that the CPU usage is constantly rising. You should see this if you (in process explorer) right click on the process you mentioned to bring up the "Properties" view or if you expand the general view to show also CPU history for each process.

Let Process explorer open for a while so you see if there is one process which does not free up all the resources it takes in the first instance.

Hi Martin

I've got a better graph showing utilization over more than one week.

Data collection is scheduled every eight hours, so every eight hours cpu uses resources. The last time I had to kill ANIServer data collection was running for about 6 hours or something like this.

Hmm... I searched in this forum and I found some interesting discussions about DEP.  Could this problem be triggered because I haven't disabled DEP for LMS?

DEP usually makes problems during installation. Also processes and tasks could fail because of DEP. I wouldn't have brought DEP in relation with your problem but you should disable it anyway - you will see if it helps.

I am still thinking about what could be the reason for the permanent increase of CPU usage. In fact, I saw DataCollection consuming less CPU and RAM for the first run after dmgtd restart. I think it is because subsequent collections look for any changes in regard to previous runs.

What are the hardware specs of this server. 

HI Martin

Till yesterday I had following specs:

Windows Server 2008 R2 64-bit

VMWare ESX

4 CPUs  @ 2.93 GHz

6 GB RAM

Now I have only 2 CPU. I had to give two CPUs back because of this problem. But it doesn't matter, the problem occurs with two or four CPUs.

are the CPUs strictly assigned to the server or is there a dynamic option active?

How many devices are managed with this server?

Check if the pagefile is statically assigned? It should have a size of 12 GB (min).

If you have to change the pagefile settings also, you'll have to reboot the server. Always stop dmgtd first (net stop crmdmgtd), wait until this is finished, now reboot the server.

Also check if antivirus software is running on the server? If yes, make sure that CSCOpx directory and the location to where the database backups are written are excluded from on-access scanning.

Can you provide the ani.log?

See attached ani.log. I changed some names and IPs but I think it should be ok.

It seems that you have port-channels configured. There is a bug first found in LMS 4.0.1 by Marvin concerning data collection and port channels (BugId CSCto06189). The log entries looks very similar, though there are other errors which shouldn't be there...

see this thread:

https://supportforums.cisco.com/message/3313091#3313091

I cannot find the bug mentioned in the release notes for LMS 4.1 not as still active, nor as resolved. A patchis available on CCO only for LMS 4.0.1:

http://tools.cisco.com/search/display?url=http%3A%2F%2Fwww.cisco.com%2Fcisco%2Fpub%2Fsoftware%2Fportal%2Fselect.html%3Fi%3D%21s%26imageGuId%3D7C1A18B387C996B073129358E81B34810C01B311&pos=1&strqueryid=3&websessionid=xDK8wuWdT_DZX_--eehG2hD

You could open a TAC case along with the ani.log to confirm if this is the bug (what I assume) and to get an official patch.

...perhaps Joe could confirm it also...

Hi Martin

Thanks for your help. I'll check this with our account manager.

Have a nice weekend!

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Innovations in Cisco Full Stack Observability - A new webinar from Cisco