LMS 3.2 Dynamic User Tracking CPU issue

[andrewswanson]

Dynamic User Tracking CPU issue

Hello

     I'm using LMS 3.2 for dynamic UT updates for around 140 catalyst 2950/2960 switches. Server details are:

VM - Win2008 SP2 64bit - 2 Quad-Core AMD Opteron 2.31GHz, 6GB RAM

LMS -

1.  Campus Manager 5.2.1

2.  CiscoView 6.1.9

3.  CiscoWorks Assistant 1.2.0

4.  CiscoWorks Common Services 3.3.0

5.  Device Fault Manager 3.2.0

6.  Integration Utility 1.9.0

7.  LMS Portal 1.2.0

8.  Resource Manager Essentials 4.3.1

     VM cpu is consistently high (90%-100%). When i switch off dynamic updates process, cpu drops dramtically. i'm seeing about 4 add/delete events a second in the macuhic log. Dynamic UT is setup as follows:

validate snmp community:         yes

validate trap source:                  no

debugging:                               off

am i overloading lms with 140 switches (around 40 of these are 48port, the rest are 24) for dynamic UT?

thanks

andy

[Joe Clarke]

First, try turning off community string validation.  See if this brings down the CPU utilization.  That would help narrow down the cause.  However, to fully understand what is going on, I recommend you open a TAC service request so some thread dumps of the MACUHIC and UTManager processes can be captured to see what they are doing that is causing excessive CPU utilization.  This process is not very straightforward on Windows prior to LMS 4.0.  You can click the "Open Service Request" link in the actions panel for this thread to transfer this information into the service request.

[andrewswanson]

thanks for the response - i switched off validate snmp community after i had originally posted and it didn't make any difference.

originally the LMS VM had 1 CPU and was working fine before i started adding more switches configured for dynamic utu - eventually the cpu flatlined at 100%. now the VM has 2 cpu's it still gets hit hard - i've attached screenshots of the impact of switching off dynamic utu on LMS.

of the 140 switches (catalyst 2950 and 2960) configured for dynamic utu, i had to disable ip dhcp snooping on the catlayst 2950s because it was causing problems with Microsoft WDS multicasting.

i'll open a TAC as suggested.

thanks again

andy

[Joe Clarke · Accepted solution]

This is the first I've heard of Dynamic UT being a performance hog.  Typically, one vCPU is fine unless the server really has enough unused CPUs to dedicate to a given VM.  The thread dumps will be helpful to figure out what the processes are doing.

[andrewswanson]

Issue resolved with TAC.

Cause of problem was very embarrasing but in a spirit of openess, here goes:

Over a period of about 1 month, 1 pc on 1 unsecured switchport added over 8000 mac addresses to the LMS UT DB. during this time, LMS cpu rose steadily until it flatlined at 100%. the pc in question was removed but lms cpu usage stayed at 90-100% with dynamic utu enabled. the following steps were taken to resolve cpu problem:

• Purged UT database of inactive entries - this reduced UT database to 4000 hosts plus 800 phones (this is what i'd expect to see). Ran full Data Collection followed by UT acquisition but dynamic UTU still caused high CPU of 90-100%

• Purged UT database of all entries. Ran full Data Collection followed by UT acquisition - i now have a UT database of 4000 hosts plus 800 phones with dynamic UTU enabled and cpu usage has dropped to an average of 15%

Thanks to Nadi at TAC for his help. I think i'll pay more attention to ciscoworks in the future.

Cheers

Andy

[Joe Clarke]

Thanks for the follow-up.