user tracking problem - lms 3.2

[cmarva]

Noticed a pretty weird one. LMS 3.2 on windows 2008.

Everything had been running normally. A while back we noticed the user tracking

tables diminishing, and found out it was due to a bug in the code we were running

on our 4500s. So we started an upgrade cycle and the tables started growing again.

UT runs fine, but there is nothing newer than Feb 25th. Even if I do a manual UT discovery

it seems to run fine, but there are no new entries in the table, or rather nothing with the

last seen date of today when I just ran it.

I did do the usual, run the logBackup script, clean some other log files, I even rebooted

the server, but it still doesn't update with any new UT entries. I have checked and can

do snmpwalks per vlan on the cam tables, so I don't believe it's anything with the switches,

I think this is something within UT or campus.

I would be happy to send up any log files that might be useful. Thanks - chris

[cmarva]

I do see in the process status that the ani server process is in a "running with busy flag

set" state. I don't know if this is normal or not.

It's goofy because UT runs but doesn't update the tables. Once again, most recent last seen

date on all entries is 25-Feb.

chris

[Gaganjeet Chug]

Hi,

Its known bug in LMS 3.2 for Campus Manager. BUG ID = CSCtd49439

You can view this bug at the following URL :-

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?caller=pluginredirector&method=fetchBugDetails&bugId=CSCtd49439

Here are the steps to fix this issue.

1> Stop the daemon manager - Run this command from CLI of the CisocWorks server - net stop crmdmgtd

2> Replace this file PortDetailsXml.class at the location with the old one ( instead take the backup of old file to another location and delete it from here ) :-

CSCOpx/campus/lib/classpath/com/cisco/nm/ani/server/core

3> Make sure casuser have proper permission on this file as it was before

4> Start the Dameon manager - net start crmdmgtd and wait for 20- 25 minutes before logging back to CiscoWorks.

5> Run the new Data Collection from Campus Manager home page and after it complete, run the new User Tracking Acquisition.

Now check if it display the correct data.

Note :- Before applying the patch, you need to make sure you are running Campus Manager version 5.2.1, which you can check from Common Services > Software Center > Software Updates.In case it's, 5.2 only, then download Campus Manager 5.2.1 and install it first before applying the patch

Attaching the patch for the Campus Manager version 5.2.1 for windows

Thanks

Gaganjeet

[cmarva]

ok, thanks. I had seen this in other posts, but it didn't seem to me that

it was a match on what I am seeing.

I will apply this patch when I get some time and see what happens. One

question, though. Is a new PortDetailXML.class file included in the patch?

(I guess I'll see when I unzip it). The existing class file appears to be the original.

At any rate, I'll give this a shot and see what happens. I also have enabled debugging

on the UT jobs, so I should have a good log file from that after it runs this morning.

Thanks, I appreciate it - chris

[cmarva]

I have applied the patch. Being it is the end of the week and I'm just about cooked, I will check later

after the regular scheduled job runs and see if this is resolved.

Thanks again for the help - chris

[cmarva]

applying this patch does not seem to have helped. I am seeing the same thing, the UT runs as scheduled and

completes successfully, it's just that nothing is getting written to the UT database.

I have had to increase the snmp timeout for our 6500s as we are upgrading to 12.2.33-sxi5. I'm wondering if

maybe I should try increasing the timeout on all devices. I doubt this would help since it worked until Feb 25th.

Anyway, I appreciate the help. It's looking more ad more like a TAC case.

chris

[Gaganjeet Chug]

Hi,

Thanks for the update. Kindly post the screenshot of Campus Manager > Home and the two logs files :-

1> ani.log  (Location :- CSCOpx\log )

2> ut.log    (Location :- CSCOpx\log )

If didnt find much information on these logs, will suggest to go for TAC case. You can also open TAC case directly from here and all this information will be incorporated in your TAC case. Kindly look for Action tab on the right hand side of this page for opening TAC case from this thread.

Thanks,

Gaganjeet

[cmarva]

Gaganjeet,

thanks for the response. As UT debugging is enabled, the log file ~20M. Also attached is the ani log and the screentshot.

I forced the UT job to run. I did the usual logfile roll script as I do usually on a weekly basis, so everything hadn't restarted

by 8am, the the UT run kicks off.

I just cannot figure out why the tables would have just stopped getting updated. I am the only one who does maintenance

on this server or the apps, and I cannot track anything back to Feb 25th that got updated or changed that would affect this.

Everything else seems to be working fine (syslog automated actions, devices get discovered, dfm notification emails are working)

even other stuff in campus such as the topo maps work.

Anyway, thanks again for the help, I appreciate it.

[Gaganjeet Chug]

Hi,

Kindly point me to the direction where we have MAC Address or IP address not showing up in the End User tracking. From the logs, its updating the database.

Thanks,

Gaganjeet

[cmarva]

Gaganjeet,

that's interesting (and frustrating). In the screenshot I sent yesterday, the UT count is 12500. We should be well over 13000 end hosts.

In doing a search through the UT.log, one of the IPs that is not showing up when I search user tracking is:

2011/03/28 10:02:06 EvalTask-vmpsadmin-05 VmpsAdminSMFGetIpXlateTable: router=10.255.3.130 ip=10.242.9.15
mac=00-19-99-91-a1-14 subnet=10.242.9.0 mask=255.255.255.0 vtp=PETERSACC_10.255.3.130(T) vlan=PETERS-C-USER-1
vlan Index=1709

It is clearly in the logfile. But if you refer to the attached screenshot titled peters-ut-screenshot, if I search on just 10.242, I get

"no end hosts found". This site was brought online a couple weeks ago.

The second screenshot, titled PACS-subnet, shows no end hosts found. This is due to the SW bug we found in 12.2.50-sg8 and

the entries already having been purged. This switch was upgraded to 12.2.53-sg3 on March 14th so the entries should be showing

up. I have not searched ut.log for this subnet. The bug we found with 12.2.50-sg8 was that an snmpwalk of the mac tables, per vlan,

returned nothing. With 12.2.53-sg3, an snmpwalk of the mac tables per vlan does return the mac tables per vlan.

The third screenshot is just searching on 10.255, which covers our datacenter. While there are a lot of entries, the "last seen" date

is Feb 25th. As these servers are always on, I would expect to see the last seen date as yesterday when the last ut run was executed.

I have not searched ut.log for any of these entries.

So it does indeed appear that the ut discovery finds the end hosts. But when you pull up the user tracking utility to search, there is nothing newer

than February 25th. I hope this helps.

Thanks again for the help, I appreciate it - chris

[cmarva]

don't know if this has any bearing on the problem or not. While digging around and looking for anythin

g, I found a file in nmsroot\campus, called utupgrade.txt. The contents go something like this:

error: can't find version info file

exception: can't find nmsroot\campus\etc\cwsi\version.info (system can't find specified file)

inporting files into db

error: ensure utdata data file is in nmsroot\campus\etc\cwsi folder

error: ensure phonedata data file is in nmsroot\campus\etc\cwsi folder

the date stamp on this text file was yesterday afternoon, actually around the time I applied the

new mdf update as well as the new CM device update.

once again, don't know if it has any bearing or not. Don't know if running reinitdb would help either.

thanks - chris

[Gaganjeet Chug]

HI,

Good catch on those errors. Lets enable the debugging for User Tracking and UTLite to see the exceptions in details.

For UTLite debugging :-

Add a -debug flag on to the end of the UTLite33 line in the UTLite batch script

Restart UTLite (e.g. have the user log out and log back in)

Debug output is written to C:\utlite_user.txt

For User Tracking :-

Go to Campus Manager > Admin > Debugging Options > User Tracking Server > Check enable debug and select module "User Tracking , core , core, frontend and framework". SInce you have good number of end host so the debug ut.log file will be huge. You can collect ut.log in between , so that we can check if the funtinality is really broken and do we need to re-initialize the database.

Thanks,

Gaganjeet

[cmarva]

Gaganjeet,

thanks for the reply. I will add the increased debugging and get the logs posted.

just an fyi, lms3.2 serivce pack was applied on Friday.

Thanks again for the help, I appreciate it.

chris

[cmarva]

ok, the ut.log file with the increased debugging is attached. Hope it leads somewhere.......

thanks again for the help, I appreciate it.

chris

[Gaganjeet Chug]

Hi,

Kindly let me know one IP Address and the MAC address that is not showing up in the reports however it should be.

Thanks

Gaganjeet