Re-initialize the DFM database for CWLMS 2.6

[Mohammed Tolib]

Hi all,

We are using CWLMS 2.6 on a UNIX machine. And recently we changed the SNMP String to our network devices. One of L2 switches keeps logging the following message:

%SNMP-3-AUTHFAIL: Authentication failure for SNMP req from host 10.x.x.1

Where 10.x.x.1 is ciscoworks LMS server. I found a solution on many sites that suggest resetting DFM database. I stopped first the daemon manager and tried to apply the perl script:

perl dbRestoreOrig.pl dsn=dfmInv dmprefix=INV npwd=cisco

but it gives me the following error:

Can't locate CRM.pm in @INC (@INC contains: /usr/perl5/5.6.1/lib/sun4-solaris-64int /usr/perl5/5.6.1/lib /usr/perl5/site_perl/5.6.1/sun4-solaris-64int /usr/perl5/site_perl/5.6.1 /usr/perl5/site_perl /usr/perl5/vendor_perl/5.6.1/sun4-solaris-64int /usr/perl5/vendor_perl/5.6.1 /usr/perl5/vendor_perl .) at dbRestoreOrig.pl line 31.

BEGIN failed--compilation aborted at dbRestoreOrig.pl line 31.

CRM.pm already exists in the path ENV{NMSROOT}/lib/perl/db

At line 31 of dbRestoreOrig.pl – the error – I found the following:

push(@INC, "$ENV{NMSROOT}/cgi-bin/dbadmin/pdbadmin");

use lib "$ENV{NMSROOT}/lib/perl/db";

I gave the system the path of NMSROOT and run the script again but it gives me the same error “Can't locate CRM.pm”

I will appreciate any response.

Thanks in advance

[Martin Ermel]

I assume your problem is the way how you call perl. You have to call it with the full path:

    /opt/CSCOpx/bin/perl

otherwise /usr/perl is called

/opt/CXSCOpx/bin/perl /opt/CSCOpx/bin/dbRestoreOrig.pl dsn=dfmInv dmprefix=INV npwd=cisco

But when you reinitialize the INV database of DFM you also have to remover the rps files located here:

    NMSROOT/objects/smarts/local/repos/DFM.rps

See this thread for details:

    https://supportforums.cisco.com/message/645253#645253

Before doing the above steps, just try to delete the device from DFM and readd it, than see if the problem is gone.

[Mohammed Tolib]

Thanks Martin. You are right. All scripts ran with no errors. After that I deleted the DFM.rps file. However, the main issue still exists.

Any more ideas.

[Martin Ermel]

when the DFM database was reinitialized and before it was refilled with device information was the message gone?

you can verify the credentials LMS is working with by exporting the credential to a csv or xml file:   

     Common Services > Device and Credentials > Device Management

          then click Export

If you have verified the credential and it is ok, check the following:

what is the time interval for the syslog message to appear? Is it every 3-4 mins, every 2 hours, every 4 hours? The time intervall sometimes points to the app that is responsible for doing the bad request.

Before the reinit of the DFM databases have you deleted and readded the device?

[Mohammed Tolib]

Thanks again for the reply.

For the last question: Yes, I deleted the device and added it again. The credentials is OK as I can run Inventory collection jobs and the devices is in normal state in the Inventory.

I even deleted all databases and imported all devices again but the same log message appears.

We ran sniffing on the Switch port and found somthing strange:

The SNMP credintials are sent right for SNMP get-request, get-response and half of get-next-request. at athe point:

551 483.020374 10.X.Y.Z 10.A.B.C SNMP get-next-request 1.3.6.1.2.1.17.2.15.1.2 1.3.6.1.2.1.17.2.15.1.3 1.3.6.1.2.1.17.2.15.1.4 1.3.6.1.2.1.17.2.15.1.8 1.3.6.1.2.1.17.2.15.1.9 1.3.6.1.2.1.17.2.15.1.5 1.3.6.1.2.1.1.3

(Where 10.X.Y.Z is the CWLMS server IP Address and 10.A.B.C is the L2 Access Switch)

we found extra characters are concatenated to the SNMP community string. Those characters are like (@1, @110, @111, @120, .... etc). At the same time the previous syslog message is generated.

The strange thing is that we have a twin CWLMS server with same specs and has the same devices and we don't get this syslog message.

The time of the log message are the same from day to day it appears every 2 hours for 10 min. and sometimes for a whole hour.

Thanks again

[Martin Ermel]

this sheds some light on it...

the time intervall let me assume that it is a process under the hood of ANIServer wich polls all devices managed by Campus..

The "strange characters" at the end of the community string are caused by a mechanism called "community string indexing" which Cisco uses to get MAC address tables per VLAN; have a look here

http://www.cisco.com/en/US/tech/tk648/tk362/technologies_tech_note09186a00801576ff.shtml

The following are known problems which could be related to your problem:

1) (because you are on LMS 2.6 I assume the following could help you:)

I assume that this is the problem, the switch answers with an SNMP AUTH failure if it is queried for an unused VLAN

make a backup of the following files, than check the settings for these parameters

(if they do not exist, add them at the end of the file; do not change the extension of the file when you save it!!)

    NMSROOT/campus/etc/cwsi/ANIServer.properties

        UTGetSuspendedVlans=0

        UTGetVlansOnDownPorts=0

        UTGetVlansWithUserPortsIOS=1

I am not sure if LMS 2.6 has ut.properties yet; If yes, check this also:

    NMSROOT/campus/etc/cwsi/ut.properties

        UTGetVlansOnDownPorts=0

        UTGetVlansOnUserPortsIOS=1

Always make backup of the file before changing any value !

These are also known issues which could apply:

2) https://supportforums.cisco.com/message/3341436#3341436

3)if the device being polled is a cat2900XL, this could be bugID CSCdp45404, and it is fixed in 12.0(5)XU

https://supportforums.cisco.com/message/627308#627308

[Mohammed Tolib]

Thanks a lot Martin. Those discussion realy lighted me up. However, I can't do anything now as this is our week end. I will try next Saturday and let you know.

[Martin Ermel]

Ok, I am curious about if your issue is solved...

[Mohammed Tolib]

Hi Martin,

Bug CSCdp45404 should be fixed in 12.0(5)XU and higher code. We have

IOS (tm) C2950 Software (C2950-I6Q4L2-M), Version 12.1(20)EA1a, RELEASE SOFTWARE (fc1)

Copyright (c) 1986-2004 by cisco Systems.

Do we still have to upgrade?

Thanks in advance.