Auth Failure Traps

[baotran09]

After i changed snmp strings on our network devices , I see a list of devices with Auth Failure Traps on Syslog server.

Ive check the snmp credential strings on CW for each device and they're correct.

This is the error message on my syslog server:

mm-dd-yyyy    11:23:16    Local0.Info    10.1.1.1    10.1.1.2.150 4 0  Authentication failure 10.1.1.254(CiscoWorks) 1 10.1.1.254(CiscoWorks)

This message wasnt there before i re-new the snmp community string. After I chnage the snmp string on my routers and switches, I a lots of traps on my syslog server.

How can I stop this?

Thank you for your help

Thanks

[baotran09]

Any ideas?

please advise

[Joe Clarke]

If you are absolutely sure DCR is correct for these devices, then check your CS Discovery SNMP settings under Common Services > Device and Credentials > Device Discovery > Discovery Settings.  Make sure those community strings are correct as well.  If you still can't figure it out, start a sniffer trace filtering on udp/161 traffic between the LMS server and the device, and check to see what SNMP objects are being polled using the incorrect community string.  That should help narrow down the problem application.

[baotran09]

Hi Clarke,

Thanks for your help.

I used Ethereal to sniff the packets. The result shows that CW is polling using the OLD snmp string even though I have updated the DCR database with the new strings (ive verified the snmp string with device credential and doublecheck in DCR/exports to convirm the device credentialg - DCR db has the new strings). My question is why it uses the old string to poll? I've deleted the whole database and import it again but still seeing "snmp authentication failure" on syslog server.

How can I track which aplication is doing the snmp polling (I have RME, IPM, DFM and CS all on one server)

How can I stop the snmp polling completely? Ive stopped daemon and change aaa to local but no success.

[Joe Clarke]

If you've stopped Daemon Manager, and the polling continues, then the problem is not LMS.  You must have some other application installed on the server which is doing the polling.

[baotran09]

There's no other applciation, only Ciscoworks on this server. Etherreal showing missing MIB (see atatachment)

All I did was changing the snmp community string on routers and switches and removed the one one.

What else can I check?

[Martin Ermel]

exactly where do you see the authentication failure ? Is it on the LMS server? In which application? Is it on another server, which application do you use to get this message?
If you have stopped the LMS daemons (net stop crmdmgtd on windows or \etc\init.d\dmgtd stop on solaris), open 2 terminal sessions to one of the devices and  issue the following commands in the first sessions:
term mon
debug snmp packets

use the second session to easily disable the debugging if the message ouput in the first session is too much. Use this command:
undebug all

and you should see the source of the request. Is it still pointing to LMS?

[baotran09]

Hi Mermel,

Thank you for your rhelp.

I see the 'authentication failure message' on the syslog server, not on LMS server. Im using Kiwi syslog service manager to capture these messages.

As advised, tos atrtw ith I enable daemon and performed the debug command, I could see the source of the request (Ciscoworks server) after I entered 'debugged snmp packets'

Feb 25 19:36:45.375 AWST: SNMP: Packet received via UDP from 10.1.1.1 (CWserver) on Serial0/0/1
Feb 25 19:36:45.379 AWST: SNMP: Queuing packet to 10.1.1.2 (syslogserver)
Feb 25 19:36:45.379 AWST:
Outgoing SNMP packet
Feb 25 19:36:45.379 AWST: v1 packet
Feb 25 19:36:45.379 AWST: community string: readstring
Feb 25 19:36:45.379 AWST: SNMP: V1 Trap, ent snmpTraps, addr 192.168.1.1 (remote router loopback interface), gentrap 4, spectrap 0
lsystem.5.0 = 10.1.1.1
ciscoMgmt.412.1.1.1.0 = 1
ciscoMgmt.412.1.1.2.0 = 10.1.1.1
Feb 25 19:36:45.631 AWST: SNMP: Packet sent via UDP to 10.1.1.2

How can I stop this?

[Joe Clarke]

Based on the packets you captured previously, it could be DFM or HUM doing the polling.  Try shutting down DfmServer and DfmServer1, and see if the polling stops.  If not, shutdown UPMProcess.

But just to be clear, if you do "net stop crmdmgtd" does the polling stop?

[baotran09]

I did "net stop crmdmgtd" but the polling didnt stop. I have only 1 ciscoworks server and no other application running in the background other than Ciscoworks.

Correction, I mentioned CW used old snmp string, I was wrong. It uses a new string when polling, but I dont know why it giving me a authentication failure. I've check thwe string on my routers and switches and again on CW.

I shutdown dfmserver and dfmserver 1 but still see the polling, my server doesnt have HUM so there's no UPMprocess option to shutdown

Every minute I get about 100 traps, I have 1000+ routers and switches, so basically sonner or later it going to killing my syslog server.

Attached is the screen capture of the repated polling message on my syslog server

Thank you for your input.

[Joe Clarke]

With Daemon Manager shutdown.  Post a list of all processes running on the server.

[baotran09]

Hi Clarke,

Document 1 contains a list of processes with with daemon shutdown.

Document 2 contains a list of processes with daemon turned on.Im seeing alot of cwjava.exe processes, is it normal?

Should sm_server processes be stopped when I stopped the daemons? Is this as bug (CSCsx23656-DFM3.2: sm_server does not stop when daemons are stopped. ?)

Which processes can I kill in order to stop the snmp polling?

[Joe Clarke]

You need to apply the fix for CSCta56151 from http://tools.cisco.com/support/downloads/go/ImageList.x?relVer=3.2.0&mdfid=282640771&sftType=CiscoWorks+Device+Fault+Manager+Patches&optPlat=Windows&nodecount=2&edesignator=null&modelName=CiscoWorks+Device+Fault+Manager+3.2&treeMdfId=268439477&treeNa... .  Then reboot, and this problem should go away.

[baotran09]

You mean reboot the server?

What is the root cause?

[Joe Clarke]

Yes, I mean reboot the server.  The root cause is that the DFM polling processes are not shutting down when Daemon Manager goes down.