Re: Ciscoworks IPM and sending SNMP traps - Page 2

miwitte · ‎11-24-2006

HAs anyone been able to successfully deploy this? I have IPM working fine, I just want to be able to send traps from our source device to our syslog server so we get threshold and Timeout notification. I have TAC case opened so far no success. this is what I have on my 7206

ip sla monitor logging traps

snmp-server community blogblog RO 2

snmp-server community blahblah RW 2

snmp-server enable traps syslog

snmp-server enable traps rtr

snmp-server host 10.24.160.20 blahblah tty config entity envmon syslog rtr snmp

logging trap debugging

logging 10.24.160.20

Here is the doc I am going by;

http://www.cisco.com/univercd/cc/td/doc/product/software/ios124/124cg/hsla_c/hsthresh.htm

miwitte · ‎12-06-2006

does it need to be informs instead of traps? I saw this doc. Don't want to change anything that would affect the actual monitoring of the device as it is our main hub router that everything goes through..

snmp-server enable traps rtr

To enable the sending of Service Assurance Agent (SAA) Simple Network Management Protocol (SNMP) notifications, use the snmp-server enable traps rtr command in global configuration mode. To disable SAA SNMP notifications, use the no form of this command.

snmp-server enable traps rtr

no snmp-server enable traps rtr

Usage Guidelines

SNMP notifications can be sent as traps or inform requests. This command enables both traps and inform requests.

This command controls (enables or disables) Cisco Service Assurance Agent notifications, as defined in the Response Time Monitor MIB (CISCO-RTTMON-MIB). The Service Assurance Agent was previously called the Response Time Reporter (RTR); the RTR syntax is retained in this command.

This command enables the following notifications:

?rttMonConnectionChangeNotification?(valid for echo or pathEcho operations only) A rttMonConnectionChangeNotification indicates that a connection to a target (not to a hop along the path to a target) has either failed on establishment or been lost and when reestablished.

?rttMonTimeoutNotification?A rttMonTimeoutNotification indicates the occurrence of a timeout for a SAA operation.

?rttMonThresholdNotification?A rttMonThresholdNotification indicates the occurrence of a threshold violation for a SAA operation.

?rttMonVerifyErrorNotification?A rttMonVerifyErrorNotification indicates the occurrence of data corruption in an SAA operation

The snmp-server enable traps syslog command is used in conjunction with the snmp-server host command. Use the snmp-server host command to specify which host or hosts receive SNMP notifications. To send SNMP notifications, you must configure at least one snmp-server host command.

The following example enables the router to send SAA related informs to the host at the address myhost.cisco.com using the community string defined as public:

Router(config)# snmp-server enable traps rtr

Router(config)# snmp-server host myhost.cisco.com informs version 2c public rtr

Joe Clarke · ‎12-06-2006

You do not need to enable informs. Traps are sufficient. The commands "snmp-server enable traps rtr" and "snmp-server host x.x.x.x traps public rtr" are good enough to send SNMPv1 traps.

miwitte · ‎12-06-2006

I will try to use the debug ip sla monitor trace 270 (our application is 270) sometime early am in the next couple of days. Anything else? Since it appears that my router config is correct and looking at the stats, I can see that I am crossing the threshold the question is why won't it send the stupid trap. Running ethereal show nothing coming into the Kiwi box from the router so I know it is not sending traps. I does however show traffic when you logon to the router or make a config change so I know it is sending traps. We had some trouble with a serial line a week ago and we saw plenty of syslog traps. I really do not know what else to check. Thanks so far!

Joe Clarke · ‎12-06-2006

WHAT traffic do you see when you make a config change? Everything I've seen so far has been a syslog message. Traps are completely different. They are sent on UDP port 162. If the device isn't sending any SNMP TRAPS, a "debug snmp packet" would be useful.

As I said twice before, the "show snmp" stats before a threshold is crossed, then immediately after would also show if IOS is even trying to send a trap.

miwitte · ‎12-06-2006

I had looked at that earlier and I do not see the traps going up. Here are the snippets I just ran; As you can see there were 2 threshold violations, and no trap PDU's. I am sure it is a misconfiguration I just cannot find it.

dr03-clt#sh snmp

Chassis: dr03-clt Cisco7206-Ser# 72789415

2345007 SNMP packets input

0 Bad SNMP version errors

2199 Unknown community name

0 Illegal operation for community name supplied

20 Encoding errors

16567291 Number of requested variables

2703 Number of altered variables

1952333 Get-request PDUs

388382 Get-next PDUs

432 Set-request PDUs

2351388 SNMP packets output

0 Too big errors (Maximum packet size 1500)

85819 No such name errors

0 Bad values errors

0 General errors

2342788 Response PDUs

8600 Trap PDUs

SNMP logging: enabled

Logging to 10.24.160.20.162, 0/10, 4305 sent, 0 dropped.

Logging to 10.28.140.55.162, 0/10, 4295 sent, 0 dropped.

dr03-clt#sh ip sla monitor collection-statistics 270

Entry number: 270

Start Time Index: 17:42:26.164 EST Wed Dec 6 2006

Number of successful operations: 7

Number of operations over threshold: 0

dr03-clt#sh ip sla monitor collection-statistics 270

Entry number: 270

Start Time Index: 17:42:26.166 EST Wed Dec 6 2006

Number of successful operations: 14

Number of operations over threshold: 2

Number of failed operations due to a Disconnect: 0

Number of failed operations due to a Timeout: 0

Number of failed operations due to a Busy: 0

Number of failed operations due to a No Connection: 0

Number of failed operations due to an Internal Error: 0

Number of failed operations due to a Sequence Error: 0

Number of failed operations due to a Verify Error: 0

RTT Values:

RTTAvg: 76 RTTMin: 75 RTTMax: 84

NumOfRTT: 16 RTTSum: 1225 RTTSum2: 93867

dr03-clt#sh snmp

Chassis: dr03-clt Cisco7206-Ser# 72789415

2345043 SNMP packets input

0 Bad SNMP version errors

2199 Unknown community name

0 Illegal operation for community name supplied

20 Encoding errors

16567509 Number of requested variables

2703 Number of altered variables

1952361 Get-request PDUs

388390 Get-next PDUs

432 Set-request PDUs

2351424 SNMP packets output

0 Too big errors (Maximum packet size 1500)

85820 No such name errors

0 Bad values errors

0 General errors

2342824 Response PDUs

8600 Trap PDUs

SNMP logging: enabled

Logging to 10.24.160.20.162, 0/10, 4305 sent, 0 dropped.

Logging to 10.28.140.55.162, 0/10, 4295 sent, 0 dropped.

Joe Clarke · ‎12-06-2006

Well, the router is sending traps which is good. I just now tested your configuration, and I am able to get rising and falling traps as well as a timeout traps, so your config is definitely good.

Go ahead and get the "debug ip sla trace 270" and "debug snmp packet" when one of the thresholds is crossed. Note: if a rising threshold violation has occurred, but the latest RTT value is still above the falling threshold, no more traps will be sent until the falling threshold is crossed.

miwitte · ‎12-07-2006

I'll try to get in early tommorow to do that. I know there is a.00001% chance that these debugs will affect anything, but i am very careful about touching anything in prod especially our hub MPLS router. I had a supervisor fail over on a couple of months ago just adding a vlan interface and all the phones went off line. Luckily it was 7am so it didn't affect anyone you just never know.

I agree that the router is sending traps, just not when the threshold is crossed as can be seen.

As can be seen here we have sent 16 more snmp pdu's since the last post.

dr03-clt#sh snmp

Chassis: dr03-clt Cisco7206-Ser# 72789415

2358128 SNMP packets input

0 Bad SNMP version errors

2199 Unknown community name

0 Illegal operation for community name supplied

20 Encoding errors

16676758 Number of requested variables

2703 Number of altered variables

1963697 Get-request PDUs

390120 Get-next PDUs

432 Set-request PDUs

2364525 SNMP packets output

0 Too big errors (Maximum packet size 1500)

86063 No such name errors

0 Bad values errors

0 General errors

2355909 Response PDUs

8616 Trap PDUs

miwitte · ‎12-08-2006

I have enable the debugs, crossed over the threshold, and have the results. It appears to work, however the sniffer does not show the trap being sent.

1) According to the debugs the threshold caused a snmp V1 trap to be sent to our syslog server

2) A sniffer trace on our internal network shows the syslog debug messages(udp 514) being sent to our syslog server x.x.160.2 showing the same thing as the debug on the router. It never shows a snmpV1 trap being sent. This is a capture on the same vlan as the routers snmp source and logging source x.x.70.2

3) A ethereal dump on the syslog server also shows no snmpV1 packet coming in. It does also show the syslog(udp 514) messages coming in showing the debug output on the router.

4) The Kiwi server has also captured these syslog messages.

So the problem appears to be;

a)Even though the IOS debug shows a snmpV1 packet being sent to x.x.160.2 from x.x.70.2, it appears to not really send it at least out the snmp source port like it said it did.

b)This trap is not adhearing to the configuration of our snmp source of the syslog server for traps, while it does for the syslog messages.

Debugging turned on

debug snmp packets

debug ip sla monitor trace 270

Please note that

x.x.70.2 is the router

x.x.160.20 is the Kiwi server

x.x.140.55 is the IPM server

Dec 8 08:29:33 EST: IP SLA Monitor(270) Scheduler: Starting an operation

Dec 8 08:29:33 EST: IP SLA Monitor(270) echo operation: Sending an echo operation

Dec 8 08:29:33 EST: IP SLA Monitor(270) echo operation: RTT=80

Dec 8 08:29:33 EST: IP SLA Monitor(270) Scheduler: Updating result

Dec 8 08:29:33 EST: SNMP: Queuing packet to x.x.160.20

Dec 8 08:29:33 EST: SNMP: V1 Trap, ent rttMonNotificationsPrefix, addr x.x.70.2, gentrap 6, spectrap 3

rttMonCtrlAdminTag.270 = LTL_ECHO_1400

rttMonHistoryCollectionAddress.270 = 0A 01 3C 12

rttMonCtrlOperOverThresholdOccurred.270 = 1

Dec 8 08:29:33 EST: SNMP: Queuing packet to x.x.140.55

Dec 8 08:29:33 EST: SNMP: V1 Trap, ent rttMonNotificationsPrefix, addr x.x.70.2, gentrap 6, spectrap 3

rttMonCtrlAdminTag.270 = LTL_ECHO_1400

rttMonHistoryCollectionAddress.270 = 0A 01 3C 12

rttMonCtrlOperOverThresholdOccurred.270 = 1

Dec 8 08:29:33 EST: SNMP: Packet sent via UDP to x.x.160.20

Dec 8 08:29:33 EST: SNMP: Packet sent via UDP to x.x.140.55

Dec 8 08:29:40 EST: SNMP: Packet received via UDP from x.x.140.55 on Serial2/0

Dec 8 08:29:40 EST: SNMP: Get request, reqid 13853, errstat 0, erridx 0

sysUpTime.0 = NULL TYPE/VALUE

rttMonEchoAdminTargetAddress.270 = NULL TYPE/VALUE

rttMonLatestRttOperCompletionTime.270 = NULL TYPE/VALUE

rttMonLatestRttOperSense.270 = NULL TYPE/VALUE

rttMonLatestRttOperTime.270 = NULL TYPE/VALUE

rttMonLatestRttOperAddress.270 = NULL TYPE/VALUE

Dec 8 08:29:40 EST: SNMP: Response, reqid 13853, errstat 0, erridx 0

sysUpTime.0 = 2284120129

rttMonEchoAdminTargetAddress.270 = 0A 01 3C 12

rttMonLatestRttOperCompletionTime.270 = 80

rttMonLatestRttOperSense.270 = 3

rttMonLatestRttOperTime.270 = 2284119372

rttMonLatestRttOperAddress.270 =

Dec 8 08:29:40 EST: SNMP: Packet sent via UDP to x.x.140.55

Dec 8 08:29:43 EST: IP SLA Monitor(270) Scheduler: Starting an operation

Dec 8 08:29:43 EST: IP SLA Monitor(270) echo operation: Sending an echo operation

Dec 8 08:29:43 EST: IP SLA Monitor(270) echo operation: RTT=76

Dec 8 08:29:43 EST: IP SLA Monitor(270) Scheduler: Updating result

Joe Clarke · ‎12-08-2006

SNMP traps are sent to upd/162. The most likely cause for what you are seeing is an access-list or firewall somewhere between the router in question and the Kiwi server. This would explain why the router claims to be sending traps, but your Kiwi server doesn't see a one of them (even the non-rtr traps).

The best way to determine this is to put a sniffer directly on the router's outbound interface (i.e. the interface closest to the Kiwi server). See if the traps are showing up there.

miwitte · ‎12-08-2006

Good news. I have been able to get this to work by stopping and starting the snmp server on the router. I see packets on my Kiwi sniffer and also in the logs of the Kiwi. Now I just have to figure out how to get it to just send me a email on just those traps.....

Thanks for all the help!!!

This is from the Kiwi;

08-12-2006 18:02:25 Local7.Debug 10.24.70.2 community=r2fqb26q2td7pc8n enterprise=1.3.6.1.4.1.9.9.42.2.3 enterprise_mib_name=rttMonThresholdNotification uptime=-2007410855 agent_ip=10.24.70.2 generic_num=6 specific_num=3 version=Ver1 var01_oid=1.3.6.1.4.1.9.9.42.1.2.1.1.3.4632 var01_value=Den_ECHO_1400 var01_mib_name=rttMonCtrlAdminTag.4632 var01_value=Den_ECHO_1400 var02_oid=1.3.6.1.4.1.9.9.42.1.4.1.1.5.4632 var02_value="Hex String=0A F9 00 02" var02_mib_name=rttMonHistoryCollectionAddress.4632 var02_value="Hex String=0A F9 00 02" var03_oid=1.3.6.1.4.1.9.9.42.1.2.9.1.7.4632 var03_value=2 var03_mib_name=rttMonCtrlOperOverThresholdOccurred.4632 var03_value=2

OT I saw something about a patch coming out for DFM so it would accept rtt traps from this. Any ideas when it will come out??

Joe Clarke · ‎12-08-2006

Unfortunately, there is no ETA on when DFM will support CISCO-RTTMON-MIB traps. It will most likely not be in the next DFM release.

miwitte · ‎12-09-2006

Thanks for all the help.