EEM Script for Monitoring Site Equipment

[snetherland]

Like most companies, our primary monitoring strategy centers around SNMP with one or a few management stations that are usually remote from the equipment being monitored. I would really like to move towards leveraging EEM for some of our monitoring. At the moment, I am using IP SLA with EOT and EEM applets to monitor on-site LAN equipment and send out email notifications. This can be fairly resource heavy and I know that I am not taking advantage of the SNMP library that the current version of EEM supports.

It would really be great if there was a TCL script which performed the following:

-monitor local equipment via SNMP

-generate customizable alerts and possibly attempt to resolve on-site issues with SNMP Set commands

-maintain a relatively small footprint on the device(resource usage low as possible)

Has anyone been successful in deploying something similar to what I am describing?

Thanks to everyone who will take the time to read this. I appreciate any help.

[Joe Clarke]

You might want to browse the repository at http://www.cisco.com/go/ciscobeyond . You may find something there that can get you started. Your requirements are fairly general, but I can say that EEM+EOT+IP SLA is not that heavy-weight. It's a very good system for doing near real-time monitoring of the network (as opposed to just a single device).

If you do want to get started on something yourself, see the EEM Tcl policy guide at http://www.cisco.com/en/US/docs/ios/netmgmt/configuration/guide/nm_eem_policy_tcl_ps6441_TSD_Products_Configuration_Guide_Chapter.html#wp1043715 . Feel free to ask more questions in this forum about specific issues you encounter.

[snetherland]

Joe, thanks for taking the time to respond. I think I speak for all networking engineers who have ZERO scripting background when I say that you are a valuable resource to us. I have actually looked through the EEM scripting community repository before, but after I received your message I went back and spent a little time crawling through each script. The problem that I seem to be running into is that I do not have a scripting background(however, seeing the massive potential for improved[or even introducing new] features and customizable operations, I am defiantly adding this to my current studies).

Something that would help me here at the outset would be an understanding of what built-in environment variables I have to work with. I know that EEM supports predefined variables that tie-in to system processes and I know that those variables can be referenced in both EEM applets and TCL scripts, but so far I haven't been able to find a full listing of what environment variables are offered by default for a particular IOS image. I have found a few partial lists, and have pieced together other supported variables that I have gleaned from other documentation, but by no means do I currently have an in-depth list. Do you know of any document(s) that may supply this information? Or, have you(or anyone else that may be reading) put together a list of supported variables that you use for writing policies?

Again, thanks for taking the time to read through this. I appreciate everyone's help.

[Joe Clarke]

If you have a device running EEM 2.4 or higher (e.g. 12.4(20)T) then you can run the command:

show event manager detector DETECTOR detail

To see all of the supported environment variables per event detector. As for documents, internally, the links I've already posted are what we have now. I believe some customers have put together some good external sites. You might try searching Google for EEM links.

Something we're getting ready to start is the EASy program. The goal of EASy is to make leveraging EEM and other embedded technologies, easy. We will be posting well-documented, well-tested packages which accomplish specific tasks which have been suggested by our customers and sales teams. These packages will be easy to install using a Tcl-based installer which runs directly on IOS. Once that program launches, you will have an opportunity to suggest new use cases which meet your requirements.

[snetherland]

Joe,

Thanks so much for your response. All of your posts have been very helpful in getting us started, and I am happy to report that we are making excellent progress. I have a question in regards to the snmp library found in EEM 3.0; more specifically issuing an snmp-set command from the router to another device on the LAN. To provide you with a scenario, our test lab consists of an 871 running 12.4(24)T1, a 3550 EMI, and a few 1200 series Aironets running as autonomous APs. Our immediate goal is to use IP SLA to monitor AP availability, track it with EOT, and, in the event an AP is no longer responsive, issue an snmp-set command informing the switch to administratively drop the corresponding interface(removeing power to the AP) and then bring the interface back up(in effect power-cycling that access point). The issues I am running into have more to do with my lack of SNMP knowledge than anything. Looking at the IF-MIB, I am seeing ifAdminStatus as a read-write that allows you to specify a 1(up) or 2(down) for interface status. Using the Object Navigator tool, I am seeing the OID as 1.3.6.1.2.1.2.2.1.7

My questions are: 1)how do I define the specific interface that I wish to apply the set command to?(for instance, I wish to bounce interface fa0/10)

2)should the value specifying the up/down command be appended to the end of this sting?(for example, 1.3.6.1.2.1.2.2.1.7.2)

Thanks to everyone that will take the time to read through this post. All help is greatly appreciated.

[Joe Clarke]

1. You need to find the ifIndex for this interface. This can be done using the command, "show snmp mib ifmib ifindex" on the target device.

2. No. The ifIndex will need to be appended to the OID (e.g. 1.3.6.1.2.1.2.2.1.7.2.3 for ifIndex 3). The value will be specified in the appropriate value field in your Tcl command. For example, to set the interface down:

sys_reqinfo_snmp_set oid 1.3.6.1.2.1.2.2.1.7.2.3 int 2 community private ipaddr 1.1.1.1

[snetherland]

Joe, this worked fantastic! I am at the point now where I am attempted to visualize a logic flow for a potential TCL policy. This is what I have so far(I appreciate your input and guidance):

-ip sla is used as the probing method to determine device availability

-eot is used as both the event detector as well as for variable reference

-for each AP that is probed(let's say 10), an ip sla operation as well as an eot track is configured

-one ANDed track referencing all other tracks is configured; this will be used as the event detector

-each eot that is tracking an sla operation is mapped to a unique OID used by the snmp-set command

-once ANDed track = down, Policy1 starts

Policy1

-read each track state

-if track state = down

then: a)lookup OID value mapped to that track

b)reference OID value in 2 snmp-set commands(down & then up)

c)verify interface has moved from admin down to up state

d)remove tracked object from ANDed track

e)start Policy2

-end Policy1

Policy2

-read track state provided by Policy1 at specific intervals for a specific time period(rebooted device may take a minute or two to come up)

-if track state = up

then: a)gather diagnostic information

b)generate email or syslog message containing information

c)add track back to ANDed track

-if track state = down

then: a)gather diagnostic information

b)generate email or syslog message containing information

-end Policy2

I suppose the first real legitimate question I should ask is do you feel this will work to satisfy our needs? Could you see this easily fleshed-out with TCL scripting?

My primary concern is the occurrence of multiple track states dropping. For instance, if 2 APs drop 10 seconds a part from each other, Policy1 will have started for the first and the second will possibly interrupt that policy or simply be dropped.

We are completely open to suggestions and we appreciate everyone's help.

[Joe Clarke]

The first question is really yours to answer. If this logic is sufficient to your needs, then the answer is yes.

Yes, both policies could be fleshed out with Tcl. You don't have to worry about overlap since by default, there is only one policy thread running. Additionally, Policy1 will be single-threaded, so a second Policy1 will not run.