IP SLA/EEM running out of VTY lines and failing

[Doug Boreham]

I am using IP SLA to ping network devices to detect network failures from an AS5400XM voice gateway.  The AS5400XM platform is limited to 5 vty lines ( vty 0 4).  When simulating a simultaneous outage, there are not enough tty lines available to process my EEM events.

However, when simulating a simultaneous outage, we have a new issue – there are not enough lines available:
Feb  9 15:16:22.970 CST: %HA_EM-3-FMPD_CLI_CONNECT: Unable to establish CLI session: no tty lines available, minimum of 2 required by EEM
Feb  9 15:16:22.970 CST: %HA_EM-3-FMPD_ERROR: Error executing applet ReportIPSLAevent_1005065020_up statement 1.1
Feb  9 15:16:22.974 CST: %HA_EM-3-FMPD_CLI_CONNECT: Unable to establish CLI session: no tty lines available, minimum of 2 required by EEM
Feb  9 15:16:22.978 CST: %HA_EM-3-FMPD_ERROR: Error executing applet ReportIPSLAevent_1005081020_up statement 1.1
Feb  9 15:16:22.986 CST: %HA_EM-3-FMPD_CLI_CONNECT: Unable to establish CLI session: no tty lines available, minimum of 2 required by EEM
Feb  9 15:16:22.986 CST: %HA_EM-3-FMPD_ERROR: Error executing applet ReportIPSLAevent_100000226_up statement 1.1
Feb  9 15:16:22.994 CST: %HA_EM-3-FMPD_CLI_CONNECT: Unable to establish CLI session: no tty lines available, minimum of 2 required by EEM
Feb  9 15:16:22.994 CST: %HA_EM-3-FMPD_ERROR: Error executing applet ReportIPSLAevent_1001012021_up statement 1.1
Feb  9 15:16:23.006 CST: %HA_EM-3-FMPD_CLI_CONNECT: Unable to establish CLI session: no tty lines available, minimum of 2 required by EEM
Feb  9 15:16:23.006 CST: %HA_EM-3-FMPD_ERROR: Error executing applet ReportIPSLAevent_1061247018_up statement 1.1

How can I detect/wait until there are enough lines free before processing the EEM rules?

Regards,

-Doug

[Joe Clarke]

Post your current applet policy.  What version of IOS is this router running?

[Doug Boreham]

ip sla responder

ip sla logging traps

ip sla 100000226

icmp-echo 10.0.0.226 source-interface Loopback0

request-data-size 200

tos 184

history hours-of-statistics-kept 3

history lives-kept 1

history enhanced interval 900 buckets 100

ip sla schedule 100000226 life forever start-time now

ip sla 100500226

icmp-echo 10.5.0.226 source-interface Loopback0

request-data-size 200

tos 184

history hours-of-statistics-kept 3

history lives-kept 1

history enhanced interval 900 buckets 100

ip sla schedule 100500226 life forever start-time now

ip sla 100564197

icmp-echo 10.5.64.197 source-interface Loopback0

request-data-size 200

tos 184

history hours-of-statistics-kept 3

history lives-kept 1

history enhanced interval 900 buckets 100

ip sla schedule 100564197 life forever start-time now

ip sla 1001011011

icmp-echo 10.0.11.11 source-interface Loopback0

request-data-size 200

tos 184

history hours-of-statistics-kept 3

history lives-kept 1

history enhanced interval 900 buckets 100

ip sla schedule 1001011011 life forever start-time now

ip sla 1001012021

icmp-echo 10.0.12.21 source-interface Loopback0

request-data-size 200

tos 184

history hours-of-statistics-kept 3

history lives-kept 1

history enhanced interval 900 buckets 100

ip sla schedule 1001012021 life forever start-time now

ip sla 1005019195

icmp-echo 10.5.19.195 source-interface Loopback0

ip sla 1005065020

icmp-echo 10.5.65.20 source-interface Loopback0

request-data-size 200

tos 184

history hours-of-statistics-kept 3

history lives-kept 1

history enhanced interval 900 buckets 100

ip sla schedule 1005065020 life forever start-time now

ip sla 1005081020

icmp-echo 10.5.81.20 source-interface Loopback0

request-data-size 200

tos 184

history hours-of-statistics-kept 3

history lives-kept 1

history enhanced interval 900 buckets 100

ip sla schedule 1005081020 life forever start-time now

ip sla 1061247018

icmp-echo 10.61.247.18 source-interface Loopback0

request-data-size 200

tos 184

history hours-of-statistics-kept 3

history lives-kept 1

history enhanced interval 900 buckets 100

ip sla schedule 1061247018 life forever start-time now

ip sla reaction-configuration 100000226 react timeout threshold-type xOfy 3 5 action-type trapOnly

ip sla reaction-configuration 100500226 react timeout threshold-type xOfy 3 5 action-type trapOnly

ip sla reaction-configuration 100564197 react timeout threshold-type xOfy 3 5 action-type trapOnly

ip sla reaction-configuration 1001012021 react timeout threshold-type xOfy 3 5 action-type trapOnly

ip sla reaction-configuration 1005065020 react timeout threshold-type xOfy 3 5 action-type trapOnly

ip sla reaction-configuration 1005081020 react timeout threshold-type xOfy 3 5 action-type trapOnly

ip sla reaction-configuration 1061247018 react timeout threshold-type xOfy 3 5 action-type trapOnly

!

event manager environment _mail_smtp my.nameserver.com

event manager environment _mail_rcpt myemailaddress@mydomainname.com

event manager environment _mail_domain mydomainname.com

event manager applet weekdays

event timer cron name weekdays cron-entry "0 9 * * 1-5"

action 1.00001 cli command "enable"

action 1.00002 cli command "conf t"

action 1.00381 cli command "dial-peer voice 38 voip"

action 1.00382 cli command "max-conn 0"

action 1.14851 cli command "dial-peer voice 1485 voip"

action 1.14852 cli command "max-conn 0"

action 9.99997 cli command "end"

action 9.99998 cli command "wr mem"

action 9.99999 cli command "exit"

event manager applet afterhours

event timer cron name afterhours cron-entry "0 21 * * 1-5"

action 1.00001 cli command "enable"

action 1.00002 cli command "conf t"

action 1.00381 cli command "dial-peer voice 38 voip"

action 1.00382 cli command "max-conn "

action 1.14851 cli command "dial-peer voice 1485 voip"

action 1.14852 cli command "max-conn "

action 9.99997 cli command "end"

action 9.99998 cli command "wr mem"

action 9.99999 cli command "exit"

event manager applet weekends

event timer cron name weekends cron-entry "1 21 * * 5"

action 1.00001 cli command "enable"

action 1.00002 cli command "conf t"

action 1.00381 cli command "dial-peer voice 38 voip"

action 1.00382 cli command "max-conn "

action 1.14851 cli command "dial-peer voice 1485 voip"

action 1.14852 cli command "max-conn "

action 9.99997 cli command "end"

action 9.99998 cli command "wr mem"

action 9.99999 cli command "exit"

!

event manager applet ReportIPSLAevent_100000226_down

event syslog pattern "RTT-3-IPSLATHRESHOLD: IP SLA.*100000226.* Threshold Occurred for timeout"

action 1.0     info type routername

action 1.1     cli command "show ip sla statistics 100000226"

action 2.0     mail server "$_mail_smtp" to "$_mail_rcpt" from "$_info_routername@$_mail_domain" subject "IP SLA alert - $_info_routername connectivity to [other_server] is down" body "$_cli_result"

action 9.99999 cli command "exit"

event manager applet ReportIPSLAevent_100000226_up

event syslog pattern "RTT-3-IPSLATHRESHOLD: IP SLA.*100000226.* Threshold Cleared for timeout"

action 1.0     info type routername

action 1.1     cli command "show ip sla statistics 100000226"

action 2.0     mail server "$_mail_smtp" to "$_mail_rcpt" from "$_info_routername@$_mail_domain" subject "IP SLA alert - $_info_routername connectivity to [other_server] has been restored" body "$_cli_result"

action 9.99999 cli command "exit"

event manager applet ReportIPSLAevent_100500226_down

event syslog pattern "RTT-3-IPSLATHRESHOLD: IP SLA.*100500226.* Threshold Occurred for timeout"

action 1.0     info type routername

action 1.1     cli command "show ip sla statistics 100500226"

action 2.0     mail server "$_mail_smtp" to "$_mail_rcpt" from "$_info_routername@$_mail_domain" subject "IP SLA alert - $_info_routername connectivity to [other_server] is down" body "$_cli_result"

action 9.99999 cli command "exit"

event manager applet ReportIPSLAevent_100564197_down

event syslog pattern "RTT-3-IPSLATHRESHOLD: IP SLA.*100564197.* Threshold Occurred for timeout"

action 1.0     info type routername

action 1.1     cli command "show ip sla statistics 100564197"

action 2.0     mail server "$_mail_smtp" to "$_mail_rcpt" from "$_info_routername@$_mail_domain" subject "IP SLA alert - $_info_routername connectivity to [other_server] is down" body "$_cli_result"

action 9.99999 cli command "exit"

event manager applet ReportIPSLAevent_1001011011_down

event syslog pattern "RTT-3-IPSLATHRESHOLD: IP SLA.*1001011011.* Threshold Occurred for timeout"

action 1.0     info type routername

action 1.1     cli command "show ip sla statistics 1001011011"

action 2.0     mail server "$_mail_smtp" to "$_mail_rcpt" from "$_info_routername@$_mail_domain" subject "IP SLA alert - $_info_routername connectivity to [other_server] is down" body "$_cli_result"

action 9.99999 cli command "exit"

event manager applet ReportIPSLAevent_1001012021_down

event syslog pattern "RTT-3-IPSLATHRESHOLD: IP SLA.*1001012021.* Threshold Occurred for timeout"

action 1.0     info type routername

action 1.1     cli command "show ip sla statistics 1001012021"

action 2.0     mail server "$_mail_smtp" to "$_mail_rcpt" from "$_info_routername@$_mail_domain" subject "IP SLA alert - $_info_routername connectivity to [other_server] is down" body "$_cli_result"

action 9.99999 cli command "exit"

event manager applet ReportIPSLAevent_1005065020_down

event syslog pattern "RTT-3-IPSLATHRESHOLD: IP SLA.*1005065020.* Threshold Occurred for timeout"

action 1.0     info type routername

action 1.1     cli command "show ip sla statistics 1005065020"

action 2.0     mail server "$_mail_smtp" to "$_mail_rcpt" from "$_info_routername@$_mail_domain" subject "IP SLA alert - $_info_routername connectivity to [other_server] is down" body "$_cli_result"

action 9.99999 cli command "exit"

event manager applet ReportIPSLAevent_1005081020_down

event syslog pattern "RTT-3-IPSLATHRESHOLD: IP SLA.*1005081020.* Threshold Occurred for timeout"

action 1.0     info type routername

action 1.1     cli command "show ip sla statistics 1005081020"

action 2.0     mail server "$_mail_smtp" to "$_mail_rcpt" from "$_info_routername@$_mail_domain" subject "IP SLA alert - $_info_routername connectivity to [other_server] is down" body "$_cli_result"

action 9.99999 cli command "exit"

event manager applet ReportIPSLAevent_1005019195_down

event syslog pattern "RTT-3-IPSLATHRESHOLD: IP SLA.*1005019195.* Threshold Occurred for timeout"

action 1.0     info type routername

action 1.1     cli command "show ip sla statistics 1005019195"

action 2.0     mail server "$_mail_smtp" to "$_mail_rcpt" from "$_info_routername@$_mail_domain" subject "IP SLA alert - $_info_routername connectivity to [other_server] is down" body "$_cli_result"

action 9.99999 cli command "exit"

event manager applet ReportIPSLAevent_1061247018_down

event syslog pattern "RTT-3-IPSLATHRESHOLD: IP SLA.*1061247018.* Threshold Occurred for timeout"

action 1.0     info type routername

action 1.1     cli command "show ip sla statistics 1061247018"

action 2.0     mail server "$_mail_smtp" to "$_mail_rcpt" from "$_info_routername@$_mail_domain" subject "IP SLA alert - $_info_routername connectivity to [other_server] is down" body "$_cli_result"

action 9.99999 cli command "exit"

event manager applet ReportIPSLAevent_100500226_up

event syslog pattern "RTT-3-IPSLATHRESHOLD: IP SLA.*100500226.* Threshold Cleared for timeout"

action 1.0     info type routername

action 1.1     cli command "show ip sla statistics 100500226"

action 2.0     mail server "$_mail_smtp" to "$_mail_rcpt" from "$_info_routername@$_mail_domain" subject "IP SLA alert - $_info_routername connectivity to [other_server] has been restored" body "$_cli_result"

action 9.99999 cli command "exit"

event manager applet ReportIPSLAevent_100564197_up

event syslog pattern "RTT-3-IPSLATHRESHOLD: IP SLA.*100564197.* Threshold Cleared for timeout"

action 1.0     info type routername

action 1.1     cli command "show ip sla statistics 100564197"

action 2.0     mail server "$_mail_smtp" to "$_mail_rcpt" from "$_info_routername@$_mail_domain" subject "IP SLA alert - $_info_routername connectivity to [other_server] has been restored" body "$_cli_result"

action 9.99999 cli command "exit"

event manager applet ReportIPSLAevent_1001011011_up

event syslog pattern "RTT-3-IPSLATHRESHOLD: IP SLA.*1001011011.* Threshold Cleared for timeout"

action 1.0     info type routername

action 1.1     cli command "show ip sla statistics 1001011011"

action 2.0     mail server "$_mail_smtp" to "$_mail_rcpt" from "$_info_routername@$_mail_domain" subject "IP SLA alert - $_info_routername connectivity to [other_server] has been restored" body "$_cli_result"

action 9.99999 cli command "exit"

event manager applet ReportIPSLAevent_1001012021_up

event syslog pattern "RTT-3-IPSLATHRESHOLD: IP SLA.*1001012021.* Threshold Cleared for timeout"

action 1.0     info type routername

action 1.1     cli command "show ip sla statistics 1001012021"

action 2.0     mail server "$_mail_smtp" to "$_mail_rcpt" from "$_info_routername@$_mail_domain" subject "IP SLA alert - $_info_routername connectivity to [other_server] has been restored" body "$_cli_result"

action 9.99999 cli command "exit"

event manager applet ReportIPSLAevent_1005065020_up

event syslog pattern "RTT-3-IPSLATHRESHOLD: IP SLA.*1005065020.* Threshold Cleared for timeout"

action 1.0     info type routername

action 1.1     cli command "show ip sla statistics 1005065020"

action 2.0     mail server "$_mail_smtp" to "$_mail_rcpt" from "$_info_routername@$_mail_domain" subject "IP SLA alert - $_info_routername connectivity to [other_server]  has been restored" body "$_cli_result"

action 9.99999 cli command "exit"

event manager applet ReportIPSLAevent_1005081020_up

event syslog pattern "RTT-3-IPSLATHRESHOLD: IP SLA.*1005081020.* Threshold Cleared for timeout"

action 1.0     info type routername

action 1.1     cli command "show ip sla statistics 1005081020"

action 2.0     mail server "$_mail_smtp" to "$_mail_rcpt" from "$_info_routername@$_mail_domain" subject "IP SLA alert - $_info_routername connectivity to [other_server] has been restored" body "$_cli_result"

action 9.99999 cli command "exit"

event manager applet ReportIPSLAevent_1005019195_up

event syslog pattern "RTT-3-IPSLATHRESHOLD: IP SLA.*1005019195.* Threshold Cleared for timeout"

action 1.0     info type routername

action 1.1     cli command "show ip sla statistics 1005019195"

action 2.0     mail server "$_mail_smtp" to "$_mail_rcpt" from "$_info_routername@$_mail_domain" subject "IP SLA alert - $_info_routername connectivity to [other_server] has been restored" body "$_cli_result"

action 9.99999 cli command "exit"

event manager applet ReportIPSLAevent_1061247018_up

event syslog pattern "RTT-3-IPSLATHRESHOLD: IP SLA.*1061247018.* Threshold Cleared for timeout"

action 1.0     info type routername

action 1.1     cli command "show ip sla statistics 1061247018"

action 2.0     mail server "$_mail_smtp" to "$_mail_rcpt" from "$_info_routername@$_mail_domain" subject "IP SLA alert - $_info_routername connectivity to [other_server] has been restored" body "$_cli_result"

action 9.99999 cli command "exit"

!

end

[Joe Clarke]

Do you want to maintain all of these separate syslog policies?  They could be combined into one.

[Doug Boreham]

Sure - I would be open to combining them.  I'm relatively new to writing EEM programs, so I am open to suggestions on ways to streamline the configuration. 

-Doug

[Joe Clarke]

Here you go.  This policy should replace all of your occurred and cleared IP SLA applets.  It will also automatically scale to any new IP SLA collectors you create.  You will need to first create a directory on local flash to store EEM Tcl policies.  Then copy this script into that directory (I use something like flash:/policies).  Then configure the following:

event manager directory user policy POLDIR
event manager policy sl_ip_sla_report.tcl

Where POLDIR is the directory where you copied the sl_ip_sla_report.tcl script.

Be sure to remove all of your IP SLA applet policies first.

[Doug Boreham]

Joe,

Thanks for the update.  I now realize that I missed giving you a key piece of information.  In order to not list the specific hostnames in this post, I removed those individual names and replaced it with the generic text "[other_server]"

Subject: IP SLA alert - $_info_routername connectivity to [other_server]

There is a specific server tied to each IP SLA poll, and I want the e-mail subject to reflect the hostname of the IP address being polled.  In most but not all cases, that hostname is registered in DNS if that helps.

For example, in the code:

ip sla 100000226

icmp-echo 10.0.0.226 source-interface Loopback0

request-data-size 200

tos 184

history hours-of-statistics-kept 3

history lives-kept 1

history enhanced interval 900 buckets 100

ip sla schedule 100000226 life forever start-time now

ip sla 100500226

icmp-echo 10.5.0.226 source-interface Loopback0

request-data-size 200

tos 184

history hours-of-statistics-kept 3

history lives-kept 1

history enhanced interval 900 buckets 100

ip sla schedule 100500226 life forever start-time now

let's say that the hostname of 10.0.0.226 is appserver01 and the hostname of 10.5.0.226 is devbox.  I would want to send out e-mails with the following respective subjects if ping [fails/restores]:

For 10.0.0.226:

Subject: IP SLA alert - $_info_routername connectivity to appserver01 {is down | has been restored}

For 10.5.0.226:

Subject: IP SLA alert - $_info_routername connectivity to devbox {is down | has been restored}

-Doug

[Joe Clarke]

Try this version.  You will need to first set an environment variable for each IP-to-hostname mapping.  For example, if 10.0.0.226 maps to appserver01, and 10.5.0.226 maps to devbox, add:

event manager environment hostname_10_5_0_226 devbox
event manager environment hostname_10_0_0_226 appserver01

Do this for each IP SLA target IP.

[Doug Boreham]

Joe,

I am getting a compilation error:

show debug

General OS:
  Event Manager debugging is on

Embedded Event Manager:
  Debug EEM Tcl commands debugging is on

router(config)#event manager policy sl_ip_sla_report.tcl
EEM configuration: failed to create the event spec for policy sl_ip_sla_report.tcl: regular expression compilation error
router(config)#
Feb 16 14:38:17.069 CST: fh_tcl_get_mode: mode = 3, StartupScript = flash:/policies/sl_ip_sla_report.tcl, RealScript = flash:/policies/sl_ip_sla_report.tcl
Feb 16 14:38:17.073 CST: fh_tcl_assoc_data_delproc: freeing tctx=0x64E304BC
Feb 16 14:38:17.077 CST: fh_tcl_get_mode: mode = 0, StartupScript = flash:/policies/sl_ip_sla_report.tcl, RealScript = flash:/policies/sl_ip_sla_report.tcl
Feb 16 14:38:17.077 CST: fh_set_tclpath_global: tcl_library is set to tmpsys:/lib/tcl
Feb 16 14:38:17.077 CST: fh_set_tclpath_global: auto_path is set to tmpsys:/eem_lib_user tmpsys:/eem_lib_system
Feb 16 14:38:17.077 CST: fh_register_evreg_cmds: tctx=64E304BC, dummy=0
Feb 16 14:38:17.081 CST: fh_register_event_cmn: filename=flash:/policies/sl_ip_sla_report.tcl
Feb 16 14:38:17.085 CST: fh_event_register_cmd: policy_name=sl_ip_sla_report.tcl
Feb 16 14:38:17.085 CST: fh_event_register_cmd: stmt=::cisco::eem::event_register_syslog#pattern#RTT-3-IPSLATHRESHOLD: IP SLA.*+.* Threshold (Cleared|Occurred) for timeout#
Feb 16 14:38:17.085 CST: EM: No consumer registered for any event type
Feb 16 14:38:17.085 CST: Keyword: pattern Argument: "RTT-3-IPSLATHRESHOLD: IP SLA.*+.* Threshold (Cleared|Occurred) for timeout"
Feb 16 14:38:17.085 CST: fh_event_register_cmd: rc=0
Feb 16 14:38:17.085 CST: fh_tcl_assoc_data_delproc: freeing tctx=0x64E304BC

[Joe Clarke]

It looks like you modified my script.  The first line should read:

::cisco::eem::event_register_syslog pattern "RTT-3-IPSLATHRESHOLD: IP SLA.*[0-9]+.* Threshold (Cleared|Occurred) for timeout"

This is a valid regular expression.

[Doug Boreham]

Joe,

There must be some other reason, as the script was installed as sent via FTP.

vgw01.it.wstia#dir
Directory of flash:/policies/

   15  -rw-        1922  Feb 16 2010 14:34:44 -06:00  sl_ip_sla_report.tcl

128733184 bytes total (95617024 bytes free)

Variables:

event manager environment __easy_PREFIX flash:/easy
event manager environment _mail_smtp mailserver.mydomain.com
event manager environment _mail_rcpt hiddenemail@mydomain.com
event manager environment _mail_domain mydomain.com
event manager environment hostname_10_0_0_226 hostname01

event manager environment hostname_10_5_64_197 hostname02
event manager environment hostname_10_0_12_21 hostname03
event manager environment hostname_10_0_11_11 hostname04
event manager environment hostname_10_0_11_12 hostname05
event manager environment hostname_10_5_81_20 hostname06
event manager environment hostname_10_8_19_20 hostname07

Here is the script from the router:

::cisco::eem::event_register_syslog pattern "RTT-3-IPSLATHRESHOLD: IP SLA.*[0-9]+.* Threshold (Cleared|Occurred) for timeout"

namespace import ::cisco::eem::*
namespace import ::cisco::lib::*

array set _sinfo [sys_reqinfo_routername]
set _info_routername $_sinfo(routername)

if [catch {cli_open} result] {
    error $result $errorInfo
} else {
    array set cli1 $result
}

array set arr_einfo [event_reqinfo]
set msg $arr_einfo(msg)

if { ! [regexp {RTT-3-IPSLATHRESHOLD: IP SLA.*\(([0-9]+)\).* Threshold (Cleared|Occurred) for timeout} $msg -> collector type] } {
    action_syslog msg "Failed to find collector ID in syslog"
    exit 1
}

if [catch {cli_exec $cli1(fd) "show ip sla statistics $collector"} iplsa_stat] {
    error $ipsla_stat $errorInfo
}

set stat {is down}
if { $type == "Cleared" } {
    set stat {has been restored}
}

if { [catch {cli_exec $cli1(fd) "show ip sla conf $collector"} _cli_result] } {
    error $_cli_result $errorInfo
}

set other_server {Unknown}
if { [regexp {Target address: ([0-9\.]+)} $_cli_result -> target_ip] } {
    regsub -all {\.} $target_ip {_} varname
    if { [info exists hostname_${varname}] } {
        set other_server [expr "\$hostname_$varname"]
     } else {
         set other_server $target_ip
     }
}

set mail_pre "Mailservername: $_mail_smtp\n"
append mail_pre "From: $_info_routername@$_mail_domain\n"
append mail_pre "To: $_mail_rcpt\n"
append mail_pre
append mail_pre "Subject: IP SLA alert - $_info_routername connectivity to $other_server $stat\n\n"
append mail_pre "$ipsla_stat\n\n"
set mail_msg [uplevel #0 [list subst -nobackslashes -nocommands $mail_pre]]
if [catch {smtp_send_email $mail_msg} result] {
    error $result $errorInfo
}

if [catch {cli_exec $cli1(fd) "exit"} _cli_result] {
    error $_cli_result $errorInfo
}

# Close open cli before exit.
if [catch {cli_close $cli1(fd) $cli1(tty_id)} result] {
    error $result $errorInfo
}

Regards,

-Doug

[Joe Clarke]

Ugh, now that you display what I sent I see the problem.  The regexp in the ED line is invalid Tcl.  You need to put a '\' in front of the '[':

::cisco::eem::event_register_syslog pattern "RTT-3-IPSLATHRESHOLD: IP SLA.*\[0-9]+.* Threshold (Cleared|Occurred) for timeout"

That will work.

[Doug Boreham]

Joe,

There appears to be an error - Tcl policy execute failed: can't read "ipsla_stat": no such variable:

show debug

General OS:

Event Manager debugging is on

IP SLAs:

TRACE debugging for all operations is on

Embedded Event Manager:

Debug EEM Tcl commands debugging is on

Debug EEM Tcl CLI library debugging is on

Debug EEM Tcl SMTP library debugging is on

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : IP SLAs, Infrastructure Engine-II.

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Entry number: 1061247018

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Owner:

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Tag:

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Type of operation to perform: icmp-echo

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Target address/Source address: 0.0.0.0/10.61.247.18

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Operation timeout (milliseconds): 5000

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Type Of Service parameters: 0xB8

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Vrf Name:

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Request size (ARR data portion): 200

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Verify data: No

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Schedule:

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Operation frequency (seconds): 60 (not considered if randomly scheduled)

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Next Scheduled Start Time: Start Time already passed

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Group Scheduled : FALSE

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Randomly Scheduled : FALSE

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Life (seconds): Forever

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Entry Ageout (seconds): never

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Recurring (Starting Everyday): FALSE

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Status of entry (SNMP RowStatus): Active

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : CTL : 20+ lines read from cli_read, debug output truncated.

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl: can't read "ipsla_stat": no such variable

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl: while executing

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl: "append mail_pre "$ipsla_stat\n\n""

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl: invoked from within

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl: "$slave eval $Contents"

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl: (procedure "eval_script" line 7)

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl: invoked from within

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl: "eval_script slave $scriptname"

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl: invoked from within

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl: "if {$security_level == 1} { #untrusted script

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl: interp create -safe slave

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl: interp share {} stdin slave

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl: interp share {} stdout slave

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl: ..."

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl: (file "tmpsys:/lib/tcl/base.tcl" line 50)

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl: Tcl policy execute failed:

Feb 17 12:57:37.837 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl: can't read "ipsla_stat": no such variable

Feb 17 12:57:37.841 CST: fh_tcl_esi_close: fd=11

Feb 17 12:57:37.841 CST: fh_tcl_esi_close: fd=12

Feb 17 12:57:37.841 CST: fh_tcl_assoc_data_delproc: freeing tctx=0x69968274

Regards,

-Doug

[Joe Clarke]

I had another typo.  Here is a fixed version.

[Doug Boreham]

Joe,

I think we're getting close!

Now we get an error on the e-mail portion - Tcl policy execute failed: %HA_EM-6-LOG: sl_ip_sla_report.tcl: wrong 4th line format:

Feb 17 14:09:30.669 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : IP SLAs, Infrastructure Engine-II.

Feb 17 14:09:30.669 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Entry number: 1005081020

Feb 17 14:09:30.669 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Owner:

Feb 17 14:09:30.669 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Tag:

Feb 17 14:09:30.669 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Type of operation to perform: icmp-echo

Feb 17 14:09:30.669 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Target address/Source address: 0.0.0.0/10.5.81.20

Feb 17 14:09:30.669 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Operation timeout (milliseconds): 5000

Feb 17 14:09:30.669 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Type Of Service parameters: 0xB8

Feb 17 14:09:30.669 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Vrf Name:

Feb 17 14:09:30.669 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Request size (ARR data portion): 200

Feb 17 14:09:30.669 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Verify data: No

Feb 17 14:09:30.669 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Schedule:

Feb 17 14:09:30.669 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Operation frequency (seconds): 60 (not considered if randomly scheduled)

Feb 17 14:09:30.669 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Next Scheduled Start Time: Start Time already passed

Feb 17 14:09:30.669 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Group Scheduled : FALSE

Feb 17 14:09:30.669 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Randomly Scheduled : FALSE

Feb 17 14:09:30.669 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Life (seconds): Forever

Feb 17 14:09:30.669 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Entry Ageout (seconds): never

Feb 17 14:09:30.669 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Recurring (Starting Everyday): FALSE

Feb 17 14:09:30.669 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : OUT : Status of entry (SNMP RowStatus): Active

Feb 17 14:09:30.669 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl : DEBUG(cli_lib) : CTL : 20+ lines read from cli_read, debug output truncated.

Feb 17 14:09:30.673 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl: wrong 4th line format.

Feb 17 14:09:30.673 CST: %HA_EM-6-LOG: sl_ip_sla_report.tcl: usage: Cc:

Feb 17 14:09:30.677 CST: fh_tcl_esi_close: fd=11

Feb 17 14:09:30.677 CST: fh_tcl_esi_close: fd=12

Feb 17 14:09:30.677 CST: fh_tcl_assoc_data_delproc: freeing tctx=0x6B0ED1CC

Regards,

-Doug