EEM "bounce" port when cannot ping internet

[Michael Durham]

I have a C3750v2 switch running IOS 15.0(2)SE7 and every once in a while my internet router screws up requiring a reset.  I am testing internet connectivity and if not reply after 10 seconds, bounce the port.    The internet router is by T-Mobile and runs off the PoE port so all I need to bounce (shut - no shut) the port.  I have tried setting up an ip sla and an eem to make this happen when the switch cannot ping the internet, 4.2.2.2.  However, I am missing something because its not working.  Also, I would like to have an email sent once the port and router are back up.  It takes about 3 minutes for the T-Mobile router to fully reboot.  What am I missing?  What needs changed?  This is running on the C3750v2 switch, not a Cisco router.

!
track 100 ip sla 100 reachability
delay down 10 up 60
!
ip sla 100
icmp-echo 4.2.2.2
threshold 750
timeout 900
frequency 15
ip sla schedule 100 life forever start-time now
ip sla reaction-configuration 100 react timeout threshold-type immediate
ip sla enable reaction-alerts
!
event manager applet reset_interface
event ipsla operation-id 100 reaction-type timeout
action 1 if $_ipsla_condition eq "Occurred"
action 2 cli command "conf t"
action 3 cli command "interface gi1/0/45"
action 4 cli command "shut"
action 5 wait 10
action 6 cli command "no shut"
action 7 cli command "end"
action 5 wait 200
action 8 mail server "mail.privateemail.com"
action 9 mail from "<support@certificationtrainingsolutions.com>"
action 10 mail to "<mdurham@thedurhamcorporation.com>"
action 11 mail subject "Switch Port Reset Notification"
action 12 mail body "The T-Mobile router was reset."
action 13 mail server login "<support@certificationtrainingsolutions.com>" "<SMTP_PASSWORD>"
action 14 mail server protocol smtp ssl
!

[MHM Cisco World]

Spoiler

Hi friend 
I lab EEM and I success make interface down then wait 10 then Up again depend on reaction of IP SLA,  the point it dont clear is 
why you use track ? no need track since we use reaction 
why you use if ? we dont need if in EEM 

last point the wait 10 is so short, in my lab the interface is down then Up so quickly.
waiting your reply to adjust the EEM 

track 100 ip sla 100 reachability
delay down 10 up 60

!
ip sla 100
icmp-echo 4.2.2.2
threshold 750
timeout 900
frequency 15
ip sla schedule 100 life forever start-time now
ip sla reaction-configuration 100 react timeout threshold-type immediate
ip sla enable reaction-alerts
!
event manager applet reset_interface
event ipsla operation-id 100 reaction-type timeout
action 1 if $_ipsla_condition eq "Occurred" <<- why if statement ?
action 2 cli command "conf t"
action 3 cli command "interface gi1/0/45"
action 4 cli command "shut"
action 5 wait 10
action 6 cli command "no shut"
action 7 cli command "end"

[Michael Durham]

No idea, I was just researching the web for a solution and that was part of it.  Don't know the correct config to fix it.

[MHM Cisco World]

I need more time to solve issue of wait and monitoring interface up down in multi failure case.

Update you hope tomorrow 

MHM

[balaji.bandi]

First i would start with basic see if that working - the  go to email sending other stuff for monitoring : (if you are using any TACACS server for external autentication you need to bypass)

event manager applet reset_interface
event ipsla operation-id 100 reaction-type timeout
action 1 cli command "enable"
action 2 cli command "conf t"
action 3 cli command "interface gi1/0/45"
action 4 cli command "shut"
action 5 wait 10
action 6 cli command "no shut"
action 7 cli command "end"

[Michael Durham]

I have tested the IPSLA and it seems to be working But not the EEM part.  No TACACS server.  When I try to test the EEM, I get the following error:

Shop#event manager run reset_interface
EEM policy reset_interface not registered with event none Event Detector

Shop#sh track 100
Track 100
IP SLA 100 reachability
Reachability is Up
48 changes, last change 00:26:00
Delay up 60 secs, down 10 secs
Latest operation return code: OK
Latest RTT (millisecs) 84

Shop#sh ip sla su

IPSLAs Latest Operation Summary
ID Type Destination Stats Return Last
(ms) Code Run
----------- ---------- --------------- ------ ---------- -----------------
*100 icmp-echo 4.2.2.2 RTT=91 OK 12 seconds ago

[balaji.bandi]

check the applet registered - check some troubleshooting tips :

https://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/ios-xe-16/217013-troubleshoot-and-test-eem-scripts.html#:~:text=To%20verify%20your%20EEM%20script,your%20script%20has%20not%20triggered.

[Michael Durham]

The applet is not registering.  I tried everything for testing but no luck.  Also, this is a C3750v2 switch running IOS 15.0 and some EEM commands are just not there.  Still need help getting this to work on the switch....

[Michael Durham]

I manually shut down the switch port with debug event manager running and here is the results.  As you can see, the IP SLA is working but nothing seems to be triggering the EEM that I configured.

Shop(config-if)#int g1/0/45
Shop(config-if)#shut
Shop(config-if)#
Dec 23 11:46:42: %LINK-5-CHANGED: Interface GigabitEthernet1/0/45, changed state to administratively down
Dec 23 11:46:43: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to down
Shop(config-if)#
Dec 23 11:47:03: %TRACKING-5-STATE: 100 ip sla 100 reachability Up->Down
Shop(config-if)#no shut
Shop(config-if)#
Dec 23 11:49:08: %ILPOWER-7-DETECT: Interface Gi1/0/45: Power Device detected: IEEE PD
Shop(config-if)#
Dec 23 11:49:09: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to down
Shop(config-if)#
Dec 23 11:49:09: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/45: Power granted
Shop(config-if)#
Dec 23 11:50:05: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to up
Dec 23 11:50:06: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to up
Shop(config-if)#
Dec 23 11:51:09: %TRACKING-5-STATE: 100 ip sla 100 reachability Down->Up

[Michael Durham]

I tried making two changes but still no luck.    What the heck am I missing??????

event manager applet reset_interface
event ipsla operation-id 100 reaction-type connectionLoss dest-ip-addr 4.2.2.2
action 1 track read 100
action 2 cli command "conf t"
action 3 cli command "interface gi1/0/45"
action 4 cli command "shut"
action 5 wait 10
action 6 cli command "no shut"
action 7 cli command "end"

[Dan Frey]

Use the event track action to trigger the eem policy. Default timeout of eem is 20 seconds and the sum of the wait time configured is 210 seconds.   You need to adjust the maxrun time to account for "wait" and may want to test IP connectivity before attempting to send email.

event manager applet ipsladown
 event track 100 state down maxrun 30
 action 010 cli command "enable"
 action 020 cli command "conf t"
 action 030 cli command " interface gig 1/0/45"
 action 040 cli command "shutdown"
 action 050 wait 10
 action 060 cli command "no shutdown"

[Michael Durham]

There is no event track 100 state down maxrun 30 in the applet command on the C3750v2 switch which is why this is kicking my butt!!!

Here are the only options:

Shop(config-applet)#?
Event Manager Applet Entry Configuration Commands:
action Add or modify an action statement
description Add or modify an applet description
event Add or modify event information
exit Exit from Event Manager applet configuration submode
help Description of the interactive help system
no Negate a command or set its defaults
trigger Enter applet trigger configuration submode

Shop(config-applet)#ac
Shop(config-applet)#action ?
WORD Label

Shop(config-applet)#action IPSAL_Down ?
add Add
append Append to a variable
break Break out of a conditional loop
cli Execute a CLI command
cns-event Send a CNS event
comment add comment
context Save or retrieve context information
continue Continue to next loop iteration
counter Modify a counter value
decrement Decrement a variable
divide Divide
else else conditional
elseif elseif conditional
end end conditional block
exit Exit from applet run
force-switchover Force a software switchover
foreach foreach loop
gets get line of input from active tty
handle-error On error action
help Read/Set parser help buffer
if if conditional
increment Increment a variable
info Obtain system specific information
mail Send an e-mail
multiply Multiply
policy Run a pre-registered policy
publish-event Publish an application specific event
puts print data to active tty
regexp regular expression match
reload Reload system
set Set a variable
snmp-object-value Specify value for the SNMP get request
snmp-trap Send an SNMP trap
string string commands
subtract Subtract
syslog Log a syslog message
track Read/Set a tracking object
wait Wait for a specified amount of time
while while loop

Shop(config-applet)#eve
Shop(config-applet)#event ?
application Application specific event
cli CLI event
config Configuration policy event
counter Counter event
env Environmental event
gold GOLD event
interface Interface event
ioswdsysmon IOS WDSysMon event
ipsla IPSLA Event
mat MAC address table event
neighbor-discovery Neighbor Discovery event
none Manually run policy event
oir OIR event
routing Routing event
rpc Remote Procedure Call event
snmp SNMP event
snmp-notification SNMP Notification Event
snmp-object SNMP object event
syslog Syslog event
tag event tag identifier
timer Timer event

Shop(config-applet)#tr
Shop(config-applet)#trigger ?
delay Time to delay raising event after all conditions met
occurs The number of occurrences before raising the event
period Number of occurrences must occur within this time period
period-start Cron entry for period start
<cr>

[rcarrara]

At the moment I can’t look closely at the config but off the top of my head I see you’re trying to go into configuration mode without first entering enable mode.

Issue “enable” action on the line preceding “conf t”.

You should use a different labeling scheme for the actions so you can easily insert addition lines if needed. Examples being something like 010, 020, 030, etc. this provides the option of 9 other lines between each action if needed. Or even 1.0, 2.0, etc.

[Michael Durham]

I have modified my config but I think its still not working.  I manually shut down port g1/0/45 to kill the internet.  I would think that this would cause the EEM to kick in.  But its NOT!!! UUUGGG!!!!!!

!
ip sla 100
icmp-echo 4.2.2.2
threshold 750
timeout 900
frequency 15
ip sla schedule 100 life forever start-time now
ip sla reaction-configuration 100 react timeout threshold-type immediate
ip sla enable reaction-alerts
!
event manager applet ipsladown
description When Internet is down, reset the T-Mobile Business router
action 010 cli command "enable"
action 020 cli command "conf t"
action 030 cli command " interface gig 1/0/45"
action 040 cli command "shutdown"
action 050 wait 10
action 060 cli command "no shutdown"
action 070 cli command "end"

Here is the screen results for several commands:

Shop(config)#int g1/0/45
Shop(config-if)#shut
Dec 23 21:43:06: %LINK-5-CHANGED: Interface GigabitEthernet1/0/45, changed state to administratively down
Dec 23 21:43:07: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to down
Shop(config-if)#
Dec 23 21:43:20: %TRACKING-5-STATE: 100 ip sla 100 reachability Up->Down
Shop(config-if)#end
Shop#
Shop#sh eve man his events det
No. Job Id Proc Status Time of Event Event Type Name

Shop#sh eve man his tr
No. Time Trap Type Name
Shop#
Shop#sh track 100
Track 100
IP SLA 100 reachability
Reachability is Down
115 changes, last change 00:01:30
Delay up 60 secs, down 10 secs
Latest operation return code: Timeout
Shop(config)#int g1/0/45
Shop(config-if)#no shut
Shop(config-if)#
Dec 23 21:45:40: %ILPOWER-7-DETECT: Interface Gi1/0/45: Power Device detected: IEEE PD
Shop(config-if)#
Dec 23 21:45:40: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to down
Shop(config-if)#end
Shop#
Dec 23 21:45:40: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/45: Power granted
Shop#
Dec 23 21:45:42: %SYS-5-CONFIG_I: Configured from console by mdurham on vty0 (192.168.69.101)
Shop#
Dec 23 21:46:38: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to up
Dec 23 21:46:39: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to up

Dec 23 21:47:41: %TRACKING-5-STATE: 100 ip sla 100 reachability Down->Up
Shop#sh track 100
Track 100
IP SLA 100 reachability
Reachability is Up
116 changes, last change 00:00:03
Delay up 60 secs, down 10 secs
Latest operation return code: OK
Latest RTT (millisecs) 91

!

[balaji.bandi]

Try below simple EEM Script and let us know how that goes :

ip sla 100
 icmp-echo 4.2.2.2
 threshold 500
 frequency 15
ip sla schedule 100 life forever start-time now
ip sla reaction-configuration 100 react timeout threshold-type immediate
ip sla enable reaction-alerts

event manager applet ISP_DOWN
 event ipsla operation-id 100 reaction-type timeout
 action 1.0 cli command "enable"
 action 2.0 cli command "conf t"
 action 3.0 cli command "interface gig 1/0/45"
 action 4.0 cli command "shut"
 action 5.0 wait 10
 action 6.0 cli command "no shutdown"
 action 7.0 cli command "end"
!

 Note : make sure you remove all the old configuration