EEM "bounce" port when cannot ping internet

[Michael Durham]

I have a C3750v2 switch running IOS 15.0(2)SE7 and every once in a while my internet router screws up requiring a reset.  I am testing internet connectivity and if not reply after 10 seconds, bounce the port.    The internet router is by T-Mobile and runs off the PoE port so all I need to bounce (shut - no shut) the port.  I have tried setting up an ip sla and an eem to make this happen when the switch cannot ping the internet, 4.2.2.2.  However, I am missing something because its not working.  Also, I would like to have an email sent once the port and router are back up.  It takes about 3 minutes for the T-Mobile router to fully reboot.  What am I missing?  What needs changed?  This is running on the C3750v2 switch, not a Cisco router.

!
track 100 ip sla 100 reachability
delay down 10 up 60
!
ip sla 100
icmp-echo 4.2.2.2
threshold 750
timeout 900
frequency 15
ip sla schedule 100 life forever start-time now
ip sla reaction-configuration 100 react timeout threshold-type immediate
ip sla enable reaction-alerts
!
event manager applet reset_interface
event ipsla operation-id 100 reaction-type timeout
action 1 if $_ipsla_condition eq "Occurred"
action 2 cli command "conf t"
action 3 cli command "interface gi1/0/45"
action 4 cli command "shut"
action 5 wait 10
action 6 cli command "no shut"
action 7 cli command "end"
action 5 wait 200
action 8 mail server "mail.privateemail.com"
action 9 mail from "<support@certificationtrainingsolutions.com>"
action 10 mail to "<mdurham@thedurhamcorporation.com>"
action 11 mail subject "Switch Port Reset Notification"
action 12 mail body "The T-Mobile router was reset."
action 13 mail server login "<support@certificationtrainingsolutions.com>" "<SMTP_PASSWORD>"
action 14 mail server protocol smtp ssl
!

[Michael Durham]

That is now working... too good.  The port keeps bouncing on/off because before the T-Mobile router can finish resetting, the ping fails and the port is shut down again.  This keeps going in a loop.  I need about 1 minute and 15 seconds for it to come back up.  Which timer do I change?

[balaji.bandi]

May be try below wait time with maxirum time and test it :

Try increasing the threshold value in ipsla to more.

event manager applet ISP_DOWN
 event ipsla operation-id 100 reaction-type timeout maxrun 200
 action 1.0 cli command "enable"
 action 2.0 cli command "conf t"
 action 3.0 cli command "interface gi1/0/45"
 action 4.0 cli command "shut"
 action 5.0 wait 10
 action 6.0 cli command "no shutdown"
 action 7.0 wait 120
 action 8.0 cli command "end"

if that still not working worth looking below delay time :

https://community.cisco.com/t5/networking-knowledge-base/how-to-introduce-large-delays-in-eem-policies/ta-p/3161702

[Michael Durham]

We are still having an issue with bouncing right after it comes back up.  I think something in the IP SLA area needs changed to tell it to test the pinging a little less often.  I don't do Cisco as much as I used to and it seems that I have forgotten a lot. ;(

Here is the output from the latest testing over a three minute period.  The port should only shutdown once in this test, not twice.

Shop(config)#ip sla enable reaction-alerts
Shop(config)#
Shop(config)#
Shop(config)#do sh run | s ip sla
track 100 ip sla 100 reachability
delay down 10 up 60
ip sla 100
icmp-echo 4.2.2.2
threshold 750
timeout 900
frequency 15
ip sla schedule 100 life forever start-time now
ip sla reaction-configuration 100 react timeout threshold-type immediate
ip sla enable reaction-alerts
Shop(config)#do sh run | s event
event manager applet ISP_DOWN
event ipsla operation-id 100 reaction-type timeout maxrun 200
action 1.0 cli command "enable"
action 2.0 cli command "conf t"
action 3.0 cli command "interface gi1/0/45"
action 4.0 cli command "shut"
action 5.0 wait 10
action 6.0 cli command "no shutdown"
action 7.0 wait 120
action 8.0 cli command "end"
Shop(config)#
Shop(config)#
Shop(config)#int g1/0/45
Shop(config-if)#shut
Shop(config-if)#
Dec 26 10:05:17: %LINK-5-CHANGED: Interface GigabitEthernet1/0/45, changed state to administratively down
Dec 26 10:05:18: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to down
Shop(config-if)#
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : CTL : cli_open called.
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : OUT : Shop#
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : IN : Shop#enable
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : OUT : Shop#
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : IN : Shop#conf t
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : OUT : Enter configuration commands, one per line. End with CNTL/Z.
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : OUT : Shop(config)#
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : IN : Shop(config)#interface gi1/0/45
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : OUT : Shop(config-if)#
Shop(config-if)#
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : IN : Shop(config-if)#shut
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : OUT : Shop(config-if)#
Shop(config-if)#
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : IN : Shop(config-if)#no shutdown
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : OUT : Shop(config-if)#
Shop(config-if)#
Dec 26 10:05:31: %TRACKING-5-STATE: 100 ip sla 100 reachability Up->Down
Shop(config-if)#
Dec 26 10:05:32: %ILPOWER-7-DETECT: Interface Gi1/0/45: Power Device detected: IEEE PD
Shop(config-if)#
Dec 26 10:05:33: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to down
Shop(config-if)#
Dec 26 10:05:33: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/45: Power granted
Shop(config-if)#
Dec 26 10:06:29: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to up
Dec 26 10:06:30: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to up
Shop(config-if)#
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : CTL : cli_open called.
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : OUT : Shop#
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : IN : Shop#enable
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : OUT : Shop#
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : IN : Shop#conf t
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : OUT : Enter configuration commands, one per line. End with CNTL/Z.
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : OUT : Shop(config)#
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : IN : Shop(config)#interface gi1/0/45
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : OUT : Shop(config-if)#
Shop(config-if)#
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : IN : Shop(config-if)#shut
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : OUT : Shop(config-if)#
Shop(config-if)#
Dec 26 10:06:37: %LINK-5-CHANGED: Interface GigabitEthernet1/0/45, changed state to administratively down
Dec 26 10:06:38: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to down
Shop(config-if)#
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : IN : Shop(config-if)#no shutdown
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : OUT : Shop(config-if)#
Shop(config-if)#
Dec 26 10:06:46: %ILPOWER-7-DETECT: Interface Gi1/0/45: Power Device detected: IEEE PD
Dec 26 10:06:47: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/45: Power granted
Shop(config-if)#
Dec 26 10:06:47: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to down
Shop(config-if)#
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : CTL : cli_open called.
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : OUT : Shop#
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : IN : Shop#enable
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : OUT : Shop#
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : IN : Shop#conf t
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : OUT : Enter configuration commands, one per line. End with CNTL/Z.
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : OUT : Shop(config)#
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : IN : Shop(config)#interface gi1/0/45
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : OUT : Shop(config-if)#
Shop(config-if)#
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : IN : Shop(config-if)#shut
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : OUT : Shop(config-if)#
Shop(config-if)#
Dec 26 10:06:52: %LINK-5-CHANGED: Interface GigabitEthernet1/0/45, changed state to administratively down
Shop(config-if)#
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : IN : Shop(config-if)#no shutdown
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : OUT : Shop(config-if)#
Shop(config-if)#
Dec 26 10:07:02: %ILPOWER-7-DETECT: Interface Gi1/0/45: Power Device detected: IEEE PD
Dec 26 10:07:02: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/45: Power granted
Shop(config-if)#
Dec 26 10:07:03: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to down
Shop(config-if)#
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : IN : Shop(config-if)#end
Shop(config-if)#
Dec 26 10:07:31: %SYS-5-CONFIG_I: Configured from console by on vty1 (EEM:ISP_DOWN)
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : OUT : Shop#
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : CTL : cli_close called.
7w6d: tty is now going through its death sequence
Shop(config-if)#
Dec 26 10:07:58: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to up
Dec 26 10:07:59: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to up
Shop(config-if)#
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : CTL : cli_open called.
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : OUT : Shop#
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : IN : Shop#enable
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : OUT : Shop#
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : IN : Shop#conf t
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : OUT : Enter configuration commands, one per line. End with CNTL/Z.
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : OUT : Shop(config)#
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : IN : Shop(config)#interface gi1/0/45
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : OUT : Shop(config-if)#
Shop(config-if)#
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : IN : Shop(config-if)#shut
7w6d: %HA_EM-6-LOG: ISP_DOWN : DEBUG(cli_lib) : : OUT : Shop(config-if)#
Shop(config-if)#
Dec 26 10:08:07: %LINK-5-CHANGED: Interface GigabitEthernet1/0/45, changed state to administratively down
Dec 26 10:08:08: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to down
Shop(config-if)#no ip sla enable reaction-alerts

[balaji.bandi]

You can try below :

Add Track :

track 100 ip sla 100 reachability

Remove existing EEM applet

no event manager applet ISP_DOWN

Add below scripts applet : (this will only kick in when the down to up) - so after adding this script shutdown the interface 45 and bring up, so this will add EEM script).

event manager environment q "
event manager applet ISP_UP
event track 100 state up
action 001 cli command "enable"
action 002 cli command "config t"
action 003 cli command "event manager applet ISP_DOWN"
action 004 cli command "event ipsla operation-id 100 reaction-type timeout"
action 005 cli command "action 1.0 cli command enable"
action 006 cli command "action 2.0 cli command $q config t$q"
action 007 cli command "action 3.0 cli command $q interface gig 1/0/45$q"
action 008 cli command "action 4.0 cli command $q shut$q"
action 009 cli command "action 5.0 wait 10"
action 010 cli command "action 6.0 cli command $q no shutdown$q"
action 011 cli command "action 7.0 cli command $q exit$q"
action 012 cli command "action 8.0 cli command $q no event manager applet ISP_DOWN$q"
action 013 cli command "action 9.0 cli command end"

As soon as the Track comeup above applet run add new EEM script, and when the sla down, it will run once and remove Applet - so it will not run again.

test and feedback.

[Michael Durham]

I really like your idea but one big problem, there is NO event track 100 state up option on the C3750v2 IOS 15.0 switch.  Maybe we could use a little of my coding with yours to achieve the goal.

[Michael Durham]

also, what does the  $q  mean?

[balaji.bandi · Accepted solution]

what does the  $q  mean?  - The environment variable "q"

C3750v2 IOS 15.0  - this was missed my thinking time, we always use most of the the router, so i more focused on track ( i am sure you mentioned before, apologies) - may be try based on syslog  (you can do other methods, but i see easiest way) - test and let me know.

ip sla 100
 icmp-echo 10.10.10.1
 threshold 500
 frequency 15
ip sla schedule 100 life forever start-time now
ip sla reaction-configuration 100 react timeout threshold-type immediate action-type trapOnly
ip sla logging traps
ip sla enable reaction-alerts
logging trap debugging
!
!
snmp-server enable traps ipsla
!
event manager environment q "
event manager applet ISP_UP
 event syslog pattern "%RTT-4-OPER_TIMEOUT: condition cleared, entry number = 100"
 action 001 cli command "enable"
 action 002 cli command "config t"
 action 003 cli command "event manager applet ISP_DOWN"
 action 004 cli command "event ipsla operation-id 100 reaction-type timeout"
 action 005 cli command "action 1.0 cli command enable"
 action 006 cli command "action 2.0 cli command $q config t$q"
 action 007 cli command "action 3.0 cli command $q interface gig 1/0/45$q"
 action 008 cli command "action 4.0 cli command $q shut$q"
 action 009 cli command "action 5.0 wait 10"
 action 010 cli command "action 6.0 cli command $q no shutdown$q"
 action 011 cli command "action 7.0 cli command $q exit$q"
 action 012 cli command "action 8.0 cli command $q no event manager applet ISP_DOWN$q"
 action 013 cli command "action 9.0 cli command end"
!
end

[Michael Durham]

IOS Version c3750-ipservicesk9-mz.150-2.SE7.bin

***Here is what I entered****

ip sla 100
icmp-echo 4.2.2.2
threshold 500
frequency 15
ip sla schedule 100 life forever start-time now
ip sla reaction-configuration 100 react timeout threshold-type immediate action-type trapOnly
ip sla logging traps
ip sla enable reaction-alerts
event manager environment q "
event manager applet ISP_UP
event syslog pattern "%RTT-4-OPER_TIMEOUT: condition cleared, entry number = 100"
action 001 cli command "enable"
action 002 cli command "config t"
action 003 cli command "event manager applet ISP_DOWN"
action 004 cli command "event ipsla operation-id 100 reaction-type timeout"
action 005 cli command "action 1.0 cli command enable"
action 006 cli command "action 2.0 cli command $q config t$q"
action 007 cli command "action 3.0 cli command $q interface gig 1/0/45$q"
action 008 cli command "action 4.0 cli command $q shut$q"
action 009 cli command "action 5.0 wait 10"
action 010 cli command "action 6.0 cli command $q no shutdown$q"
action 011 cli command "action 7.0 cli command $q exit$q"
action 012 cli command "action 8.0 cli command $q no event manager applet ISP_DOWN$q"
action 013 cli command "action 9.0 cli command end"

****MANUALLY SHUT PORT DOWN FOR TESTING*******

Shop(config)#int g1/0/045
Shop(config-if)#shut
Shop(config-if)#
Dec 30 15:40:24: %LINK-5-CHANGED: Interface GigabitEthernet1/0/45, changed state to administratively down
Dec 30 15:40:25: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to down
Shop(config-if)#
Dec 30 15:40:33: %RTT-4-OPER_TIMEOUT: condition occurred, entry number = 100
Shop(config-if)#
Dec 30 15:40:33: %RTT-3-IPSLATHRESHOLD: IP SLAs(100): Threshold Occurred for timeout

****Waited 2 minutes and nothing happened so I turned the port back on and the following happened / Something should start after 15 seconds of being down****

Shop(config-if)#no shut
Shop(config-if)#
Dec 30 15:42:06: %ILPOWER-7-DETECT: Interface Gi1/0/45: Power Device detected: IEEE PD
Dec 30 15:42:07: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/45: Power granted
Shop(config-if)#
Dec 30 15:42:07: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to down
Dec 30 15:43:03: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to up
Dec 30 15:43:04: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to up
Dec 30 15:43:13: %RTT-4-OPER_TIMEOUT: condition cleared, entry number = 100
Dec 30 15:43:13: %RTT-3-IPSLATHRESHOLD: IP SLAs(100): Threshold Cleared for timeout
8w3d: %HA_EM-6-LOG: ISP_UP : DEBUG(cli_lib) : : CTL : cli_open called.
8w3d: %HA_EM-6-LOG: ISP_UP : DEBUG(cli_lib) : : OUT : Shop#
8w3d: %HA_EM-6-LOG: ISP_UP : DEBUG(cli_lib) : : IN : Shop#enable
8w3d: %HA_EM-6-LOG: ISP_UP : DEBUG(cli_lib) : : OUT : Shop#
8w3d: %HA_EM-6-LOG: ISP_UP : DEBUG(cli_lib) : : IN : Shop#config t
8w3d: %HA_EM-6-LOG: ISP_UP : DEBUG(cli_lib) : : OUT : Enter configuration commands, one per line. End with CNTL/Z.
8w3d: %HA_EM-6-LOG: ISP_UP : DEBUG(cli_lib) : : OUT : Shop(config)#
8w3d: %HA_EM-6-LOG: ISP_UP : DEBUG(cli_lib) : : IN : Shop(config)#event manager applet ISP_DOWN
8w3d: %HA_EM-6-LOG: ISP_UP : DEBUG(cli_lib) : : OUT : Shop(config-applet)#
8w3d: %HA_EM-6-LOG: ISP_UP : DEBUG(cli_lib) : : IN : Shop(config-applet)#event ipsla operation-id 100 reaction-type timeout
8w3d: %HA_EM-6-LOG: ISP_UP : DEBUG(cli_lib) : : OUT : Shop(config-applet)#
8w3d: %HA_EM-6-LOG: ISP_UP : DEBUG(cli_lib) : : IN : Shop(config-applet)#action 1.0 cli command enable
8w3d: %HA_EM-6-LOG: ISP_UP : DEBUG(cli_lib) : : OUT : Shop(config-applet)#
8w3d: %HA_EM-6-LOG: ISP_UP : DEBUG(cli_lib) : : IN : Shop(config-applet)#action 2.0 cli command " config t"
8w3d: %HA_EM-6-LOG: ISP_UP : DEBUG(cli_lib) : : OUT : Shop(config-applet)#
8w3d: %HA_EM-6-LOG: ISP_UP : DEBUG(cli_lib) : : IN : Shop(config-applet)#action 3.0 cli command " interface gig 1/0/45"
8w3d: %HA_EM-6-LOG: ISP_UP : DEBUG(cli_lib) : : OUT : Shop(config-applet)#
8w3d: %HA_EM-6-LOG: ISP_UP : DEBUG(cli_lib) : : IN : Shop(config-applet)#action 4.0 cli command " shut"
8w3d: %HA_EM-6-LOG: ISP_UP : DEBUG(cli_lib) : : OUT : Shop(config-applet)#
8w3d: %HA_EM-6-LOG: ISP_UP : DEBUG(cli_lib) : : IN : Shop(config-applet)#action 5.0 wait 10
8w3d: %HA_EM-6-LOG: ISP_UP : DEBUG(cli_lib) : : OUT : Shop(config-applet)#
8w3d: %HA_EM-6-LOG: ISP_UP : DEBUG(cli_lib) : : IN : Shop(config-applet)#action 6.0 cli command " no shutdown"
8w3d: %HA_EM-6-LOG: ISP_UP : DEBUG(cli_lib) : : OUT : Shop(config-applet)#
8w3d: %HA_EM-6-LOG: ISP_UP : DEBUG(cli_lib) : : IN : Shop(config-applet)#action 7.0 cli command " exit"
8w3d: %HA_EM-6-LOG: ISP_UP : DEBUG(cli_lib) : : OUT : Shop(config-applet)#
8w3d: %HA_EM-6-LOG: ISP_UP : DEBUG(cli_lib) : : IN : Shop(config-applet)#action 8.0 cli command " no event manager applet ISP_DOWN"
8w3d: %HA_EM-6-LOG: ISP_UP : DEBUG(cli_lib) : : OUT : Shop(config-applet)#
8w3d: %HA_EM-6-LOG: ISP_UP : DEBUG(cli_lib) : : IN : Shop(config-applet)#action 9.0 cli command end
8w3d: %HA_EM-6-LOG: ISP_UP : DEBUG(cli_lib) : : OUT : Shop(config-applet)#
Shop(config-if)#
8w3d: %HA_EM-6-LOG: ISP_UP : DEBUG(cli_lib) : : CTL : cli_close called.
8w3d: tty is now going through its death sequence

[balaji.bandi]

The condition of the script is - make sure first time the Trap appeared in the Logs - in your case when you no shut - that triggered :

%RTT-4-OPER_TIMEOUT: condition cleared, entry number = 100

this time you see applet created - event manager applet ISP_DOWN  ( first time it is not going be there ?)

if you see show run you have applet now - event manager applet ISP_DOWN (with content)

now you can try shutdown the port and test it.

then  the event will kick in - event manager applet ISP_DOWN and shutdown and no shutdown and remove applet  - no event manager applet ISP_DOWN - then once IP sla up and able to ping 4.2.2.2 - then it will add - event manager applet ISP_DOWN

[Michael Durham]

Yes, its working now, working too good.  Which line should I change to have it wait for 30 seconds of downtime before running the script?  It seems T-Mobile is dropping out for a few seconds then back up.

Here is all that has happened since I posted last time:

Dec 30 16:13:03: %RTT-4-OPER_TIMEOUT: condition occurred, entry number = 100
Dec 30 16:13:03: %RTT-3-IPSLATHRESHOLD: IP SLAs(100): Threshold Occurred for timeout
Dec 30 16:13:05: %LINK-5-CHANGED: Interface GigabitEthernet1/0/45, changed state to administratively down
Dec 30 16:13:06: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to down
Dec 30 16:13:14: %ILPOWER-7-DETECT: Interface Gi1/0/45: Power Device detected: IEEE PD
Dec 30 16:13:14: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/45: Power granted
Dec 30 16:13:15: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to down
Dec 30 16:14:11: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to up
Dec 30 16:14:12: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to up
Dec 30 16:14:14: %RTT-4-OPER_TIMEOUT: condition cleared, entry number = 100
Dec 30 16:14:14: %RTT-3-IPSLATHRESHOLD: IP SLAs(100): Threshold Cleared for timeout
Dec 30 17:36:18: %RTT-4-OPER_TIMEOUT: condition occurred, entry number = 100
Dec 30 17:36:18: %RTT-3-IPSLATHRESHOLD: IP SLAs(100): Threshold Occurred for timeout
Dec 30 17:36:20: %LINK-5-CHANGED: Interface GigabitEthernet1/0/45, changed state to administratively down
Dec 30 17:36:21: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to down
Dec 30 17:36:29: %ILPOWER-7-DETECT: Interface Gi1/0/45: Power Device detected: IEEE PD
Dec 30 17:36:30: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/45: Power granted
Dec 30 17:36:30: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to down
Dec 30 17:37:26: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to up
Dec 30 17:37:27: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to up
Dec 30 17:37:28: %RTT-4-OPER_TIMEOUT: condition cleared, entry number = 100
Dec 30 17:37:28: %RTT-3-IPSLATHRESHOLD: IP SLAs(100): Threshold Cleared for timeout
Dec 30 17:38:18: %RTT-4-OPER_TIMEOUT: condition occurred, entry number = 100
Dec 30 17:38:18: %RTT-3-IPSLATHRESHOLD: IP SLAs(100): Threshold Occurred for timeout
Dec 30 17:38:20: %LINK-5-CHANGED: Interface GigabitEthernet1/0/45, changed state to administratively down
Dec 30 17:38:21: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to down
Dec 30 17:38:29: %ILPOWER-7-DETECT: Interface Gi1/0/45: Power Device detected: IEEE PD
Dec 30 17:38:29: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/45: Power granted
Dec 30 17:38:30: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to down
Dec 30 17:39:26: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to up
Dec 30 17:39:27: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to up
Dec 30 17:39:48: %RTT-4-OPER_TIMEOUT: condition occurred, entry number = 100
Dec 30 17:39:48: %RTT-3-IPSLATHRESHOLD: IP SLAs(100): Threshold Occurred for timeout
Dec 30 17:39:50: %LINK-5-CHANGED: Interface GigabitEthernet1/0/45, changed state to administratively down
Dec 30 17:39:51: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to down
Dec 30 17:40:00: %ILPOWER-7-DETECT: Interface Gi1/0/45: Power Device detected: IEEE PD
Dec 30 17:40:00: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to down
Dec 30 17:40:01: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/45: Power granted
Dec 30 17:40:56: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to up
Dec 30 17:40:57: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to up
Dec 30 17:45:33: %RTT-4-OPER_TIMEOUT: condition occurred, entry number = 100
Dec 30 17:45:33: %RTT-3-IPSLATHRESHOLD: IP SLAs(100): Threshold Occurred for timeout
Dec 30 17:45:35: %LINK-5-CHANGED: Interface GigabitEthernet1/0/45, changed state to administratively down
Dec 30 17:45:36: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to down
Dec 30 17:45:45: %ILPOWER-7-DETECT: Interface Gi1/0/45: Power Device detected: IEEE PD
Dec 30 17:45:45: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/45: Power granted
Dec 30 17:45:45: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to down
Dec 30 17:46:41: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to up
Dec 30 17:46:42: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to up
Dec 30 17:46:44: %RTT-4-OPER_TIMEOUT: condition cleared, entry number = 100
Dec 30 17:46:44: %RTT-3-IPSLATHRESHOLD: IP SLAs(100): Threshold Cleared for timeout
Dec 30 17:48:03: %RTT-4-OPER_TIMEOUT: condition occurred, entry number = 100
Dec 30 17:48:03: %RTT-3-IPSLATHRESHOLD: IP SLAs(100): Threshold Occurred for timeout
Dec 30 17:48:05: %LINK-5-CHANGED: Interface GigabitEthernet1/0/45, changed state to administratively down
Dec 30 17:48:06: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to down
Dec 30 17:48:15: %ILPOWER-7-DETECT: Interface Gi1/0/45: Power Device detected: IEEE PD
Dec 30 17:48:15: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to down
Dec 30 17:48:16: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/45: Power granted
Dec 30 17:49:12: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to up
Dec 30 17:49:13: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to up
Dec 30 18:13:33: %RTT-4-OPER_TIMEOUT: condition occurred, entry number = 100
Dec 30 18:13:33: %RTT-3-IPSLATHRESHOLD: IP SLAs(100): Threshold Occurred for timeout
Dec 30 18:13:35: %LINK-5-CHANGED: Interface GigabitEthernet1/0/45, changed state to administratively down
Dec 30 18:13:36: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to down
Dec 30 18:13:44: %ILPOWER-7-DETECT: Interface Gi1/0/45: Power Device detected: IEEE PD
Dec 30 18:13:45: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/45: Power granted
Dec 30 18:13:45: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to down
Dec 30 18:14:41: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to up
Dec 30 18:14:42: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to up
Dec 30 18:14:44: %RTT-4-OPER_TIMEOUT: condition cleared, entry number = 100
Dec 30 18:14:44: %RTT-3-IPSLATHRESHOLD: IP SLAs(100): Threshold Cleared for timeout
Dec 30 18:15:48: %RTT-4-OPER_TIMEOUT: condition occurred, entry number = 100
Dec 30 18:15:48: %RTT-3-IPSLATHRESHOLD: IP SLAs(100): Threshold Occurred for timeout
Dec 30 18:15:50: %LINK-5-CHANGED: Interface GigabitEthernet1/0/45, changed state to administratively down
Dec 30 18:15:51: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to down
Dec 30 18:16:00: %ILPOWER-7-DETECT: Interface Gi1/0/45: Power Device detected: IEEE PD
Dec 30 18:16:00: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to down
Dec 30 18:16:01: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/45: Power granted
Dec 30 18:16:56: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to up
Dec 30 18:16:57: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to up
Dec 30 18:16:59: %RTT-4-OPER_TIMEOUT: condition cleared, entry number = 100
Dec 30 18:16:59: %RTT-3-IPSLATHRESHOLD: IP SLAs(100): Threshold Cleared for timeout
Dec 30 18:17:48: %RTT-4-OPER_TIMEOUT: condition occurred, entry number = 100
Dec 30 18:17:48: %RTT-3-IPSLATHRESHOLD: IP SLAs(100): Threshold Occurred for timeout
Dec 30 18:17:50: %LINK-5-CHANGED: Interface GigabitEthernet1/0/45, changed state to administratively down
Dec 30 18:17:51: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to down
Dec 30 18:17:59: %ILPOWER-7-DETECT: Interface Gi1/0/45: Power Device detected: IEEE PD
Dec 30 18:18:00: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/45: Power granted
Dec 30 18:18:00: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to down
Dec 30 18:18:56: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to up
Dec 30 18:18:57: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to up
Dec 30 18:18:59: %RTT-4-OPER_TIMEOUT: condition cleared, entry number = 100
Dec 30 18:18:59: %RTT-3-IPSLATHRESHOLD: IP SLAs(100): Threshold Cleared for timeout
Dec 30 18:21:33: %RTT-4-OPER_TIMEOUT: condition occurred, entry number = 100
Dec 30 18:21:33: %RTT-3-IPSLATHRESHOLD: IP SLAs(100): Threshold Occurred for timeout
Dec 30 18:21:35: %LINK-5-CHANGED: Interface GigabitEthernet1/0/45, changed state to administratively down
Dec 30 18:21:36: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to down
Dec 30 18:21:44: %ILPOWER-7-DETECT: Interface Gi1/0/45: Power Device detected: IEEE PD
Dec 30 18:21:45: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/45: Power granted
Dec 30 18:21:45: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to down
Dec 30 18:22:42: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to up
Dec 30 18:22:43: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to up
Dec 30 18:23:03: %RTT-4-OPER_TIMEOUT: condition occurred, entry number = 100
Dec 30 18:23:03: %RTT-3-IPSLATHRESHOLD: IP SLAs(100): Threshold Occurred for timeout
Dec 30 18:23:05: %LINK-5-CHANGED: Interface GigabitEthernet1/0/45, changed state to administratively down
Dec 30 18:23:06: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to down
Dec 30 18:23:15: %ILPOWER-7-DETECT: Interface Gi1/0/45: Power Device detected: IEEE PD
Dec 30 18:23:15: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to down
Dec 30 18:23:16: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/45: Power granted
Dec 30 18:24:11: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to up
Dec 30 18:24:12: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to up
Dec 30 18:24:14: %RTT-4-OPER_TIMEOUT: condition cleared, entry number = 100
Dec 30 18:24:14: %RTT-3-IPSLATHRESHOLD: IP SLAs(100): Threshold Cleared for timeout
Dec 30 18:26:18: %RTT-4-OPER_TIMEOUT: condition occurred, entry number = 100
Dec 30 18:26:18: %RTT-3-IPSLATHRESHOLD: IP SLAs(100): Threshold Occurred for timeout
Dec 30 18:26:20: %LINK-5-CHANGED: Interface GigabitEthernet1/0/45, changed state to administratively down
Dec 30 18:26:21: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to down
Dec 30 18:26:30: %ILPOWER-7-DETECT: Interface Gi1/0/45: Power Device detected: IEEE PD
Dec 30 18:26:30: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to down
Dec 30 18:26:31: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/45: Power granted
Dec 30 18:27:26: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to up
Dec 30 18:27:27: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to up
Dec 30 18:34:48: %RTT-4-OPER_TIMEOUT: condition occurred, entry number = 100
Dec 30 18:34:48: %RTT-3-IPSLATHRESHOLD: IP SLAs(100): Threshold Occurred for timeout
Dec 30 18:34:50: %LINK-5-CHANGED: Interface GigabitEthernet1/0/45, changed state to administratively down
Dec 30 18:34:51: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to down
Dec 30 18:34:59: %ILPOWER-7-DETECT: Interface Gi1/0/45: Power Device detected: IEEE PD
Dec 30 18:35:00: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/45: Power granted
Dec 30 18:35:00: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to down
Dec 30 18:35:56: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to up
Dec 30 18:35:57: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to up
Dec 30 18:35:59: %RTT-4-OPER_TIMEOUT: condition cleared, entry number = 100
Dec 30 18:35:59: %RTT-3-IPSLATHRESHOLD: IP SLAs(100): Threshold Cleared for timeout
Dec 30 18:36:18: %RTT-4-OPER_TIMEOUT: condition occurred, entry number = 100
Dec 30 18:36:18: %RTT-3-IPSLATHRESHOLD: IP SLAs(100): Threshold Occurred for timeout
Dec 30 18:36:30: %ILPOWER-7-DETECT: Interface Gi1/0/45: Power Device detected: IEEE PD
Dec 30 18:36:30: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to down
Dec 30 18:36:31: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/45: Power granted
Dec 30 18:37:26: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to up
Dec 30 18:37:27: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to up
Dec 30 18:37:28: %RTT-4-OPER_TIMEOUT: condition cleared, entry number = 100
Dec 30 18:37:28: %RTT-3-IPSLATHRESHOLD: IP SLAs(100): Threshold Cleared for timeout
Dec 30 18:37:39: %SYS-5-CONFIG_I: Configured from console by mdurham on vty1 (192.168.69.252)
Dec 30 18:40:18: %RTT-4-OPER_TIMEOUT: condition occurred, entry number = 100
Dec 30 18:40:18: %RTT-3-IPSLATHRESHOLD: IP SLAs(100): Threshold Occurred for timeout
Dec 30 18:40:20: %LINK-5-CHANGED: Interface GigabitEthernet1/0/45, changed state to administratively down
Dec 30 18:40:21: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to down
Dec 30 18:40:30: %ILPOWER-7-DETECT: Interface Gi1/0/45: Power Device detected: IEEE PD
Dec 30 18:40:30: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to down
Dec 30 18:40:31: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/45: Power granted
Dec 30 18:41:27: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to up
Dec 30 18:41:28: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to up
Dec 30 18:41:29: %RTT-4-OPER_TIMEOUT: condition cleared, entry number = 100
Dec 30 18:41:29: %RTT-3-IPSLATHRESHOLD: IP SLAs(100): Threshold Cleared for timeout
Shop(config-if)#exit
Shop(config)# no ip sla schedule 100 life forever start-time now

[Michael Durham]

Once I get that final adjustment, is there a way to log each time T-Mobile drops off and them back on so that I can send them this info and get to the bottom of the issue.  Need the log just to show down time and them time back up.  It could even be an email that I then put into a spreadsheet.

Thanks so much for your help.

Happy New Year!!!

[balaji.bandi]

Once I get that final adjustment, is there a way to log each time T-Mobile drops off and them back on so that I can send them this info and get to the bottom of the issue.  Need the log just to show down time and them time back up.  It could even be an email that I then put into a spreadsheet.

You can setup a syslog server locally to send all the logs to syslog server.

if you using email if the ISP down, what path it used to send emails ?

Yes, its working now, working too good.  Which line should I change to have it wait for 30 seconds of downtime before running the script?  It seems T-Mobile is dropping out for a few seconds then back up.

can you post the latest config (show run to look again to suggest)

You may need to look the timer increase  ip sla frequency and threshold and check.

looking at the logs, the provider connection is not stable looks for me ?

[Michael Durham]

I'm just not grasping the difference between Threshold, Timeout, and Frequency to figure out how to have the switch wait 30 seconds before doing anything and then not doing anything for 2 minutes as the T-Mobile router comes back up.  I either end up in a loop or nothing seems to happen within 5 minutes.

Also, when I reboot the switch, do I need to do a shut/no shut on port g1/0/45 to get this started again?  Of just once and we are good to go?

[balaji.bandi]

You have many scenarios that need to be test if and but kind of situation based on the outcome take actions.

but the thing here is you have limited option on 3750, you can try TCL scripts 

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750/software/release/12-2_52_se/configuration/guide/3750scg/sweem.pdf

 do I need to do a shut/no shut on port g1/0/45 to get this started again? 

The one i have suggested can not run automatically until it meet the requirements as mentioned on other post - that need to meet

But you can run different EEM Script and remove them after reload if that is required, below guide help you after reload EEM script to run.

https://community.cisco.com/t5/networking-knowledge-base/how-to-run-an-eem-policy-every-time-a-device-boots/tac-p/4468389#M6857

[MHM Cisco World]

I test it many cases, the event IP SLA shut the interface that use as source 
so it chicken or egg.
you can use EEM to detect internet is failed and send syslog 
MHM