Hello All,
I hope you are doing fine, I have an issue with my event manager created on an operational Cisco router
I raised a CISCO TAC and the assigned engineer advised me to check here in the EEM community issues and raise my query so i came to you.
The issue : I have applet manager with event track and action reset crypto and interfaces, the APPLET work as expected some time and give a status successful and stuck some time at shut interfaces without unshutiing them back and give us abort status applet and that caused internet isolation to our customer because the interfaces that were shut was our GW to the internet
Below is the Chassis/IOS type and version with event manager config and status of applets
Chassis: ISR4331/K9
IOS: isr4300-universalk9.03.16.04c.S.155-3.S4c-ext.SPA.bin"
+++++++++++++event manager config is below
Event manager applet1 is when track 101 is down we need to clear crypto then reset tunnels 1 and 2 then add a sys log "MONITOR-TRACK101-DOWN"
Evenet manager applet 2 is when track 102 is down we need to shut tunnel 2 then add a sys log "MONITOR-TRACK102-DOWN"
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
event manager session cli username "mshaltou"
event manager applet MONITOR-TRACK101-DOWN
event track 101 state down
action 10 cli command "enable"
action 11 cli command "conf t"
action 12 cli command "clear crypto iskamp"
action 20 cli command "int tunnel 1"
action 21 cli command "shut"
action 22 wait 2
action 23 cli command "no shut"
action 30 cli command "int tunnel 2"
action 31 cli command "shut"
action 32 wait 2
action 33 cli command "no shut"
action 40 syslog msg "MONITOR-TRACK101-DOWN"
event manager applet MONITOR-TRACK102-DOWN
event track 102 state down
action 10 cli command "enable"
action 11 cli command "conf t"
action 30 cli command "int tunnel 2"
action 31 cli command "shut"
action 32 wait 2
action 33 cli command "no shut"
action 40 syslog msg "MONITOR-TRACK102-DOWN"
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+++++++++++Show event manager
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
pmil3736# sh event manager policy registered
No. Class Type Event Type Trap Time Registered Name
1 applet user track Off Wed May 31 14:25:16 2017 MONITOR-TRACK101-DOWN
track 101 state down
maxrun 20.000
action 10 cli command "enable"
action 11 cli command "conf t"
action 12 cli command "clear crypto iskamp"
action 20 cli command "int tunnel 1"
action 21 cli command "shut"
action 22 wait 2
action 23 cli command "no shut"
action 30 cli command "int tunnel 2"
action 31 cli command "shut"
action 32 wait 2
action 33 cli command "no shut"
action 40 syslog msg "MONITOR-TRACK101-DOWN"
2 applet user track Off Wed May 31 14:25:27 2017 MONITOR-TRACK102-DOWN
track 102 state down
maxrun 20.000
action 10 cli command "enable"
action 11 cli command "conf t"
action 30 cli command "int tunnel 2"
action 31 cli command "shut"
action 32 wait 2
action 33 cli command "no shut"
action 40 syslog msg "MONITOR-TRACK102-DOWN"
pmil3736#sh event manager version
Embedded Event Manager Version 4.00
Component Versions:
eem: (rel9)1.2.12
eem-gold: (rel1)1.0.2
eem-call-home: (rel2)1.0.5
Event Detectors:
Name Version Node Type
application 01.00 node0/0 RP
rf 01.00 node0/0 RP
syslog 01.00 node0/0 RP
neighbor-discovery 01.00 node0/0 RP
generic 01.00 node0/0 RP
track 01.00 node0/0 RP
routing 02.00 node0/0 RP
resource 01.00 node0/0 RP
cli 01.00 node0/0 RP
counter 01.00 node0/0 RP
interface 01.00 node0/0 RP
ioswdsysmon 01.00 node0/0 RP
none 01.00 node0/0 RP
oir 01.00 node0/0 RP
snmp 01.00 node0/0 RP
snmp-object 01.00 node0/0 RP
snmp-notification 01.00 node0/0 RP
timer 01.00 node0/0 RP
test 01.00 node0/0 RP
config 01.00 node0/0 RP
env 01.00 node0/0 RP
ds 01.00 node0/0 RP
crash 01.00 node0/0 RP
gold 01.00 node0/0 RP
rpc 01.00 node0/0 RP
pmil3736#
pmil3736#sh event MANAger history events detailed
No. Job Id Proc Status Time of Event Event Type Name
1 379 Actv success Sun Aug27 01:13:13 2017 track applet: MONITOR-TRACK102-DOWN
2 380 Actv success Sun Aug27 01:31:14 2017 track applet: MONITOR-TRACK102-DOWN
3 381 Actv success Thu Aug31 12:49:13 2017 track applet: MONITOR-TRACK102-DOWN
4 382 Actv success Fri Sep 1 01:41:10 2017 track applet: MONITOR-TRACK102-DOWN
5 383 Actv success Fri Sep 1 02:59:12 2017 track applet: MONITOR-TRACK102-DOWN
6 384 Actv abort Fri Sep 1 03:15:12 2017 track applet: MONITOR-TRACK102-DOWN >>>>>>>>>>>>> aborted
7 385 Actv success Fri Sep 1 09:59:11 2017 track applet: MONITOR-TRACK102-DOWN
8 386 Actv abort Fri Sep 1 10:02:11 2017 track applet: MONITOR-TRACK102-DOWN >>>>>>>>>>>>>> aborted
9 387 Actv success Mon Sep 4 10:12:13 2017 track applet: MONITOR-TRACK102-DOWN
10 388 Actv abort Mon Sep 4 12:10:11 2017 track applet: MONITOR-TRACK102-DOWN >>>>>>>>> aborted
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+++++++++++++++++++++My SYS Log output showing Behavior of the EVENT manager and displaying when interface was shut and kept in the shut status is below
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
pmil3736.controlnet.equant.net|2017-08-20 17:52:01|TRACK|6|STATE|201 list boolean and Down -> Up|
pmil3736.controlnet.equant.net|2017-08-20 17:52:01|TRACK|6|STATE|101 ip sla 101 reachability Down -> Up|
pmil3736.controlnet.equant.net|2017-08-20 17:51:20|LINEPROTO|5|UPDOWN|Line protocol on Interface Tunnel2, changed state to up| >>>>>>>>>>>>>>>> action taken by event manager on tu2 to unshut
pmil3736.controlnet.equant.net|2017-08-20 17:51:20|LINK|5|CHANGED|Interface Tunnel2, changed state to administratively down| >>>>>>>>>>>>>>>>>>>>>> action taken by event manager on tu2
pmil3736.controlnet.equant.net|2017-08-20 17:51:20|LINK|5|CHANGED|Interface Tunnel1, changed state to administratively down| >>>>>>>>>>>>>>>>>>>>>> action taken by event manager on tu1 where Tunnel 1 stuck at admin down !!!!!!!!!!!!!!!********************
pmil3736.controlnet.equant.net|2017-08-20 17:51:14|HA_EM|6|LOG|MONITOR-TRACK101-DOWN: MONITOR-TRACK101-DOWN| ++++++ SYSLOG Added by event manager
pmil3736.controlnet.equant.net|2017-08-20 17:51:13|LINK|3|UPDOWN|Interface Tunnel1, changed state to up|
pmil3736.controlnet.equant.net|2017-08-20 17:51:06|TRACK|6|STATE|101 ip sla 101 reachability Up -> Down|
pmil3736.controlnet.equant.net|2017-08-20 17:48:06|TRACK|6|STATE|202 list boolean and Down -> Up|
pmil3736.controlnet.equant.net|2017-08-20 17:48:06|TRACK|6|STATE|102 ip sla 102 reachability Down -> Up|
pmil3736.controlnet.equant.net|2017-08-20 17:48:01|TRACK|6|STATE|201 list boolean and Down -> Up|
pmil3736.controlnet.equant.net|2017-08-20 17:48:01|TRACK|6|STATE|101 ip sla 101 reachability Down -> Up|
pmil3736.controlnet.equant.net|2017-08-20 17:47:25|LINEPROTO|5|UPDOWN|Line protocol on Interface Tunnel2, changed state to up|>>>>>>>>>>>>>>>> action taken by event manager on tu1 to unshut
pmil3736.controlnet.equant.net|2017-08-20 17:47:21|LINEPROTO|5|UPDOWN|Line protocol on Interface Tunnel1, changed state to up|>>>>>>>>>>>>>>>> action taken by event manager on tu2 to unshut
pmil3736.controlnet.equant.net|2017-08-20 17:47:17|LINK|3|UPDOWN|Interface Tunnel2, changed state to up|>>>>>>>>>>>>>>>> action taken by event manager on tu2 to unshut
pmil3736.controlnet.equant.net|2017-08-20 17:47:16|HA_EM|6|LOG|MONITOR-TRACK102-DOWN: MONITOR-TRACK102-DOWN| ++++++ SYSLOG Added by event manager
pmil3736.controlnet.equant.net|2017-08-20 17:47:15|HA_EM|6|LOG|MONITOR-TRACK101-DOWN: MONITOR-TRACK101-DOWN| ++++++ SYSLOG Added by event manager
pmil3736.controlnet.equant.net|2017-08-20 17:47:14|LINK|5|CHANGED|Interface Tunnel2, changed state to administratively down| >>>>>>>>>>>>>>>> action taken by event manager on tu2 to shut
pmil3736.controlnet.equant.net|2017-08-20 17:47:13|LINK|3|UPDOWN|Interface Tunnel1, changed state to up|>>>>>>>>>>>>>>>> action taken by event manager on tu1 to unshut
pmil3736.controlnet.equant.net|2017-08-20 17:47:10|LINK|5|CHANGED|Interface Tunnel1, changed state to administratively down|>>>>>>>>>>>>>>>> action taken by event manager on tu1 to shut
pmil3736.controlnet.equant.net|2017-08-20 17:47:10|TRACK|6|STATE|102 ip sla 102 reachability Up -> Down|
pmil3736.controlnet.equant.net|2017-08-20 17:47:06|TRACK|6|STATE|101 ip sla 101 reachability Up -> Down|
pmil3736.controlnet.equant.net|2017-08-20 17:47:01|IKEV2|5|SA_UP|SA UP|
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Thanks a lot and appreciate to hear back from you guys the suggestions and if we need to adjust the config what would be the best practice or if we have to increase the maxrun time or the wait time between each script's line what should it be like ?
Mai Shaltout
