09-13-2017 09:19 AM - edited 03-01-2019 06:07 PM
Hello All,
I hope you are doing fine, I am VPO and I have an issue with my event manager created on the router
I raised a CISCO TAC and the assigned engineer advised me to check here in the EEM community issues and raise my query so i came to you.
The issue : I have applet manager with event track and action reset crypto and interfaces, the APPLET work as expected some time and give a status successful and stuck some time at shut interfaces without unshutiing them back and give us abort status applet and that caused internet isolation to our customer because the interfaces that were shut was our GW to the internet
Thanks a lot and appreciate to hear back from you if we need to adjust the config what would be the best practice or if we have to increase the maxrun time or the wait time between each script's line?
Below is the Chassis/IOS type and version with event manager config and status of applets.
filtered SYS log is attached showing the stuck issue.
Chassis: ISR4331/K9
IOS: isr4300-universalk9.03.16.04c.S.155-3.S4c-ext.SPA.bin"
+++++++++++++event manager config is below
Event manager applet1 is when track 101 is down we need to clear crypto then reset tunnels 1 and 2 then add a sys log "MONITOR-TRACK101-DOWN"
Evenet manager applet 2 is when track 102 is down we need to shut tunnel 2 then add a sys log "MONITOR-TRACK102-DOWN"
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
event manager session cli username "mshaltou"
event manager applet MONITOR-TRACK101-DOWN
event track 101 state down
action 10 cli command "enable"
action 11 cli command "conf t"
action 12 cli command "clear crypto iskamp"
action 20 cli command "int tunnel 1"
action 21 cli command "shut"
action 22 wait 2
action 23 cli command "no shut"
action 30 cli command "int tunnel 2"
action 31 cli command "shut"
action 32 wait 2
action 33 cli command "no shut"
action 40 syslog msg "MONITOR-TRACK101-DOWN"
event manager applet MONITOR-TRACK102-DOWN
event track 102 state down
action 10 cli command "enable"
action 11 cli command "conf t"
action 30 cli command "int tunnel 2"
action 31 cli command "shut"
action 32 wait 2
action 33 cli command "no shut"
action 40 syslog msg "MONITOR-TRACK102-DOWN"
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+++++++++++Show event manager
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
pmil3736# sh event manager policy registered
No. Class Type Event Type Trap Time Registered Name
1 applet user track Off Wed May 31 14:25:16 2017 MONITOR-TRACK101-DOWN
track 101 state down
maxrun 20.000
action 10 cli command "enable"
action 11 cli command "conf t"
action 12 cli command "clear crypto iskamp"
action 20 cli command "int tunnel 1"
action 21 cli command "shut"
action 22 wait 2
action 23 cli command "no shut"
action 30 cli command "int tunnel 2"
action 31 cli command "shut"
action 32 wait 2
action 33 cli command "no shut"
action 40 syslog msg "MONITOR-TRACK101-DOWN"
2 applet user track Off Wed May 31 14:25:27 2017 MONITOR-TRACK102-DOWN
track 102 state down
maxrun 20.000
action 10 cli command "enable"
action 11 cli command "conf t"
action 30 cli command "int tunnel 2"
action 31 cli command "shut"
action 32 wait 2
action 33 cli command "no shut"
action 40 syslog msg "MONITOR-TRACK102-DOWN"
pmil3736#sh event manager version
Embedded Event Manager Version 4.00
Component Versions:
eem: (rel9)1.2.12
eem-gold: (rel1)1.0.2
eem-call-home: (rel2)1.0.5
Event Detectors:
Name Version Node Type
application 01.00 node0/0 RP
rf 01.00 node0/0 RP
syslog 01.00 node0/0 RP
neighbor-discovery 01.00 node0/0 RP
generic 01.00 node0/0 RP
track 01.00 node0/0 RP
routing 02.00 node0/0 RP
resource 01.00 node0/0 RP
cli 01.00 node0/0 RP
counter 01.00 node0/0 RP
interface 01.00 node0/0 RP
ioswdsysmon 01.00 node0/0 RP
none 01.00 node0/0 RP
oir 01.00 node0/0 RP
snmp 01.00 node0/0 RP
snmp-object 01.00 node0/0 RP
snmp-notification 01.00 node0/0 RP
timer 01.00 node0/0 RP
test 01.00 node0/0 RP
config 01.00 node0/0 RP
env 01.00 node0/0 RP
ds 01.00 node0/0 RP
crash 01.00 node0/0 RP
gold 01.00 node0/0 RP
rpc 01.00 node0/0 RP
pmil3736#
pmil3736#sh event MANAger history events detailed
No. Job Id Proc Status Time of Event Event Type Name
1 379 Actv success Sun Aug27 01:13:13 2017 track applet: MONITOR-TRACK102-DOWN
2 380 Actv success Sun Aug27 01:31:14 2017 track applet: MONITOR-TRACK102-DOWN
3 381 Actv success Thu Aug31 12:49:13 2017 track applet: MONITOR-TRACK102-DOWN
4 382 Actv success Fri Sep 1 01:41:10 2017 track applet: MONITOR-TRACK102-DOWN
5 383 Actv success Fri Sep 1 02:59:12 2017 track applet: MONITOR-TRACK102-DOWN
6 384 Actv abort Fri Sep 1 03:15:12 2017 track applet: MONITOR-TRACK102-DOWN >>>>>>>>>>>>> aborted
7 385 Actv success Fri Sep 1 09:59:11 2017 track applet: MONITOR-TRACK102-DOWN
8 386 Actv abort Fri Sep 1 10:02:11 2017 track applet: MONITOR-TRACK102-DOWN >>>>>>>>>>>>>> aborted
9 387 Actv success Mon Sep 4 10:12:13 2017 track applet: MONITOR-TRACK102-DOWN
10 388 Actv abort Mon Sep 4 12:10:11 2017 track applet: MONITOR-TRACK102-DOWN >>>>>>>>> aborted
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
09-13-2017 12:14 PM
You are likely hitting the maxrun timer of the applet. By default this is 20 seconds. You may need to increase it to 30 or perhaps higher.
Add "maxrun 30" after the event specification lines for each applet and see if that solves your problem. If not, try going to 40 or 60.
09-13-2017 12:48 PM
Hi Joe,
Thanks for the reply, we've added an event timer watchdog as this running on ISR IOS. We'll keep it under monitoring for the time being, hopefully this should be it.
pmil3736(config)#event manager applet MONITOR-TRACK101-DOWN
pmil3736(config-applet)#event timer watchdog time 50 maxrun 30
pmil3736(config-applet)#event manager applet MONITOR-TRACK102-DOWN
pmil3736(config-applet)#event timer watchdog time 50 maxrun 30
Thanks again.
09-14-2017 04:22 PM
Hi,
Had to change the config a bit, as the one before just ran the script every 50 sec.
event manager applet MONITOR-TRACK101-DOWN
no event timer watchdog time 50 maxrun 30
event track 101 state down maxrun 40
event manager applet MONITOR-TRACK102-DOWN
event timer watchdog time 50 maxrun 30
event track 102 state down maxrun 40
09-13-2017 02:44 PM - edited 09-13-2017 02:45 PM
Hi
If you want to see where is the inconvenience or what is happening, you can execute the following debug during the execution of the EEM Script
debug event manager action
or
debug event manager all
Hope it is useful
:-)
09-14-2017 04:26 PM
Hi Julio,
Thanks for the tip... we've enabled debuging for the action CLI in case it happens again.
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: