cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2488
Views
10
Helpful
5
Replies

Event manager stuck at a cli command and the status of the applet become abort sometimes

Hello All,

I hope you are doing fine, I am VPO and I have an issue with my event manager created on the router
I raised a CISCO TAC and the assigned engineer advised me to check here in the EEM community issues and raise my query so i came to you.

The issue : I have applet manager with event track and action reset crypto and interfaces, the APPLET work as expected some time and give a status successful and stuck some time at shut interfaces without unshutiing them back and give us abort status applet and that caused internet isolation to our customer because the interfaces that were shut was our GW to the internet


Thanks a lot and appreciate to hear back from you if we need to adjust the config what would be the best practice or if we have to increase the maxrun time or the wait time between each script's line?


Below is the Chassis/IOS type and version with event manager config and status of applets.
filtered SYS log is attached showing the stuck issue.


Chassis: ISR4331/K9
IOS: isr4300-universalk9.03.16.04c.S.155-3.S4c-ext.SPA.bin"

+++++++++++++event manager config is below

Event manager applet1 is when track 101 is down we need to clear crypto then reset tunnels 1 and 2 then add a sys log "MONITOR-TRACK101-DOWN"

Evenet manager applet 2 is when track 102 is down we need to shut tunnel 2 then add a sys log "MONITOR-TRACK102-DOWN"
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

event manager session cli username "mshaltou"
event manager applet MONITOR-TRACK101-DOWN
 event track 101 state down
 action 10 cli command "enable"
 action 11 cli command "conf t"
 action 12 cli command "clear crypto iskamp"
 action 20 cli command "int tunnel 1"
 action 21 cli command "shut"
 action 22 wait 2
 action 23 cli command "no shut"
 action 30 cli command "int tunnel 2"
 action 31 cli command "shut"
 action 32 wait 2
 action 33 cli command "no shut"
 action 40 syslog msg "MONITOR-TRACK101-DOWN"
event manager applet MONITOR-TRACK102-DOWN
 event track 102 state down
 action 10 cli command "enable"
 action 11 cli command "conf t"
 action 30 cli command "int tunnel 2"
 action 31 cli command "shut"
 action 32 wait 2
 action 33 cli command "no shut"
 action 40 syslog msg "MONITOR-TRACK102-DOWN"


!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!


+++++++++++Show event manager
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

pmil3736#     sh event manager policy registered
No.  Class     Type    Event Type          Trap  Time Registered           Name
1    applet    user    track               Off   Wed May 31 14:25:16 2017  MONITOR-TRACK101-DOWN
 track 101 state down
 maxrun 20.000
 action 10 cli command "enable"
 action 11 cli command "conf t"
 action 12 cli command "clear crypto iskamp"
 action 20 cli command "int tunnel 1"
 action 21 cli command "shut"
 action 22 wait 2
 action 23 cli command "no shut"
 action 30 cli command "int tunnel 2"
 action 31 cli command "shut"
 action 32 wait 2
 action 33 cli command "no shut"
 action 40 syslog msg "MONITOR-TRACK101-DOWN"

2    applet    user    track               Off   Wed May 31 14:25:27 2017  MONITOR-TRACK102-DOWN
 track 102 state down
 maxrun 20.000
 action 10 cli command "enable"
 action 11 cli command "conf t"
 action 30 cli command "int tunnel 2"
 action 31 cli command "shut"
 action 32 wait 2
 action 33 cli command "no shut"
 action 40 syslog msg "MONITOR-TRACK102-DOWN"


pmil3736#sh event manager version
Embedded Event Manager Version 4.00
Component Versions:
eem: (rel9)1.2.12
eem-gold: (rel1)1.0.2
eem-call-home: (rel2)1.0.5
Event Detectors:
Name                Version   Node        Type
application         01.00     node0/0     RP
rf                  01.00     node0/0     RP
syslog              01.00     node0/0     RP
neighbor-discovery  01.00     node0/0     RP
generic             01.00     node0/0     RP
track               01.00     node0/0     RP
routing             02.00     node0/0     RP
resource            01.00     node0/0     RP
cli                 01.00     node0/0     RP
counter             01.00     node0/0     RP
interface           01.00     node0/0     RP
ioswdsysmon         01.00     node0/0     RP
none                01.00     node0/0     RP
oir                 01.00     node0/0     RP
snmp                01.00     node0/0     RP
snmp-object         01.00     node0/0     RP
snmp-notification   01.00     node0/0     RP
timer               01.00     node0/0     RP
test                01.00     node0/0     RP
config              01.00     node0/0     RP
env                 01.00     node0/0     RP
ds                  01.00     node0/0     RP
crash               01.00     node0/0     RP
gold                01.00     node0/0     RP
rpc                 01.00     node0/0     RP

pmil3736#


pmil3736#sh event MANAger history events detailed
No.  Job Id Proc Status   Time of Event            Event Type        Name
1    379    Actv success  Sun Aug27 01:13:13 2017  track             applet: MONITOR-TRACK102-DOWN

2    380    Actv success  Sun Aug27 01:31:14 2017  track             applet: MONITOR-TRACK102-DOWN

3    381    Actv success  Thu Aug31 12:49:13 2017  track             applet: MONITOR-TRACK102-DOWN

4    382    Actv success  Fri Sep 1 01:41:10 2017  track             applet: MONITOR-TRACK102-DOWN

5    383    Actv success  Fri Sep 1 02:59:12 2017  track             applet: MONITOR-TRACK102-DOWN

6    384    Actv abort    Fri Sep 1 03:15:12 2017  track             applet: MONITOR-TRACK102-DOWN >>>>>>>>>>>>> aborted

7    385    Actv success  Fri Sep 1 09:59:11 2017  track             applet: MONITOR-TRACK102-DOWN

8    386    Actv abort    Fri Sep 1 10:02:11 2017  track             applet: MONITOR-TRACK102-DOWN >>>>>>>>>>>>>> aborted

9    387    Actv success  Mon Sep 4 10:12:13 2017  track             applet: MONITOR-TRACK102-DOWN

10   388    Actv abort    Mon Sep 4 12:10:11 2017  track             applet: MONITOR-TRACK102-DOWN >>>>>>>>> aborted

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!



5 Replies 5

Joe Clarke
Cisco Employee
Cisco Employee

You are likely hitting the maxrun timer of the applet.  By default this is 20 seconds.  You may need to increase it to 30 or perhaps higher.

 

Add "maxrun 30" after the event specification lines for each applet and see if that solves your problem.  If not, try going to 40 or 60.

Hi Joe,

 

Thanks for the reply, we've added an event timer watchdog as this running on ISR IOS. We'll keep it under monitoring for the time being, hopefully this should be it.

 

pmil3736(config)#event manager applet MONITOR-TRACK101-DOWN

pmil3736(config-applet)#event timer watchdog time 50 maxrun 30

pmil3736(config-applet)#event manager applet MONITOR-TRACK102-DOWN
pmil3736(config-applet)#event timer watchdog time 50 maxrun 30

 

Thanks again.

Hi,

 

Had to change the config a bit, as the one before just ran the script every 50 sec.

 

 

event manager applet MONITOR-TRACK101-DOWN

no event timer watchdog time 50 maxrun 30

event track 101 state down maxrun 40

event manager applet MONITOR-TRACK102-DOWN

event timer watchdog time 50 maxrun 30

event track 102 state down maxrun 40

 

Hi 

If you want to see where is the inconvenience or what is happening, you can execute the following debug during the execution of the EEM Script 

 

debug event manager action

or

debug event manager all

 

Hope it is useful

:-)




>> Marcar como útil o contestado, si la respuesta resolvió la duda, esto ayuda a futuras consultas de otros miembros de la comunidad. <<

Hi Julio,

 

Thanks for the tip... we've enabled debuging for the action CLI in case it happens again.

 

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: