eem script issue

[STACY WOOD]

we have two eem scripts configured to look for a syslog event tracking a VRRP change.  They work great!  However we have run into an issue that has occurred 2 times now where the VRRP flapped within a millisecond time frame.  As a result, our router saw it take over as Master run the hr4700down then while it saw it fail back to backup, it could not run hr4700up since the other app was still running.  Is there any way to check that the applet failed to run and force it to try again?  Any help to automate this would be appreciated.  We cannot use IP SLA as some of our routers require licenses we do not have to run the sla commands.  So this was an alternative that has worked until it didn't.

Here's our scripts:

event manager session cli username "EEMScript"
event manager applet hr4700down
 event syslog pattern "%VRRP-6-STATECHANGE: Gi0/0.29 Grp 29 state Backup -> Master"
 action 1 cli command "enable"
 action 2 cli command "config term"
 action 2.1 cli command "router bgp 65029"
 action 2.2 cli command "redistribute connected"
 action 2.3 cli command "end"

event manager applet hr4700up
 event syslog pattern "%VRRP-6-STATECHANGE: Gi0/0.29 Grp 29 state Master -> Backup"
 action 1 cli command "enable"
 action 2 cli command "config term"
 action 2.1 cli command "router bgp 65029"
 action 2.2 cli command "no redistribute connected"
 action 2.3 cli command "end"

Here's an example of a log where it failed to run-  notice hr4700down ran at 5:43:09.023, hr4700up ran at 05:43:28.400, hr4700down ran at 06:49:28.537, vrrp flapped again, but hr4700up DID NOT RUN

Oct 23 05:43:08.923: %VRRP-6-STATECHANGE: Gi0/0.29 Grp 29 state Backup -> Master
Oct 23 05:43:08.923: %VRRP-6-STATECHANGE: Gi0/0 Grp 1 state Backup -> Master
Oct 23 05:43:08.923: %VRRP-6-STATECHANGE: Gi0/1.10 Grp 10 state Backup -> Master
Oct 23 05:43:08.923: %VRRP-6-STATECHANGE: Gi0/1.8 Grp 8 state Backup -> Master
Oct 23 05:43:09.023: %SYS-5-CONFIG_I: Configured from console by EEMScript on vty0 (EEM:hr4700down)
Oct 23 05:43:28.300: %VRRP-6-STATECHANGE: Gi0/1.8 Grp 8 state Master -> Backup
Oct 23 05:43:28.300: %VRRP-6-STATECHANGE: Gi0/1.10 Grp 10 state Master -> Backup
Oct 23 05:43:28.300: %VRRP-6-STATECHANGE: Gi0/0 Grp 1 state Master -> Backup
Oct 23 05:43:28.300: %VRRP-6-STATECHANGE: Gi0/0.29 Grp 29 state Master -> Backup
Oct 23 05:43:28.400: %SYS-5-CONFIG_I: Configured from console by EEMScript on vty0 (EEM:hr4700up)
Oct 23 05:43:29.296: %IP-4-DUPADDR: Duplicate address 172.20.29.1 on GigabitEthernet0/1.11, sourced by 0000.5e00.010b
Oct 23 06:49:28.429: %VRRP-6-STATECHANGE: Gi0/1.8 Grp 8 state Backup -> Master
Oct 23 06:49:28.429: %VRRP-6-STATECHANGE: Gi0/1.10 Grp 10 state Backup -> Master
Oct 23 06:49:28.429: %VRRP-6-STATECHANGE: Gi0/0 Grp 1 state Backup -> Master
Oct 23 06:49:28.429: %VRRP-6-STATECHANGE: Gi0/0.29 Grp 29 state Backup -> Master
Oct 23 06:49:28.537: %SYS-5-CONFIG_I: Configured from console by EEMScript on vty0 (EEM:hr4700down)
Oct 23 06:49:28.721: %VRRP-6-STATECHANGE: Gi0/1.8 Grp 8 state Master -> Backup
Oct 23 06:49:28.721: %VRRP-6-STATECHANGE: Gi0/1.10 Grp 10 state Master -> Backup
Oct 23 06:49:28.725: %VRRP-6-STATECHANGE: Gi0/0 Grp 1 state Master -> Backup
Oct 23 06:49:28.725: %VRRP-6-STATECHANGE: Gi0/0.29 Grp 29 state Master -> Backup
(no hr4700up script ran this time)

[Joe Clarke · Accepted solution]

I don't see why this second applet wouldn't run.  By default, recent versions of IOS have five applet threads allocated, so it should have executed.  Do you have other EEM policies configured?  Are you able to run some debugging when this happens?  If you look at "show event manager history events" do you see if the policy ran?  Perhaps it ran but didn't make the config changes.

[Joe Clarke]

BTW: No you can't rerun a failed policy.  You'd have to cause the syslog message to be regenerated (in this case).

[STACY WOOD]

Great info! thanks Joe.

I think that we locked the router down and prevented multiple applet threads to run.  So just made change to allow it to login more than once.  We will see if that helps.

[STACY WOOD]

https://supportforums.cisco.com/document/48891/cisco-eem-best-practices#comment-10900291

so there's my answer.  we increased the # of login's of the eem script (increase line vty 0 1) at the expense of one of the interactive vty lines.  Easy choice for us.

Thanks again Joe!