EEM CLI Script Tracking Multiple Interfaces 3850 Layer 3 Switch

[andrewdours]

Any help is greatly appreciated. Here's what I'm trying to do. See attached for PDF diagram. Overall, I'm running ip sla to check for latency above 150ms or a ping timeout. If that happens, I want to set the interface to passive in the EIGRP configuration. I can do this individually on each of the 4 routed interfaces. However, I need some logic so that I don't end up setting all interfaces to passive during some weird network anomaly. The idea was to use loopbacks on each of my core switches. Each loopback would correspond to one of the routed interfaces. I'll configure those for ip sla for ping reachability. When the routed interface experiences high latency (really the only logical case for this protection mechanism), the EEM script fires. As part of the script, the corresponding loopback will be shutdown. Before this happens, the script has to someway be able to check the ip sla for the loopbacks to be sure that this routed interface is not the last one up. Maybe there is a better way. I was looking at tags and triggers earlier today but I'm not sure how to test each ip sla condition. I can't seem to find any documentation on the variable $_ipsla_condition. The other idea I had was to use a common counter variable and increment that when the ip sla for a loopback fires. I'm not sure how I would use a common variable. It appears the the EEM script syntax is quite different between router and switch platforms. If anyone can point me to better documentation on EEM scripting for 3850 layer 3 switches that might help too.

Here's what I have right now for just one routed interface:

ip sla 192
icmp-echo 10.10.10.10 source-interface TenGigabitEthernet1/0/24
threshold 150
timeout 500
frequency 5
ip sla schedule 192 life forever start-time now
ip sla reaction-configuration 192 react timeout threshold-type consecutive 6
!
event manager applet Timeout-EIGRP-Passive-Te1/0/24
event ipsla operation-id 192 reaction-type timeout
action 1.0 if $_ipsla_condition eq "Occurred"
action 2.0 syslog msg "Timeout-IPSLA-192-down"
action 3.0 cli command "enable"
action 3.1 cli command "config t"
action 3.2 cli command "router eigrp eigrpnet"
action 3.3 cli command "address-family ipv4 unicast autonomous-system 100"
action 3.4 cli command "af-interface TenGigabitEthernet1/0/24"
action 3.5 cli command "passive-interface"
action 3.6 cli command "end"
action 4.0 else
action 5.0 syslog msg "Timeout-IPSLA-192-up"
action 6.0 cli command "enable"
action 6.1 cli command "config t"
action 6.2 cli command "router eigrp eigrpnet"
action 6.3 cli command "address-family ipv4 unicast autonomous-system 100"
action 6.4 cli command "af-interface TenGigabitEthernet1/0/24"
action 6.5 cli command "no passive-interface"
action 6.6 cli command "end"
action 9.9 end

Within this command section, I could easily shutdown the corresponding loopbck interface.  I just didn't add it in here since I don't know which direction to go to evaluate if a loopback is still up.

Thanks again,

Andrew

[Joe Clarke · Accepted solution]

First, I don't know why you think EEM applet syntax is different between switches and routers.  It is not.  It is the same across all IOS platforms provided the version of EEM is the same.

Probably the easiest way to check for a block of interface statuses is to use a tracked object list.  That is, create tracked objects for each loopback then create a threshold list to track the set.  If less than 50% of the interfaces are down, then the track is down.  You can check that within an EEM applet.  For example:

track 1 interface lo0 line-protocol

...

track 5 list threshold percentage

 threshold percentage up 50

 object 1

 object 2

 object 3

 object 4

And within the applet:

track read 5

if $_track_state eq down

! Less than 50% you are taking down the last interface

[andrewdours]

Thanks Joe?
I think I'm getting hung up on the fact that you can't have an event with the track object on this platform. On one of our other routers, I can use the "event track". This morning, I found about what you have proposed by reading the tracked object state. I believe that what you have provided is going to work great. Once I have it in configuration, I'll post for all to see.
Andrew

[andrewdours]

Joe,

With the information from your post, this is what I've come up with.  I have 2 applets for each routed interface on each core switch.  Below is the configuration from one switch.  I suppose there is a better way to consolidate these, but I kept getting an error when I attempted to use event tags with a trigger.  The last 2 applets below were an attempt to check on the status of the configuration sync between the loopback and the routed interface.  I don't know if there is ever a situation where the routed interface and the corresponding loopback states could get out of sync.  In any case, I can't figure out how to accurately check the status of whether a routed interface has been set to passive.  I used netem running on Ubuntu to induce latency to test.  Worked great.  Here are the references that I used:  https://calomel.org/network_loss_emulation.html and https://help.ubuntu.com/community/NetworkConnectionBridge.

track 10 list threshold percentage
 object 251
 object 252
 object 253
 object 254
 threshold percentage down 25 up 50
!
track 101 ip sla 201 reachability
 delay down 10 up 50
!
track 102 ip sla 202 reachability
 delay down 10 up 50
!
track 251 ip sla 251 reachability
!
track 252 ip sla 252 reachability
!
track 253 ip sla 253 reachability
!
track 254 ip sla 254 reachability
!
ip sla 201
 icmp-echo 10.10.10.10 source-interface TenGigabitEthernet1/0/24
 threshold 150
 timeout 500
 frequency 5
ip sla schedule 201 life forever start-time now
ip sla 202
 icmp-echo 10.10.10.20 source-interface TenGigabitEthernet2/0/24
 threshold 150
 timeout 500
 frequency 5
ip sla schedule 202 life forever start-time now
ip sla 251
 icmp-echo 192.168.1.251 source-interface Vlan249
 threshold 2000
 timeout 2000
 frequency 5
ip sla schedule 251 life forever start-time now
ip sla 252
 icmp-echo 192.168.1.252 source-interface Vlan249
 threshold 2000
 timeout 2000
 frequency 5
ip sla schedule 252 life forever start-time now
ip sla 253
 icmp-echo 192.168.1.253 source-interface Vlan249
 threshold 2000
 timeout 2000
 frequency 5
ip sla schedule 253 life forever start-time now
ip sla 254
 icmp-echo 192.168.1.254 source-interface Vlan249
 threshold 2000
 timeout 2000
 frequency 5
ip sla schedule 254 life forever start-time now
ip sla reaction-configuration 201 react timeout threshold-type consecutive 4
ip sla reaction-configuration 201 react rtt threshold-value 150 150 threshold-type consecutive 10
ip sla reaction-configuration 202 react timeout threshold-type consecutive 6
ip sla reaction-configuration 202 react rtt threshold-value 150 150 threshold-type consecutive 12
ip sla enable reaction-alerts
!
event manager applet Monitor-Layer3-Interface-Te1/0/24-Latency 
description Checks interface connection to core-switch1 using ip sla 201.  If latency is over 150ms for more than 50 seconds, set the interface to passive in EIGRP and shut down loopback251. 
 event ipsla operation-id 201 reaction-type rtt maxrun 90 
action 1010 syslog msg "Start EEM Applet due to IP SLA 201 State Change."
action 1020 if $_ipsla_condition eq "Occurred"
action 1030 track read 10
action 1040 if $_track_state eq "up"
action 1050 syslog msg "Disabling interface Loopback251, setting EIGRP passive for interface Te1/0/24"
action 1060  cli command "enable"
action 1070  cli command "config t"
action 1080 cli command "interface lo251"
action 1090 cli command "shutdown"
action 1100 cli command "exit"
action 1110  cli command "router eigrp eigrpnet"
action 1120  cli command "address-family ipv4 unicast autonomous-system 100"
action 1130  cli command "af-interface TenGigabitEthernet1/0/24"
action 1140  cli command "passive-interface"
action 1150  cli command "end"
action 1160  else
action 1170 syslog msg "Finish EEM Applet.  No action taken.  Cannot shut down last remaining routed interface."
action 1180 exit
action 1190  end
action 1200  else
action 1210 syslog msg "Enabling interface Loopback251, setting EIGRP no-passive for interface Te1/0/24"
action 1220   cli command "enable"
action 1230   cli command "config t"
action 1240 cli command "interface lo251"
action 1250 cli command "no shutdown"
action 1260 cli command "exit"
action 1270   cli command "router eigrp eigrpnet"
action 1280   cli command "address-family ipv4 unicast autonomous-system 100"
action 1290   cli command "af-interface TenGigabitEthernet1/0/24"
action 1300   cli command "no passive-interface"
action 1310   cli command "end"
action 1320 syslog msg "Finish EEM Applet."
action 1330 end
!
event manager applet Monitor-Layer3-Interface-Te1/0/24-Timeout 
description Checks interface connection to core-switch1 using ip sla 201.  If connection times out for more than 20 seconds, set the interface to passive in EIGRP and shut down loopback251. 
 event ipsla operation-id 201 reaction-type timeout maxrun 90 
action 1010 syslog msg "Start EEM Applet due to IP SLA 201 State Change."
action 1020 if $_ipsla_condition eq "Occurred"
action 1030 track read 10
action 1040 if $_track_state eq "up"
action 1050 syslog msg "Disabling interface Loopback251, setting EIGRP passive for interface Te1/0/24."
action 1060  cli command "enable"
action 1070  cli command "config t"
action 1080 cli command "interface lo251"
action 1090 cli command "shutdown"
action 1100 cli command "exit"
action 1110  cli command "router eigrp eigrpnet"
action 1120  cli command "address-family ipv4 unicast autonomous-system 100"
action 1130  cli command "af-interface TenGigabitEthernet1/0/24"
action 1140  cli command "passive-interface"
action 1150  cli command "end"
action 1160  else
action 1170 syslog msg "Finish EEM Applet.  No action taken.  Cannot shut down last remaining routed interface."
action 1180 exit
action 1190  end
action 1200  else
action 1210 syslog msg "Enabling interface Loopback251, setting EIGRP no-passive for interface Te1/0/24."
action 1220   cli command "enable"
action 1230   cli command "config t"
action 1240 cli command "interface lo251"
action 1250 cli command "no shutdown"
action 1260 cli command "exit"
action 1270   cli command "router eigrp eigrpnet"
action 1280   cli command "address-family ipv4 unicast autonomous-system 100"
action 1290   cli command "af-interface TenGigabitEthernet1/0/24"
action 1300   cli command "no passive-interface"
action 1310   cli command "end"
action 1320 syslog msg "Finish EEM Applet."
action 1330 end
!
event manager applet Monitor-Layer3-Interface-Te2/0/24-Latency 
description Checks interface connection to core-switch2 using ip sla 202.  If latency is over 150ms for more than 60 seconds, set the interface to passive in EIGRP and shut down loopback252. 
 event ipsla operation-id 202 reaction-type rtt maxrun 90 
action 1010 syslog msg "Start EEM Applet due to IP SLA 202 State Change."
action 1020 if $_ipsla_condition eq "Occurred"
action 1030 track read 10
action 1040 if $_track_state eq "up"
action 1050 syslog msg "Disabling interface Loopback252, setting EIGRP passive for interface Te2/0/24"
action 1060  cli command "enable"
action 1070  cli command "config t"
action 1080 cli command "interface lo252"
action 1090 cli command "shutdown"
action 1100 cli command "exit"
action 1110  cli command "router eigrp eigrpnet"
action 1120  cli command "address-family ipv4 unicast autonomous-system 100"
action 1130  cli command "af-interface TenGigabitEthernet2/0/24"
action 1140  cli command "passive-interface"
action 1150  cli command "end"
action 1160  else
action 1170 syslog msg "Finish EEM Applet.  No action taken.  Cannot shut down last remaining routed interface."
action 1180  end
action 1190 exit
action 1200  else
action 1210 syslog msg "Enabling interface Loopback252, setting EIGRP no-passive for interface Te2/0/24"
action 1220   cli command "enable"
action 1230   cli command "config t"
action 1240 cli command "interface lo252"
action 1250 cli command "no shutdown"
action 1260 cli command "exit"
action 1270   cli command "router eigrp eigrpnet"
action 1280   cli command "address-family ipv4 unicast autonomous-system 100"
action 1290   cli command "af-interface TenGigabitEthernet2/0/24"
action 1300   cli command "no passive-interface"
action 1310   cli command "end"
action 1320 syslog msg "Finish EEM Applet."
action 1330 end
!
event manager applet Monitor-Layer3-Interface-Te2/0/24-Timeout 
description Checks interface connection to core-switch2 using ip sla 202.  If connection times out for more than 20 seconds, set the interface to passive in EIGRP and shut down loopback252. 
 event ipsla operation-id 202 reaction-type timeout maxrun 90 
action 1010 syslog msg "Start EEM Applet due to IP SLA 202 State Change."
action 1020 if $_ipsla_condition eq "Occurred"
action 1030 track read 10
action 1040 if $_track_state eq "up"
action 1050 syslog msg "Disabling interface Loopback252, setting EIGRP passive for interface Te2/0/24."
action 1060  cli command "enable"
action 1070  cli command "config t"
action 1080 cli command "interface lo252"
action 1090 cli command "shutdown"
action 1100 cli command "exit"
action 1110  cli command "router eigrp eigrpnet"
action 1120  cli command "address-family ipv4 unicast autonomous-system 100"
action 1130  cli command "af-interface TenGigabitEthernet1/0/24"
action 1140  cli command "passive-interface"
action 1150  cli command "end"
action 1160  else
action 1170 syslog msg "Finish EEM Applet.  No action taken.  Cannot shut down last remaining routed interface."
action 1180 exit
action 1190  end
action 1200  else
action 1210 syslog msg "Enabling interface Loopback252, setting EIGRP no-passive for interface Te2/0/24."
action 1220   cli command "enable"
action 1230   cli command "config t"
action 1240 cli command "interface lo252"
action 1250 cli command "no shutdown"
action 1260 cli command "exit"
action 1270   cli command "router eigrp eigrpnet"
action 1280   cli command "address-family ipv4 unicast autonomous-system 100"
action 1290   cli command "af-interface TenGigabitEthernet1/0/24"
action 1300   cli command "no passive-interface"
action 1310   cli command "end"
action 1320 syslog msg "Finish EEM Applet."
action 1330 end
!
event manager applet Validate-Interface-Te1/0/24-State 
event timer cron cron-entry "0 5 * * *" 
action 1000 syslog msg "Start EEM Applet"
action 1010 track read 101
action 1020 set var101 "$_track_state"
action 1030 track read 251
action 1040 set var251 "$_track_state"
action 1050 if $var101 eq "$var251"
action 1060  exit
action 1070 elseif $var101 eq "up"
action 1080  syslog msg "Interface Te1/0/24 out of sync with Loopback 251.  Reconfiguring Loopback251, exit EEM"
action 1090 cli command "enable"
action 1100 cli command "config t"
action 1110 cli command "interface lo251"
action 1120 cli command "no shut"
action 1130 cli command "end"
action 1140 cli command "wr"
action 1150 syslog msg "Finish EEM Applet."
action 1160 exit
action 1170 elseif $var101 eq "down"
action 1180  syslog msg "Interface Te1/0/24 out of sync with Loopback 251.  Reconfiguring Loopback251, exit EEM"
action 1190 cli command "enable"
action 1200 cli command "config t"
action 1210 cli command "interface lo251"
action 1220 cli command "shut"
action 1230 cli command "end"
action 1240 cli command "wr"
action 1250 syslog msg "Finish EEM Applet."
action 1260 exit
action 1270 else
action 1280 end
!
event manager applet Validate-Interface-Te2/0/24-State 
event timer cron cron-entry "1 5 * * *" 
action 1000 syslog msg "Start EEM Applet"
action 1010 track read 102
action 1020 set var102 "$_track_state"
action 1030 track read 252
action 1040 set var252 "$_track_state"
action 1050 if $var102 eq "$var252"
action 1060  exit
action 1070 elseif $var102 eq "up"
action 1080  syslog msg "Interface Te2/0/24 out of sync with Loopback 252.  Reconfiguring Loopback252, exit EEM"
action 1090 cli command "enable"
action 1100 cli command "config t"
action 1110 cli command "interface lo252"
action 1120 cli command "no shut"
action 1130 cli command "end"
action 1140 cli command "wr"
action 1150 syslog msg "Finish EEM Applet."
action 1160 exit
action 1170 elseif $var102 eq "down"
action 1180  syslog msg "Interface Te2/0/24 out of sync with Loopback 252.  Reconfiguring Loopback252, exit EEM"
action 1190 cli command "enable"
action 1200 cli command "config t"
action 1210 cli command "interface lo252"
action 1220 cli command "shut"
action 1230 cli command "end"
action 1240 cli command "wr"
action 1250 syslog msg "Finish EEM Applet."
action 1260 exit
action 1270 else
action 1280 end
!

Thanks,

Andrew

[Joe Clarke]

You can parse the output of "show ip eigrp interface INTF" I think.  With OSPF you can search for "Passive interface" in the output.  EIGRP should have a similar result.

[andrewdours]

I cleaned up the validation EEM applet.  It checks ip protocols for interfaces set to passive.  For this to run properly I had to enter this command:  event manager session cli username "<local defined user"

event manager applet Validate-Interface-States
event timer cron cron-entry "0 5 * * *"
action 1000 track read 251
action 1010 set var251 "$_track_state"
action 1020 if $var251 eq "down"
action 1030 set var251 1
action 1040 elseif $var251 eq "up"
action 1050 set var251 0
action 1060 end
action 1070 track read 252
action 1080 set var252 "$_track_state"
action 1090 if $var252 eq "down"
action 1100 set var252 1
action 1110 elseif $var252 eq "up"
action 1120 set var252 0
action 1130 end
action 1140 cli command "show ip protocols"
action 1150 regexp "TenGigabitEthernet1/0/24" "$_cli_result"
action 1160 set var1024 "$_regexp_result"
action 1170 regexp "TenGigabitEthernet2/0/24" "$_cli_result"
action 1180 set var2024 "$_regexp_result"
action 1190 if $var251 eq $var1024
action 1200 if $var252 eq $var2024
action 1210 syslog msg "Finish EEM Applet. No action taken"
action 1220 exit
action 1230 end
action 1240 end
action 1250 if $var251 ne $var1024
action 1260 if $var1024 eq "1"
action 1270 syslog msg "Te1/0/24 is set to EIGRP passive, shutting down Loopback251"
action 1280 cli command "enable"
action 1290 cli command "config t"
action 1300 cli command "int lo251"
action 1310 cli command "shutdown"
action 1320 cli command "wr"
action 1330 else
action 1340 syslog msg "Te1/0/24 is set to EIGRP no-passive, enabling Loopback251"
action 1350 cli command "enable"
action 1360 cli command "config t"
action 1370 cli command "int lo251"
action 1380 cli command "no shutdown"
action 1390 cli command "wr"
action 1400 end
action 1410 end
action 1420 if $var252 ne $var2024
action 1430 if $var2024 eq "1"
action 1440 syslog msg "Te2/0/24 is set to EIGRP passive, shutting down Loopback252"
action 1450 cli command "enable"
action 1460 cli command "config t"
action 1470 cli command "int lo252"
action 1480 cli command "shutdown"
action 1490 cli command "wr"
action 1500 else
action 1510 syslog msg "Te2/0/24 is set to EIGRP no-passive, enabling Loopback252"
action 1520 cli command "enable"
action 1530 cli command "config t"
action 1540 cli command "int lo252"
action 1550 cli command "no shutdown"
action 1560 cli command "wr"
action 1570 end
action 1580 end
action 1590 syslog msg "Finish EEM Applet."