I've been working on an applet(s) for an issue we have with a redundant cellular connection intermittently failing and restoring. We have discovered that when the connection is bouncing or failed; we can switch the cellular technology from auto to gsm and back to auto to restore stable connectivity.
Under normal operating conditions, we have a primary MPLS connection but if the primary MPLS connection fails, we route the cellular traffic over 1 of 2 tunnels (primary and secondary BGP neighbors.) I have the main part of the nested applets working as designed but I am not sure and my concern is, if the triggers/timing are scripted effectively.
We are observing the following conditions with the cellular connection:
Scenario 1 - Our 2 BGP neighbors flap continuously(sometimes its only 1 neighbor - We're only concerned if BOTH flap) until the technology is switched.
Scenario 2 - The cellular connection fails hard(interface remains up/up - Only BGP neighbors fail) - Switching the technology often resolves this condition as well.
The behavior I'm trying to achieve from the applet is as follows:
- If both BGP neighbors flap 3 times in 5 minutes - Kickoff the applet
- If both BGP neighbors fail hard for 3 minutes (max delay down for Track) - Kick off the applet
- When the main applet is called, we want the nested applets to run every XX minutes (watchdog timer) but limited to once every XX minutes (ratelimit)
Hopefully I'm making sense thus far - Below are the configs for the IP SLA and Tracks I'm using.
ip sla 1
icmp-echo 172.20.182.1
threshold 3000
timeout 3000
frequency 3
ip sla schedule 1 life forever start-time now
ip sla 2
icmp-echo 172.20.183.1
threshold 3000
timeout 3000
frequency 3
ip sla schedule 2 life forever start-time now
track 1 ip sla 1 reachability
!
track 2 ip sla 2 reachability
!
track 3 list threshold percentage
object 1
object 2
delay down 180
And the main applet
NOTE: The times used are only for testing purposes and will be adjusted to more effective values when this is fielded.
event manager environment q "
event manager applet track-down
event tag 1.0 syslog occurs 3 pattern "%BGP-5-ADJCHANGE: neighbor 172.20.183.1 Down*" period 300
event tag 2.0 syslog occurs 3 pattern "%BGP-5-ADJCHANGE: neighbor 172.20.182.1 Down*" period 300
event tag 3.0 track 3 state down
trigger
correlate event 1.0 and event 2.0 or event 3.0
action 000 syslog msg "EEM_APPLET_KICKED_OFF"
action 001 cli command "enable"
action 002 cli command "config t"
action 003 cli command "event manager applet cellular-to-gsm"
action 004 cli command "description This applet will change the cellular technology to gsm"
action 005 cli command "event timer watchdog time 15 ratelimit 120"
action 006 cli command "action 1.0 cli command enable"
action 007 cli command "action 2.0 cli command $q config t$q"
action 008 cli command "action 2.1 cli command $q int ce 0/0/0$q"
action 009 cli command "action 2.2 cli command $q shut$q"
action 010 cli command "action 2.3 cli command $q end$q"
action 011 cli command "action 3.0 cli command $q cell 0/0/0 lte tech gsm$q"
action 012 cli command "action 4.0 cli command enable"
action 013 cli command "action 4.1 cli command $q config t$q"
action 014 cli command "action 4.2 cli command $q int ce 0/0/0$q"
action 015 cli command "action 4.3 cli command $q no shut$q"
action 016 cli command "action 4.4 cli command $q end$q"
action 017 cli command "event manager applet cellular-to-auto"
action 018 cli command "description This applet will change the cellular technology to auto"
action 019 cli command "event timer watchdog time 30 ratelimit 120"
action 020 cli command "action 1.0 cli command enable"
action 021 cli command "action 2.0 cli command $q config t$q"
action 022 cli command "action 2.1 cli command $q int ce 0/0/0$q"
action 023 cli command "action 2.2 cli command $q shut$q"
action 024 cli command "action 2.3 cli command $q end$q"
action 025 cli command "action 3.0 cli command $q cell 0/0/0 lte tech auto$q"
action 026 cli command "action 4.0 cli command enable"
action 027 cli command "action 4.1 cli command $q config t$q"
action 028 cli command "action 4.2 cli command $q int ce 0/0/0$q"
action 029 cli command "action 4.3 cli command $q no shut$q"
action 030 cli command "event manager applet cell-power-cycle"
action 031 cli command "description This applet will power cycle the cellular modem"
action 032 cli command "event timer watchdog time 45 ratelimit 120"
action 033 cli command "action 1.0 cli command enable"
action 034 cli command "action 1.1 cli command $q config t$q"
action 035 cli command "action 1.2 cli command $q service internal$q"
action 036 cli command "action 1.3 cli command $q end$q"
action 037 cli command "action 2.0 cli command $q test cellular 0/0/0 modem-power-cycle$q"
Below is the applet used to remove the nested applets upon restoral of the tracks but I cannot seem to nail down the proper scripting to effectively remove the nested apples. In the below example, you'll see when the tracks come up its trying to remove the applets multiple times. I understand based on the sytax of the applet it's working as designed but I'm unsure if theres a way to streamline the restoral and only have the nested applets removed once as opposed to 3 times.
*Sep 19 06:16:24: %TRACK-6-STATE: 1 ip sla 1 reachability Down -> Up
*Sep 19 06:16:24: %TRACK-6-STATE: 2 ip sla 2 reachability Down -> Up
*Sep 19 06:16:24: %SYS-5-CONFIG_I: Configured from console by on vty0 (EEM:track-up)
*Sep 19 06:16:24: %SYS-5-CONFIG_I: Configured from console by on vty1 (EEM:track-up)
*Sep 19 06:16:24: %TRACK-6-STATE: 3 list threshold percentage Down -> Up
*Sep 19 06:16:24: %SYS-5-CONFIG_I: Configured from console by on vty0 (EEM:track-up)
event manager applet track-up
event tag 1.0 track 1 state up
event tag 2.0 track 2 state up
event tag 3.0 track 3 state up
trigger
correlate event 1.0 or event 2.0 or event 3.0
action 001 cli command "enable"
action 002 cli command "config t"
action 003 cli command "no event manager applet cellular-to-gsm"
action 004 cli command "no event manager applet cellular-to-auto"
action 005 cli command "no event manager applet cell-power-cycle"
action 006 cli command "end"
Thank you for any input and/or suggestions to improve the functionality of this solution. If there is any confusion or details I did not provide, please feel free to ask for clarification. Also, please feel free to raise any concerns regarding the behavior and/or logic in this build and any foreseeable issues I may have overlooked or not considered.
Again, I appreciate the assistance. Thanks!
Robert
