Monitor and Shutdown interfaces

MARTIN HUERTER · ‎10-15-2012

I have been working on a script to monitor interfaces on our redudnant border routers and border firewalls and shut down interfaces from our core Catalyst 6509 switches to the border routers to fail everything over to our other border router/firewall/ISP. Basically what I would lik to do is run a script on the two core Catalyst 6509 switches running IOS s72033-adventerprise_wan-vz.122-33.SXI10 on them. Below is the script configuration I've been using and I've attached a diagram to hopefully illustrate what I am talking about. I have configured the script on the 6509 switches in a test environment, but it doesn't appear to be functioning properly when I break one of the interfaces the scripts is monitoring, nor does it take the action and shutdown interface Gig 1/1. Can someone check my configuration to see what I am missing or tell me what commands to use to see why it isn't working? Thanks!

event manager applet S1 Firewall ISP interfaces
event tag SLA1 track 1 state down
event tag SLA2 track 2 state down
event tag SLA3 track 3 state down
trigger
correlate event SLA1 or event SLA2 or event SLA3
action 1.0 cli command "enable"
action 2.0 cli command "conf t"
action 3.0 cli command "interface GigabitEthernet1/1"
action 4.0 cli command "shut"
action 5.0 cli command "exit"

track 1 ip sla 1 reachability
track 2 ip sla 2 reachability
track 3 ip sla 3 reachability
ip sla 1
icmp-echo 2.2.2.1 source-ip 10.10.10.10
frequency 5
ip sla schedule 1 life forever start-time now
ip sla 2
icmp-echo 3.3.3.1 source-ip 10.10.10.10
frequency 5
ip sla schedule 2 life forever start-time now
ip sla 3
icmp-echo 4.4.4.1 source-ip 10.10.10.10
frequency 5

event manager applet S2 Firewall ISP interfaces
event tag SLA1 track 1 state down
event tag SLA2 track 2 state down
event tag SLA3 track 3 state down
trigger
correlate event SLA1 or event SLA2 or event SLA3
action 1.0 cli command "enable"
action 2.0 cli command "conf t"
action 3.0 cli command "interface GigabitEthernet1/1"
action 4.0 cli command "shut"
action 5.0 cli command "exit"

track 1 ip sla 1 reachability
track 2 ip sla 2 reachability
track 3 ip sla 3 reachability
ip sla 1
icmp-echo 2.2.2.1 source-ip 20.20.20.20
frequency 5
ip sla schedule 1 life forever start-time now
ip sla 2
icmp-echo 3.3.3.1 source-ip 20.20.20.20
frequency 5
ip sla schedule 2 life forever start-time now
ip sla 3
icmp-echo 4.4.4.1 source-ip 20.20.20.20
frequency 5

Joe Clarke · ‎10-15-2012

This part of the config looks okay to me. Are you seeing the tracked objects go down? When you fail one of the interfaces, what does "show track" report? Do you have AAA command authorization configured on the 6500? If so, you will need to configure:

event manager session cli username USER

Where USER is a user authorized to run all of these commands.

MARTIN HUERTER · ‎10-16-2012

Joseph,

Thanks for the response. We use ACS as our AAA server, but we also have local accounts. On the configuration I was using on my lab gear I was using the local account user "root", but from your response I may have had it configured wrong. When I connect via console on the box and use root and root password it is already enabled. This is how I had it configured.

action 1.0 cli command "root"

action 1.0 cli command "rootpasword"

action 2.0 cli command "conf t"

action 3.0 cli command "interface GigabitEthernet1/1"

action 4.0 cli command "shut"

action 5.0 cli command "exit"

How should I have it configured? Are there any command I can run to see if this is where the problem is?

Thanks,

Martin

MARTIN HUERTER · ‎10-16-2012

Below is my actual configuration on the 6509 on our lab.

event manager session cli username "root"
event manager applet lab_core_to_lab_border1
event tag SLA1 track 1 state down
event tag SLA2 track 2 state down
event tag SLA3 track 3 state down
event tag SLA4 track 4 state down
event tag SLA5 track 5 state down
trigger
correlate event SLA1 or event SLA2 or event SLA3 or event SLA4 or event SLA5
action 1.0 cli command "conf t"
action 2.0 cli command "interface GigabitEthernet2/1"
action 3.0 cli command "shut"
action 4.0 cli command "exit"
action 5.0 cli command "exit"

track 1 ip sla 1 reachability
track 2 ip sla 2 reachability
track 3 ip sla 3 reachability
track 4 ip sla 4 reachability
track 5 ip sla 5 reachability

ip sla 1
icmp-echo 10.255.5.50 source-ip 10.255.5.4
frequency 10
ip sla 2
icmp-echo 10.255.5.58 source-ip 10.255.5.4
frequency 10
ip sla 3
icmp-echo 10.255.5.66 source-ip 10.255.5.4
frequency 10
ip sla 4
icmp-echo 10.255.5.74 source-ip 10.255.5.4
frequency 10
ip sla 5
icmp-echo 10.255.204.226 source-ip 10.255.5.4
frequency 10

MARTIN HUERTER · ‎10-16-2012

Below are some "show" commands I ran which I thought were relevant to the configuration above. The script is not working and I can't figure out why. Am I missing something? Are there other "show" commands I can run to see why this script isn't working?

lab-core-01#sho event manager history events

No. Job Id Status Time of Event Event Type Name

1 1 success Thu Oct 11 08:36:39 2012 track applet: lab_core_to_lab_border1

2 2 success Thu Oct 11 08:36:39 2012 track applet: lab_core_to_lab_border1

3 4 success Thu Oct 11 08:36:39 2012 track applet: lab_core_to_lab_border1

4 3 success Thu Oct 11 08:36:39 2012 track applet: lab_core_to_lab_border1

5 5 success Thu Oct 11 08:36:54 2012 track applet: lab_core_to_lab_border1

6 6 success Fri Oct 12 14:23:01 2012 track applet: lab_core_to_lab_border1

7 7 success Fri Oct 12 14:23:21 2012 track applet: lab_core_to_lab_border1

8 8 success Fri Oct 12 14:24:11 2012 track applet: lab_core_to_lab_border1

lab-core-01#

lab-core-01#sho track

Track 1

IP SLA 1 reachability

Reachability is Down

4 changes, last change 3d19h

Latest operation return code: Unknown

Tracked by:

EEM applet lab_core_to_lab_border1

Track 2

IP SLA 2 reachability

Reachability is Down

4 changes, last change 3d19h

Latest operation return code: Unknown

Tracked by:

EEM applet lab_core_to_lab_border1

Track 3

IP SLA 3 reachability

Reachability is Down

2 changes, last change 5d01h

Latest operation return code: Unknown

Tracked by:

EEM applet lab_core_to_lab_border1

Track 4

IP SLA 4 reachability

Reachability is Down

2 changes, last change 5d01h

Latest operation return code: Unknown

Tracked by:

EEM applet lab_core_to_lab_border1

Track 5

IP SLA 5 reachability

Reachability is Down

4 changes, last change 3d19h

Latest operation return code: Unknown

Tracked by:

EEM applet lab_core_to_lab_border1

lab-core-01#

lab-core-01#sho ip sla statistics details
Round Trip Time (RTT) for Index 1
Type of operation: icmp-echo
Number of successes: Unknown
Number of failures: Unknown
Operation time to live: 0
Operational state of entry: Inactive
Last time this entry was reset: Never

Round Trip Time (RTT) for Index 2
Type of operation: icmp-echo
Number of successes: Unknown
Number of failures: Unknown
Operation time to live: 0
Operational state of entry: Inactive
Last time this entry was reset: Never

Round Trip Time (RTT) for Index 3
Type of operation: icmp-echo
Number of successes: Unknown
Number of failures: Unknown
Operation time to live: 0
Operational state of entry: Inactive
Last time this entry was reset: Never

Round Trip Time (RTT) for Index 4
Type of operation: icmp-echo
Number of successes: Unknown
Number of failures: Unknown
Operation time to live: 0
Operational state of entry: Inactive
Last time this entry was reset: Never

lab-core-01#sho event manager history events
No. Job Id      Status   Time of Event             Event Type          Name
1    1           success Thu Oct 11 08:36:39 2012 track               applet: lab_core_to_lab_border1
2    2           success Thu Oct 11 08:36:39 2012 track               applet: lab_core_to_lab_border1
3    4           success Thu Oct 11 08:36:39 2012 track               applet: lab_core_to_lab_border1
4    3           success Thu Oct 11 08:36:39 2012 track               applet: lab_core_to_lab_border1
5    5           success Thu Oct 11 08:36:54 2012 track               applet: lab_core_to_lab_border1
6    6           success Fri Oct 12 14:23:01 2012 track               applet: lab_core_to_lab_border1
7    7           success Fri Oct 12 14:23:21 2012 track               applet: lab_core_to_lab_border1
8    8           success Fri Oct 12 14:24:11 2012 track               applet: lab_core_to_lab_border1
lab-core-01#

lab-core-01#sho track
Track 1
IP SLA 1 reachability
Reachability is Down
    4 changes, last change 3d19h
Latest operation return code: Unknown
Tracked by:
    EEM applet lab_core_to_lab_border1
Track 2
IP SLA 2 reachability
Reachability is Down
    4 changes, last change 3d19h
Latest operation return code: Unknown
Tracked by:
    EEM applet lab_core_to_lab_border1
Track 3
IP SLA 3 reachability
Reachability is Down
    2 changes, last change 5d01h
Latest operation return code: Unknown
Tracked by:
    EEM applet lab_core_to_lab_border1
Track 4
IP SLA 4 reachability
Reachability is Down
    2 changes, last change 5d01h
Latest operation return code: Unknown
Tracked by:
    EEM applet lab_core_to_lab_border1
Track 5
IP SLA 5 reachability
Reachability is Down
    4 changes, last change 3d19h
Latest operation return code: Unknown
Tracked by:
    EEM applet lab_core_to_lab_border1
lab-core-01#

lab-core-01#sho ip sla statistics details
Round Trip Time (RTT) for Index 1
Type of operation: icmp-echo
Number of successes: Unknown
Number of failures: Unknown
Operation time to live: 0
Operational state of entry: Inactive
Last time this entry was reset: Never

Round Trip Time (RTT) for Index 2
Type of operation: icmp-echo
Number of successes: Unknown
Number of failures: Unknown
Operation time to live: 0
Operational state of entry: Inactive
Last time this entry was reset: Never

Round Trip Time (RTT) for Index 3
Type of operation: icmp-echo
Number of successes: Unknown
Number of failures: Unknown
Operation time to live: 0
Operational state of entry: Inactive
Last time this entry was reset: Never

Round Trip Time (RTT) for Index 4
Type of operation: icmp-echo
Number of successes: Unknown
Number of failures: Unknown
Operation time to live: 0
Operational state of entry: Inactive
Last time this entry was reset: Never

MARTIN HUERTER · ‎10-16-2012

Looks like my output from my "show" commands copied and pasted twice. Sorry!

MARTIN HUERTER · ‎10-16-2012

Finally got it working!!!

Joe Clarke · ‎10-16-2012

What was the issue? Based on your last config snippet, it looked like it should work from an EEM standpoint. It seemed like there might have been an IPSLA issue.

MARTIN HUERTER · ‎10-18-2012

It was in the IP SLA configuration, but a little embarrassing. I had the wrong source IP Address configured. It is all working now, just need to do a little tweeking on it.

Question: If I wanted to suspend and then un-suspend a EEM script, how would I go about it, or what are the IOS commands? I tried to use "event manager scheduler suspend", it suspends the script but can't get it going again.

Joe Clarke · ‎10-18-2012

Re-enabling the scheduler should work. Removing that command should re-enable the scheduler and allow your policies to resume. You can try entering applet sub-mode then exit and see if that helps.