Top of Rack Nexus 9300 Have L3 Card and need to Use IP SLA in Conjunction with Event Mgr

[rmcdaniel]

I had a TOR L3 Module Failure and need to use IP SLA to ping the (2) Uplinks out of the Nexus 9300 if that fails I want to have Event Mgr take down Links to a B22 Fex in a Dell M1000e Chassis. 

This way the VM's will see the Nic go Not-connected as a result of the Uplink Path being Down and will fail to alternate B22 through other TOR Sw.

I have IP SLA working when I bring the (2) Uplinks down but Event Mgr never brings down the specified interfaces and need some assistance.

ip sla 1
  icmp-echo 10.52.0.189 source-interface Ethernet1/46
    verify-data
    threshold 100
    timeout 1000
    frequency 5
ip sla schedule 1 life forever start-time now

ip sla 2
  icmp-echo 10.52.0.181 source-interface Ethernet1/47
    verify-data
    threshold 100
    timeout 1000
    frequency 5
ip sla schedule 2 life forever start-time now

- Then , assign a track to check for reachability for each one:

config#Track 1 ip sla 1 reachability
config#Track 2 ip sla 2 reachability

- Finally , we need to create a third track of type Boolean that will track the state of both tracks assigned to the ip sla probes and ,
if they are down , the third track will go down as well :

config#Track 3 list Boolean and
config-track#object 1  
config-track#object 2
config-track#exit

event manager applet default_down
  event track 3 state down
  action 0 cli command "conf t"
  action 1 cli command "int E1/22-23"
  action 2 cli command "shut"

event manager applet default_up
  event track 3 state up
  action 0 cli command "conf t"
  action 1 cli command "int E1/22-23"
  action 2 cli command "no shut"

[Vinit Jain]

Hi

Have you verified if the track1 and track2 are going down. Could you please share the output of below commands when you bring down the uplinks. :

- show track 1

- show track 2

- show track 3

My suspect is, the object 3 doesnt go down due to 1 and 2 not being down causing EEM not to get triggered. 

I have tested your config in lab and it works, so it should work for you as well.

Hopefully will figure it out.

Thanks

Vinit

[rmcdaniel]

s1-ten9348-k2ci11# sh track 1
Track 1
  IP SLA 1 Reachability
  Reachability is DOWN
  8 changes, last change 4d16h
  Latest operation return code: Timeout
  Tracked by:
    Track List  3

s1-ten9348-k2ci11# sh track 2
Track 2
  IP SLA 2 Reachability
  Reachability is DOWN
  8 changes, last change 4d16h
  Latest operation return code: Timeout
  Tracked by:
    Track List  3

s1-ten9348-k2ci11# sh track 3
Track 3
  List  Boolean and
  Boolean and is DOWN
  7 changes, last change 4d16h
  Track List Members:
  object 2 DOWN
  object 1 DOWN

I also update the EEM Statements to reflect this:

Added the Enable statement under Event Action 0 this is being performed on a Nexus9000 93180YC-EX chassis.

ip sla 1
 icmp-echo 10.52.0.189 source-interface Ethernet1/46
    verify-data
    threshold 100
    timeout 1000
    frequency 5
ip sla schedule 1 life forever start-time now

ip sla 2
 icmp-echo 10.52.0.181 source-interface Ethernet1/47
    verify-data
    threshold 100
    timeout 1000
    frequency 5       
ip sla schedule 2 life forever start-time now

- Then , assign a track to check for reachability for each one:

config#Track 1 ip sla 1 reachability
config#Track 2 ip sla 2 reachability

- Finally , we need to create a third track of type Boolean that will track the state of both tracks assigned to the ip sla probes and ,
if they are down , the third track will go down as well :

config#Track 3 list Boolean and
config-track#object 1  
config-track#object 2
config-track#exit

event manager applet default_down
  event track 3 state down
action 0 cli command “enable”
  action 1 cli command "conf t"
  action 2 cli command "int E1/22-23"
  action 3 cli command "shut"
action 4 cli command “end”

event manager applet default_up
  event track 3 state up
action 0 cli command “enable”
  action 1 cli command "conf t"
  action 2 cli command "int E1/22-23"
  action 3 cli command "no shut"
action 4 cli command “end”

Let me know your thoughts after reviewing.

Thanks

Rob McDaniel

rob_mcdaniel@dell.com

[Joe Clarke]

Enable isn't needed on the N9K.  And as Vinit pointed out, this should work.  Are you using AAA command authz on this device?  It could be that EEM is not authorized to run those commands.  The other thing to check is can you manually type the exact same sequence of commands and see those two ports go down?

[Vinit Jain]

Hi Joe

i just had a webex with Rob and figured out that this was possibly due to a defect and also captured the debugs which can further be used to work with Cisco TAC to get this config working.

Thanks

Vinit

[rmcdaniel]

I had one more response below so when I shut the interfaces as you see below 1/46,1/47 I expect to see the interfaces E1/22,E1/23 to go down.

Normal All up and have Displays below:

s1-ten9348-k2ci11(config)# sh track 1
Track 1
  IP SLA 1 Reachability
  Reachability is UP
  9 changes, last change 00:02:01
  Latest operation return code: OK
  Latest RTT (millisecs): 1
  Tracked by:
    Track List  3

s1-ten9348-k2ci11(config)# sh track 2
Track 2
  IP SLA 2 Reachability
  Reachability is UP
  9 changes, last change 00:02:04
  Latest operation return code: OK
  Latest RTT (millisecs): 1
  Tracked by:
    Track List  3

s1-ten9348-k2ci11(config)# sh track 3
Track 3
  List  Boolean and
  Boolean and is UP
  8 changes, last change 00:02:07
  Track List Members:
  object 2 UP
  object 1 UP

Eth1/22       FEX VPC to the Del connected 1         full    10G     SFP-H10GB-CU3M
Eth1/23       FEX VPC to the Del connected 1         full    10G     SFP-H10GB-CU3M

Eth1/46       s700-tenptc E8/18  connected routed    full    10G     10Gbase-SR
Eth1/47       s1-tenptc E7/17 Ce connected routed    full    10G     10Gbase-LR

*****************************************

Shut Ports E1/46 and E1/47 then did a Sh on track 1,2,3

s1-ten9348-k2ci11(config)# int e1/46
s1-ten9348-k2ci11(config-if)# shut
s1-ten9348-k2ci11(config-if)# int e1/47
s1-ten9348-k2ci11(config-if)# shut
s1-ten9348-k2ci11(config-if)# sh track 1
Track 1
  IP SLA 1 Reachability
  Reachability is DOWN
  10 changes, last change 00:00:02
  Latest operation return code: Timeout
  Tracked by:
    Track List  3

s1-ten9348-k2ci11(config-if)# sh track 2
Track 2
  IP SLA 2 Reachability
  Reachability is DOWN
  10 changes, last change 00:00:07
  Latest operation return code: Timeout
  Tracked by:
    Track List  3

s1-ten9348-k2ci11(config-if)# sh track 3
Track 3
  List  Boolean and
  Boolean and is DOWN
  9 changes, last change 00:00:13
  Track List Members:
  object 2 DOWN
  object 1 DOWN

Eth1/22       FEX VPC to the Del connected 1         full    10G     SFP-H10GB-CU3M
Eth1/23       FEX VPC to the Del connected 1         full    10G     SFP-H10GB-CU3M

Eth1/46       s700-tenptc E8/18  disabled  routed    full    auto    10Gbase-SR
Eth1/47       s1-tenptc E7/17 Ce disabled  routed    full    auto    10Gbase-LR