Cisco 9300 setup for Oracle ZS5 with two active controllers

[alanf]

We have an Oracle ZS5 with two controllers that operate in Active-Active mode that is connected to two Cisco 9300 switches that are clustered together. A single ZS5 controller has two ethernet ports configured as LACP that connect to a port-channel (two ports) on the 9300 switch. This is duplicated on the other controller and obviously goes to a separate port-channel on the 9300 switch. Each controller has two VNICs connected to the same LACP interface and each VNIC has a separate IP address. Both configs are the same on each controller so when in Active-Active mode, controller 1 uses IP 1 and talks via VNIC 1 to port-channel 1 and controller 2 uses IP 2 and talks via VNIC 2 to port-channel 2.

When in Active-Passive mode (or say controller 1 failed) the Active controller (now 2) takes over control of both VNICs so then IP 1 and 2 traffic is directed to port-channel 2. 

For ZS5 outgoing traffic there should be no issue with the source MAC address table changing?

For ZS5 incoming traffic, how does the 9300 know to switch all inbound IP 1 traffic to port-channel 2?

Port-channels are on L2 as is LACP on the controllers so I assume I need some form of switching of MAC address table?

[Georg Pauwen]

Hello,

very interesting question. In Active/Passive mode, is the corresponding port channel on the 9300 actually up or down for the passive controller ? The default mac address table aging time for the 9300 is 300 seconds, so that cannot be the mechanism used.

Can you enable 'debug etherchannel' and initiate the failover ?

[balaji.bandi]

As per switch concern, it does not matter what device is connected, as long as it maintains the same MAC address when the failover takes place.

how does Oracle ZS5 failover take place, is the standby take over when the active fail with the same IP and MAC address, is this changed?

Can you show what your port-channel and interface config look like, Hope they are identical and VLAN is allowed in the port channel, and what IP address both use the same subnet?

[alanf]

Hi,

We are going to perform a takeover Friday morning to see what happens. Cant do anything else but analyse at the moment.

The ZS5 controller ethernet ports all have different MAC addresses so I dont yet know if I can change that.

The ZS5 can be manually set to failover using its admin interface so that you can then do maintenance on one controller (which is what we need to do). It will also failover if one controller fails in some way.

The IP addresses are the same on both ZS5 controllers (ie Controller 1 has VNIC 1 at .18 and VNIC 2 as .19 and Controller 2 is the same config. All the same subnets/netmasks).

The 9300 settings related to the ZS5: I wonder if the port-channel interfaces should be in trunk mode rather than access?

!
interface Port-channel6
description ZS5-2-2
switchport mode access
spanning-tree portfast
!
interface Port-channel7
description ZS5-2-1
switchport mode access
spanning-tree portfast
!

interface TenGigabitEthernet1/0/45
description ZS5-2-2 Net2
switchport mode access
channel-protocol lacp
channel-group 6 mode active
spanning-tree portfast trunk
ip nbar protocol-discovery
!
interface TenGigabitEthernet1/0/46
description ZS5-2-1 Net2
switchport mode access
channel-protocol lacp
channel-group 7 mode active
spanning-tree portfast trunk
ip nbar protocol-discovery
!

interface TenGigabitEthernet2/0/45
description ZS5-2-1 Net3
switchport mode access
channel-protocol lacp
channel-group 6 mode active
spanning-tree portfast trunk
ip nbar protocol-discovery
!
interface TenGigabitEthernet2/0/46
description ZS5-2-1 Net3
switchport mode access
channel-protocol lacp
channel-group 7 mode active
spanning-tree portfast trunk
ip nbar protocol-discovery
!

Channel group 6
LACP port Admin Oper Port Port
Port Flags State Priority Key Key Number State
Te1/0/45 SA bndl 32768 0x6 0x6 0x12E 0x3D
Te2/0/45 SA bndl 32768 0x6 0x6 0x22E 0x3D

Channel group 7
LACP port Admin Oper Port Port
Port Flags State Priority Key Key Number State
Te1/0/46 SA bndl 32768 0x7 0x7 0x12F 0x3D
Te2/0/46 SA bndl 32768 0x7 0x7 0x22F 0x3D

[alanf]

Just so we are all on the same page with regard to Active-Active vs Active-Passive I created this diagram

 

[alanf]

The only thing that I have come up with so far is when switching to Active-Passive, I have to login to the 9300 and do a 'clear mac address-table'. That way the 9300 will re-learn the locations and then do the same when switching back to Active-Active.

[alanf]

I think I have figured out the issue. 

The MAC address of VNIC1 on ZS5-01 is different to VNIC1 on ZS5-02, same with VNIC2. Therefore when switching to Active-passive the MAC address changes but the 9300 has no idea.

I have asked Oracle if we can manually set the VNIC MAC addresses to the same on both controller, lets hope that fixes the problem

[balaji.bandi]

what are the MAC Address timeout settings? try to reduce that. also you can set SNMP alrert and with EEM script you can clear MAC address table for that IP against MAC (other option)

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9300/software/release/16-8/configuration_guide/sys_mgmt/b_168_sys_mgmt_9300_cg/b_168_sys_mgmt_9300_cg_chapter_00.html#topic_iry_kxr_n1b

[alanf]

Hey BB,

The timeout is 300 seconds and you cant reduce that. I did the 'clear mac address-table' and that worked nicely with the allocation of MAC to port. The switch will obviously do this automatically after some time also.

I tried changing the MAC address of the VNICs on the ZS5 but the system doesn't let me so have to wait for Oracle on that.

We did the failover this morning and all Solaris servers with NFS shares that were on the ZS5-01 would not work. This is a Solaris/ZS5 config problem, I personally don't think any switch settings will help.

[balaji.bandi]

what are you looking as a solution here?

my suggestion, you need to reduce the MAC address learning, since Client side not moving the MAC address same, ( same was observed with fold Fujistu servers)

use EEM Script clear only that mac - you should not clear all the MAC addresses (this will have big impact on production clearing with clear mac address-table)

[alanf]

Hi BB,

I agree, I only cleared all dynamic MAC addresses but should only clear the connections in question.

End goal is to be able to failover to a single ZS5 controller without having to re-boot all the servers (which is what we have had to do at the moment).

I have looked further and actually I think the real issue is NFS v3 shares don't like to have a break in connection. So the answer may actually be to clear the ARP tables in the servers (or statically set them). NFS v4 seems to recover much better but still with some issues. SMB seems to work fine. 

[balaji.bandi]

Thanks for the feedback, if that is the only option due to limitations, sometimes we need to go with the solution.