ACI 4.2(5K) and IBM Main Frame duplicate IP error

[joseph.morris@safelite.com]

Hi all. Having an issue where EPM process crashes due to a duplicate IP error from our mainframe. The main frame is load balancing between two physical nics. The problem is the source MAC address changes with every packet as it round robins each pack to each nic. This obviously causes ACI to constantly rewrite the ARP DB thus driving the EPM process crazy. The main frame admin and the 3rd party support we have insist that the main frame is configured as "best practice". Cisco TAC suggested move the main frame out of ACI. We can't be the first time that ACI and a main frame were connected together can we?

[Hector Gustavo Serrano Gutierrez]

Hi joseph.morris@safelite.com 

The usual approach to dual home Servers in ACI are the below scenarios:

1. Virtual Port Channel (vPC) - Port-Channel bundling is required on the Server.
2. Active/standby NIC Teaming
3. Port Channel for active/active NIC Teaming- Port-Channel bundling is required on the Server.

See the attached image.

Moving forward, based on your description, it sounds like you are using a 4th variant where your main frame configured with NIC Teaming active/active is not bundled in a Port-Channel style on the main frame side and it is sending the same source IP from multiple NIC cards with different MAC addresses. As you've mentioned, in ACI this will cause IP flapping between those two MAC addresses on ACI endpoint learning (EPM - Endpoint Manager) due to ACI's IP data plane learning. If the teaming in your main frame configuration cannot be changed, you could disable the IP data plane learning in the VRF configuration in ACI. With the IP data plane learning disabled, the Endpoint table is updated only by the control plane just like any other switches and ARP will be the single source of MAC-IP mapping.

As a final note, there can be a 5th variant is when the server uses active/active teaming but still source the traffic with a single MAC and IP. This will not cause the IP flapping mentioned above. Instead, it causes the flapping of the entire endpoint (both MAC and IP). Even for a regular Layer 2 Switches (not ACI), this is an issue that is exposed as a MAC flapping.

[joseph.morris@safelite.com]

ok. Thank you. I will just hav to move them out of ACI for now as the business has no appetite to make changes to the mainfram configuration at this time.

[anacochristofaro]

Same issue after upgrade to 5.2(4e) even with the IP data plane learning in the VRF configuration disabled.

[Robert Burns]

With the latest software version you no longer have to disable IP DP learning on the VRF completely to support hosts where the VIP floats.  Instead you can add a /32 record under the EPG to tell ACI to ignore DP learning on individual hosts.

[Robert Burns]

Can they modify the mainframes NIC teaming?  Either change to Active/Passive or aggregate them together (Port channel).  The end goal is that we want to limit the Virtual IP bouncing between pMACs.  They may have it aligned to Best-Practices for a traditional switch, but its not best practice for connecting this type of device to ACI.
Do you have unicast routing enabled on the Bridge Domain of the Mainframe btw?

Robert

[anacochristofaro]

Hi Robert!

We can't modify the mainframes NIC teaming, and we have a PC Mode-On Policy for this kind of connections. Yes, we have unicast routing enabled on the bridge domain. We already had this problem with 5.2(3e), that was solved by this setup:

• VRF IP Data-Plane Learning Disabled
• BD ARP Flooding Enabled
• BD Limit Local IP Learning to BD/EPG Subnet Enabled
• BD Unicast Routing Enabled
• BD GARP Based Detection Enabled
• BD IP Data-Plane Learning for PBR Node Disabled 

Once the setup was validated in a lab tenant, we started the migration from other structures/tenants. Despite that, after upgrading to 5.2(4e), the problem appeared again. The environment has been unstable for five days, with no solution given by Cisco.

[Robert Burns]

@anacochristofaro  you mention above: 
"We can't modify the mainframes NIC teaming, and we have a PC Mode-On Policy for this kind of connections."

Is this correct?  I didn't think z/OS supported NIC teaming.  If the interfaces were teamed, I wouldn't expect any flapping issue on the ACI side.

Robert

[RedNectar]

Hi joseph.morris@safelite.com and @anacochristofaro ,

OK. I misread the problem. Two MACs, one IP. My mind was thinking "One MAC two NICs" when I wrote the following, which thankfully @Robert Burns has politely pointed out is wrong.

I'd call out the concept that using the same MAC address on two different physical interfaces is "best practice"

If you can give me an IEEE paper that says that using the same MAC address ACTIVELY in two places is best practice, I'll eat my hat.  In and Active/Standby scenario (as others have suggested), it is perfectly acceptable. But ACTIVE/ACTIVE on the same MAC address is going to cause MAC flapping no matter what equipment you use.

[Robert Burns]

@RedNectar  I don't think he said that.  At least not what I gathered.  Sounds like he has two individual links, which have unique MAC addresses but a floating VIP.  Typical use case for link teaming/aggregation when connecting to ACI. 

Robert