ACI - CUBE VIP Issue

[mbrandon32]

We recently migrated a pair of 4451 CUBE routers into our ACI fabric. Once we migrated layer 3 into the fabric, we started seeing issues where the CUBE VIP would go unresponsive for a minute or two - which takes the SIP trunk offline and drops all active calls. When the VIP was unresponsive, we didn't see any logs on the router stating that an interface was dropping.

Looking in the APIC, we saw that the VIP was attaching and detaching, constantly.

Is there a best practice configuration on the ACI side to handle a pair of routers with a VIP for HA?

[Shibasish]

Hi mbrandon32,

Thanks for your query.

Since you have migrated Cube HA into Cisco ACI and if I assume you have connected Cube just as an external device, two things will happen here

1. The Router will respond to ARP using the vMAC and VIP.
2. Cube Routers sources the traffic from VIP IP  but instead of vMAC it will use Physical MAC address. 

Which means you have two different MAC addresses are associated with same IP. This actually leads to issue in case with Cisco ACI since it expects the traffic to be sourced from the same VIP and vMAC because ACI can learn IP from the data plane, so it will see the same IP (VIP) associated with two MAC addresses (vMAC and Physical MAC) which will cause flapping between two MAC addresses.

There is a Bug CSCvj66014 for CUBE (CUBE-HA should Use the Same VMAC AND VIP to source SIP Packets), but it is not a CUBE feature related issue issue, it’s more on underlying platform for HA design.

The best possible solution from ACI Deployment perspective : 

 - Connect CUBE router via L3out to avoid EP learning Challenge

Also, In ACI Deployment with release 4.0(1h) or later IP data plane learning for the VRF can be disabled and can be an option, however the changes will have an impact that needs to be considered before changing the settings. This option is located at Tenant > Networking > VRFs

With ACI release 5.2(1g) ACI allows IP data plane learning to be disabled at the EPG and BD Level as well.

https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-in...

https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-739989.html#IPDataplaneLearningpersubnet

You can also learn more about ACI through our live Ask the Experts (ATXs) session. Check out Cisco ACI ATXs Resources: https://community.cisco.com/t5/data-center-and-cloud-knowledge/cisco-aci-ask-the-experts-resources/ta-p/4394491 to view the latest schedule for upcoming sessions, as well as the useful references, e.g. online guides, FAQs.

Feel free to reach out to me if you have any question.

[RedNectar]

Wow - what a great answer from @Shibasish , let's hope we see more from him/her.

All I'm going to do is summarise the already great answer

@mbrandon32 , the two most relevant choices are:

1. Connect CUBE router via L3out to avoid EP learning Challenge
2. Co to the EPG > Subnets where the CUBE device connects, add a /32 subnet for the CUBE's VIP and set the IP Data-plane Learning option to Disabled
   • This option requires ACI APIC Release 5.2(1g) or later - if you don't have this you can disable IP Data-plane learning for the whole subnet on the BD

[BANERJEE SHIBASISH]

Thanks @RedNectar !

Hi @mbrandon32, hope your query has been answered.