I have few queries with respect to NSX over ACI. Appreciate if you could help me understand better.
1) VMware Integration:
Since VMware NSX is already present in the network, routing, Switching and firewall services are provided by NSX. Should ACI act just as a transit network for VMware to External networks and VMware to Physical servers.
To be more specific, VDS is not managed by APIC and hence the network visibility in the VMware environment is obsolete. So, in that case since the VDS is not deployed from ACI, there is nothing that we can do to match the traffic on per vlan basis.
I also understand that according to the design document , we need to create 5 EPGs for NSX traffic and every EPG will be statically mapped to vlan ID. But my question is how will I know which VLAN ID should I use for these 5 EPGs? Does the traffic enter the ACI leaf as a vxlan packet or will there be a tag of the vlan on the NSX VXLAN packet hitting the leaf ?
Please let me know if I my understanding in wrong.
As per the VMware PS, ACI leaf will have a BGP peering with the NSX edge and I will be using a default route towards the External L3out connected device.
Do you recommend using BGP for the external L3out as well? If so, can I know why?
3) Design Consideration
Is there a set of services which has to be in the common tenant - any best practices based on your experiences.
I came across your post. I'm sorry that I don't have the answers but I am curious about your experience with NSX and ACI. My company have to replace traditional nexus DC and (although not in prod yet) the virtual network will be moved to NSX. If you can your experience would be appreciated. If you get value from ACI or would be better to have gone for something simpler, and interoperability with NSX.
You can see my more complete response to Raoul07 but as a follow up to your question... If you are familiar with ACI and have almost exclusively virtual workloads (no bare metal) I might still use ACI as the NSX Underlay. If I knew nothing about ACI and I didn't want to shoot myself in the foot visibility wise, I might go with a EVPN solution and I'd probably be looking at a VXLAN/DCNM deployment if I needed to stay with Cisco. As I mentioned to Raoul07, its certainly doable but please think about Operations. How comfortable are you and your team today with both ACI and NSX? How easy its it going to be for a new team member to learn how to support this design. Are you now limited to hiring people who know both ACI and NSX (small pool I suspect) or having to train them on both before they can support the network? You get the idea.
I am a fan of both ACI and NSX. Having said that, I am not a fan running NSX over ACI but it is certainly doable. I wrote a white paper on this for my company a few years ago and I'll share my conclusions below. Also, let me clarify that it is actually fun to build an NSX over ACI data center but I would run screaming from operating one.
1. Yes, use ACI as your NSX underlay. Remember that you will be dealing with double encapsulation and so you will need to make sure you map everything very clearly every step of the way. The VLAN ID does not really matter..but of course knowing what you used for what is critical.
2. I've used both OSPF and BGP for the L3Out. Both will work well and so the selection depends on your overall design and your preference. What fits best in your environment.
3. This one is hard to answer without more information but if you are just using ACI as the underlay this is likely less of an issue. In general, NSX aside, and all else being equal, I recommend putting services that you may need to use across Tenants in common as it saves you a step.
Have fun and good luck!
While NSX can certainly use ACI as its underlay technology such a solution is not recommended without a clear understanding of the trade-offs and resulting complexity, particularly in an environment which must support both bare-metal and virtual workloads.
NSX over ACI will result in double encapsulation of the fabric. The ESXi encapsulation will then be encapsulated once again as it traverses the ACI fabric.
Loss of overall visibility
ACI with Server Virtualization and L4-L7 service integration provides for comprehensive management view into the data center infrastructure and services. The ACI fabric health score takes network monitoring to another level. It is not just device “up/down” but takes into account the configuration and endpoint status. The ACI controller has, out of the box, helpful operational tools to troubleshoot the environment. All of this visibility is lost with NSX running on top of ACI.
If micro segmentation at the hypervisor layer is a business requirement, NSX and only NSX can meet this requirement but that is a rare business requirement.
NSX over ACI can allow the continuation of siloed network and compute organizational structures thus impeding many (all?) of the benefits of an SDN technology. It is often the desire to maintain these two separate silos which leads to the question “Can we deploy NSX on top of ACI?”.
One use case
In a configuration where ACI is acting onlyas the underlay and transport for ESXi hosts and the fabric will support no other connections (no legacy bare metal servers, etc.) this could be a supportable environment. However, this is rare and the caveats above (double encapsulation, loss of visibility from within ACI etc.) still apply.
Both are solid and mature solutions. With so much overlap and with significant tradeoffs when using both together, an organization should focus on which one solution which best meets their overall requirements. Both solutions together do not provide “the best of both worlds” and result in a, siloed, and often highly complex and potentially fragile infrastructure which impedes operational visibility, simplicity of use and management.