ACI Virtual Switch with UCS FI and ESX 5.5 in Native VXLAN Mode with Local Switching

[petar.forai1]

Hi,

We're playing around with AVS on blades within an UCS chassis. The UCS chassis is connected from it's IOM to the FIs and both FIs have VPC configured to two of the leaves of the ACI fabric. vCenter is running on a physical hyperV domain with an VPC to it's two ports. The management vmks of the ESX and the vcenter running on the hyperV host can all talk to each other (same EPG within mgmt tenant and that has a contract with the inband builtin EPG)

The ACI fabric is configured for inband mgmnt access and so far we can easily reach all the same subnet and gateway for the inband interface and the APIC itself from all those EPs within the same subnet on the inb VRF of the management tenant . 

We're hitting a road block with the VMM integration via VXLAN and AVS with local switching.

The AEP for the UCS was configured as bare metal with the infrastructure VLAN extended to the FIs (in addition to the storage ports and management ports both tagged and encapsed as vlan-363 and vlan-350). We've deployed the AVS software and the AVS is visible within the vcenter. 

Now we're not sure if we need to remove the AEP for UCS as for bare metal and just use a VXLAN host AEP or if we can even define a second AEP on the same ports with VXLAN ESX.

We've configured the DHCP and the DHCP relay labels on the subnet to the APIC's DHCP server for the infrastructure VLAN in order for the vleaf to talk to the ACI's leafs, but we can't see if those interfaces are actually getting an IP or not or where they (vteps on AVS?)

So a couple of questions:

• How should the AEP for an UCS FI be configured if a AVS is going to be configured behind the FI? We need local encaps vlan-350 and vlan-363  in addition to the ACI infra VLAN for reaching the ESX management and where the vcenter and APIC can talk to each other for inband management and NFS traffic to ESX datastores.
• What about the infra VLAN and the FI config? our current AEP pushes that VLAN (4093) to the FI but we didn't configure that VLAN on the UCS blade vNICs - neither as native nor an additional vmk interface for that
• How to verify that the AVS has connectivity to the infrastructure VLAN to the spine/leafs and the APIC? (Where is that interfaces actually?)
• The documentation has confusing info about LLDP and CDP config to the FI and from the FI to the dvSwich (CDP to FI and LLDP to vSwitch according to virtualization guide for ACI 1.1(2j))

Thanks

P

[dpita · Accepted solution]

Thanks for the great diagram and the text!

Ill answer your questions first then have a discussion. 

1. AEPs are a curious object. i like to use AEPs to group a particular type of endpoint. For example, my UCS C series use a single AEP. My UCSB uses a different AAEP. My 2600 router uses its own AEP and my n3k router uses a different AEP. etc. To your specific question, i use one AEP for both FI's since its one chassis that is behind that. The same AEP is tied to the two different interface policy groups for the vPC's down to the FIs though. The AEP will have infra vlan checked and the dynamic vlan pool will have vlan 350, 363. 
2. The UCS blade vNICs definitely need the infra vlan configured on them. That is how the hypervisor will be able to forward traffic from ACI to the vmk port and finally form the vxlan tunnel. 
3. Once the vmk on the AVS receives a DHCP address form the infra address space the tunnel is more than likely built. You can SSH to the hypervisor and run the following command "vemcmd show openflex" and make sure there is a status of 12 (active). From the leaf that has the UCS-FI connected, you can explore the VM Networking > Inventory tab for alot of information and faults
4. This last question is a good one and is dependant on the version of UCSM you are running. Normally, CDP was the only option to get the adjacencies between ACI and the hypervisory working. You needed to configure a network control policy for the vnic as well to allow CDP from the vNIC to the FI. ACI then stiches the information from the hypervisor adjacency (should be a vEth) and then the FI to the leaf to from a complete path. On newer versions of UCSM (2.2.4b if im not mistaken) LLDP is finally supported. Regarding the Vswitch, it doesnt really need to form adjacencies since the configuration requires a static path on the ACI side instead of the dynamic path used by the dvSwitch. 

I hope i answered most of your questions. Please let me know if there are any other questions i can help answer. Have a nice day, and thank you for using Support Forums. 

[gjimenez]

Hello.

I found your discussion very interesting, and now I have a question related to OpenFlex.

What happens with Status 5? I have a situation where the AVS is installed and the vmk1 gets an IP address, but when I run the command "vemcmd show openflex" the result is 5.

I already tried to find tshoot recommendations for is very basic.

I hope you can provide more clarification on this status.

Greetings