Having installed the N1Kv many times in the past in a layer 2 mode and never had any issues I thought it was time to build out an environment with the N1Kv in layer 3 mode. My environment is as followed - Cisco UCS - 2 x 6248 FIs, 2 x B200 M3(esxi 5.1 hosts), a pair of Nexus 5Ks(L3) upstream connected with a VPC to each FI. I have installed the VSM manually and deployed the VEMs to the hosts using VUM which is working fine, when I run show mod I can see the VEMs, also the SVS connection is active to vCentre. My VSM MGMT0 interface and esxi hosts(VMKs) are on the same vlan (25) and they can all ping each other. The VSM is connected to the standard vswitch0 on ESXi01 and no VMs reside on ESXi02. After creating the UPLINKS vethernet port-profile and the MANAGEMENT ethernet port profile I successfully migrated the vnics and management VMK on ESXi02. At this point the VEMs are still connected and everything is happy. Now here is the strange part, after migrating the vnics on ESXi01(hosting the VSM), the vmk(ESXi01 management) and the virtual machine network for the VSM to the N1kv I loose connectivity to the VSM. Also the VSM can no longer communicate with the ESXi hosts or vCentre. I have configured vlan 25 as the system vlan for the ethernet and vethernet port-profiles and added the capability L3control command to the vethernet port-profile. As a test I created a new vethernet port-profile(
VSM_MGMT) moved the VSM management(MGMT0) to another IP subnet on vlan 27. So now both vlans are trunked to the 5ks where the inter-vlan routing take place. This worked fine and the VSM was able to communicate with the ESXi hosts and vCentre.
So my question is if the VSM management interface (MGMT0) and ESXi vmks are on the same subnet/vlan should layer 3 mode work? in this case my conclusion is no. Has anybody else found this true????
VSM and VEMs can be in the same subnet for L3 mode of connectivity.
The problem I see in your setup is usage of "capability l3control" port-profile on VSM's mgmt interface.
Since VSM's mgmt interface is also needed to be in the same subnet as of ESXi vmks, you can create another port-profile for same vlan 25 without "capability l3control" and applying it on the VSM's mgmt interface should work fine.