VMs in different VMM domain, same EPG, but could not communicate

[Trinh Nguyen]

Hi,

I have two VMM domains VMD1 and VMD2.  A Green_EPG includes these two domains as show in the picture (pic cred to ACI book)

Problem: a virtual host in VMD1 can not ping another virtual host in VMD2.  These two hosts are on the same Green_EPG, IP addresses are on the same subnet.

Host 1 IP address 10.10.10.1 shows dynamic Vlan 3670, host 2 IP address 10.10.10.2 shows dynamic Vlan 3669 (is this a problem?)

VMD1 and VMD2 share the same Vlan pool 3500-4000, DVS mode.  The fault on one of the Vcenter Domain code F013: “VMD1 due to error: Discovery Protocol on DVS is different than the one in the policy”

Something obvious in ACI configuration? I don’t know anything about the VM administrator, anything I can ask VM team to check?  ACI version 2.2(1o)

Best Regards.

[Jason Williams · Accepted solution]

In regards to the fault you observed "VMD1 due to error: Discovery Protocol on DVS is different than the one in the policy", this should mean there is a mismatch in the discovery protcol configured by the APIC and what the DVS is vCenter is actually using. 

Under the ACI GUI navigate to VM Networking -> VMware -> <vmm domain> (The VMM domain with the fault raised) 

In the right pane, scroll to the bottom and you should see vswitch policies. You should have either LLDP or CDP enabled under the v-switch policies. It's best practice to have 1 selected (not both). 

Once you've confirmed that LLDP or CDP is enabled, then log into vCenter and check the DVS settings. Go to Networking -> DVS folder -> DVS. Right click the DVS and choose edit settings. Under the settings window go to Advanced. You should see LLDP or CDP as the discovery protocol type. Choose the one which matches your ACI vswitch policies. If this change does not resolve the fault, then you could try triggering an inventory sync so that the APIC will sync with vCenter. 

In regards to your reachability between the two VMs, can both VMs ping their own gateway? 

If not, then log into the switch(es) which connect to directly to each VM and make sure the appropriate VLAN/EPG is allowed on the interface. You can use show vlan extended command to verify the VLANs allowed. 

Jason

[Trinh Nguyen]

Jason,

This is the main confusion when migrate from traditional network to ACI: same subnet, different Vlans, but still in the same broadcast domain? 

The Discovery Protocol fault was resolved with CDP sync with VM DVS.  The reach-ability from Domain 2 is another issue.  While Domain 1 is working normal in and out of the ACI, Domain 2, a new VXRack just installed, were not able to communicate.  Switches (Leaves, and VXrack switches) show all Vlans/EPG were tagged correctly.  I am getting on the case with VXRack for their support now.

Many thanks for your helps.     

[Jason Williams]

Trinh, 

"This is the main confusion when migrate from traditional network to ACI: same subnet, different Vlans, but still in the same broadcast domain?"

Yes, the bridge domain is considered the L2 domain (broadcast domain). The EPG is the policy boundary. Anything within the same EPG can communicate without policy (contract) deployed. If you were to separate the 2 VLANs into their own EPGs, then you would need contracts between the two but each VLAN/EPG use the same subnet. If they share the same subnet, then you should place them into the same bridge domain.