VMM problem: VM's can't communicate between ESXi hosts/different EPG's

[Philip Schmid]

Hi guys,

I set up a small testing Cisco ACI and a VMware environment. Unfortunately I’m not able to communicate between two VM’s.

web01: static IP 10.18.8.50/24 -> Network: BLABLA|MyTwoTierApp|web

db01: static IP 10.18.8.30/24 -> Network: BLABLA|MyTwoTierApp|web

 

On the APIC web GUI I can see that the ESXi hosts seems to be connected properly:

  

I’m able to ping from one VM to the other, if both VM’s run on the same ESXi host and are placed in the same distributed port group (e.g. BLABLA|MyTwoTierApp|web). But if I place one VM in the other distributed port group (on the same ESXi host) they are not able to ping each other anymore. A regarding contract is defined and the EPG’s are assigned to it:

 

Also I’m not sure what exactly I should see on the two leaf switches (esxi01 is connected to leaf01 Eth1/1 and esxi02 to leaf02 Eth1/1). Currently I see the following:

acile01# show vlan
 VLAN Name                             Status    Ports                        
 ---- -------------------------------- --------- -------------------------------
 7    infra:default                    active    Eth1/48
 8    BLABLA:aciTestBD2                   active    --     
 
 VLAN Type  Vlan-mode 
 ---- ----- ----------
 7    enet  CE        
 8    enet  CE        
acile01# show endpoint detail
Legend:
 O - peer-attached    H - vtep             a - locally-aged     S - static         
 V - vpc-attached     p - peer-aged        L - local            M - span           
 s - static-arp       B - bounce         
+-----------------------------------+---------------+-----------------+--------------+-------------+------------------------------+
      VLAN/                           Encap           MAC Address       MAC Info/       Interface     Endpoint Group
      Domain                          VLAN            IP Address        IP Info                       Info
+-----------------------------------+---------------+-----------------+--------------+-------------+------------------------------+
overlay-1                                                 10.18.5.253 L                         lo0

 
acile02# show vlan
 
 VLAN Name                             Status    Ports                           
 ---- -------------------------------- --------- -------------------------------
 7    infra:default                    active    Eth1/48
 8    BLABLA:aciTestBD2                   active    --     
 
 VLAN Type  Vlan-mode 
 ---- ----- ----------
 7    enet  CE        
 8    enet  CE        
acile02# show endpoint detail
Legend:
 O - peer-attached    H - vtep             a - locally-aged     S - static         
 V - vpc-attached     p - peer-aged        L - local            M - span           
 s - static-arp       B - bounce         
+-----------------------------------+---------------+-----------------+--------------+-------------+------------------------------+
      VLAN/                           Encap           MAC Address       MAC Info/       Interface     Endpoint Group
      Domain                          VLAN            IP Address        IP Info                       Info
+-----------------------------------+---------------+-----------------+--------------+-------------+------------------------------+
overlay-1                                                 10.18.5.255 L                         lo0
7/overlay-1                          vxlan-16777209    58ac.78f2.2e0f L                     eth1/48                  infra:default

 Is there any other commend which is usefull to debug such problems?

Does anyone from you guys see where the problem could be? Any hint is welcome.

Thank you.

Regards, 

Philip

[gmonroy]

Hello Philip,

There are a variety of things I could think to check for, but it may be a bit much to write here. What I can say, is the best place to start would be to check what faults, if any, exist under either EPG and the associated VMM domain.

To summarize your findings above you found:

1. VM1 Web EPG  >  VM2 Web EPG  =  WORKS

2. VM1 Web EPG  >  VM2 Db EPG  =  FAILS

3. VM1 Db EPG  >  VM2 Db EPG  =  ???

Another thing you can always check is the first point of entry into the fabric; its gateway reachability. Assuming the gateway for these two VMs is an SVI (subnet) defined on the BD, please perform ping tests to see if they are always able to reach their gateways given the above scenarios.

Finally, depending on the host configuration and anything in the path towards the leaf (example, UCS-B has fabric interconnects), there could be a variety of other configurations preventing this from working properly (such as not allowing vlan 501 on the path, only vlan 500).

If the above is unable to assist you in diagnosing your setup, I would recommend opening a TAC case for further troubleshooting.

Cheers,

-Gabriel

[Philip Schmid]

Thanks to the answer.

1. VM1 Web EPG  >  VM2 Web EPG  =  works (same ESXi host)

2. VM1 Db EPG  >  VM2 Db EPG  =  works (same ESXi host)

3. VM1 Web EPG  >  VM2 Web EPG  =  FAILS (NOT same ESXi host)

4. VM1 Web EPG  >  VM2 Db EPG  =  FAILS (same ESXi host)

5. VM1 Web EPG  >  VM2 Db EPG  =  FAILS (NOT same ESXi host)

Gateway reachability: I configured 10.18.8.1/24 on the aciTestDB2 (Scope: Primary to VRF). I'm not able to ping 10.18.8.1 on both VM's (the VM's have 10.18.8.1 set as default GW).

ESXi host (Fujitsu workstations) connection to ACI environment: Directly connected to leafs on Eth1/1. So there is nothing in between ACI leafs and ESXi hosts, what could block VLAN 50X.

Regards,

Philip

[lpember]

Hi Philip,

Have you confirmed that the VLANs for the EPGs are deployed going to your ESXi hosts? You can confirm with the 'show vlan extended' command on the leafs? Also, per Gabe's suggestion, are there any faults under the EPGs in question?

[Robert Burns · Accepted solution]

Those CLI commands are helpful.  You mentioned above this is the VMware DVS, so this would be in VLAN mode.  From the CLI output we see the BD has been programmed, but we don't see the EPG or any active interfaces.

Should look something like this:

Leaf1# show vlan

 VLAN Name                             Status    Ports
 ---- -------------------------------- --------- -------------------------------
 13   infra:default                    active    Eth1/2, Eth1/15, Eth1/16, Po1,
                                                 Po2
 15   common:default                   active    Eth1/15, Eth1/16, Po1, Po2
 16   common:default:VMotion           active    Eth1/15, Eth1/16, Po1, Po2
 17   common:default:Management-176    active    Eth1/15, Eth1/16, Po1, Po2
 18   roberbur:bd1                     active    Eth1/15, Eth1/16, Po1, Po2
 19   roberbur:ProjectExodus:Mars      active    Eth1/15, Eth1/16, Po1, Po2
 20   roberbur:ProjectExodus:Mercury   active    Eth1/15, Eth1/16, Po1, Po2
 21   roberbur:ProjectExodus:Venus     active    Eth1/15, Eth1/16, Po1, Po2

Might want to check the AEP and Interface Policies.

Robert

[Philip Schmid]

Hi Robert, 

thanks for your answer. You hint was leading into the solution. The problem was that the interface policy group was missing the VMM AEP...

Now I can see the EPGs on the leafs and can ping from VM to VM/GW.

Thank you very much!

Regards,

Philip