Unable to connect to endpoints

[roysm]

Hi

I now have a bit of a problem with something that was working last week but is not working now. I am now unable to connect to any of our VMs. I have removed all related EPGs, BDs and VMM domain and recreated everything from scratch with no effect. I feel there is something I'm missing somewhere or I've come across some bug. 

We have an EPG with a VMM domain asociation. I have added the subnet in the EPG. I have added provide and consume contracts using the common/default for both. 

From outside ACI, I can ping the EPG gateway IP but I cannot ping the VMs in the EPG. The VMs in the EPG can ping each other but cannot ping their default gateway (i.e. the EPG subnet IP).

From the VM i  get the message 

          Reply from X.X.X.X: Destination host unreachable (where X.X.X.X is the IP of the VM)

On the BD, ARP flooding is disable, with Hardware Proxy set. I know the VRF is routing, as there are other EPGs to physical devices working fine. On the EPG, Operational tab, I see the VMs listed as the learned endpoints. 

On the contracts tab, I do not see any traffic, so I'm wondering if a contract/acl has got messed up somewhere. Can anyone suggest any troubleshooting I can check? 

For info, we have ACI v.2.3(1f) and VMWare 6.5.

Thanks in advance

Roy

[gmonroy]

Roy Smith,

    If the Endpoints are unable to reach the Gateway programmed in ACI, then that likely means that there is some config missing/issue with the VMM domain to EPG assignment. This may be causing the VLAN to not get programmed on the Front Panel interface so that the EP can be classified/let into the fabric/reach the gateway. 

Whether or not the VLAN gets programmed depends on the Access policies in conjunction with the Resolution immediacy set on the VMM Domain to EPG assignment:

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/aci-fundamentals/b_ACI-Fundamentals/b_ACI-Fundamentals_chapter_01011.html#concept_EF87ADDAD4EF47BDA741EC6EFDAECBBD

The other point to note would be to check if you see any faults on the EPG or within the VMM domain itself. Those may help pinpoint the issue.

-Gabriel

[roysm]

Hi Gabriel

I am not seeing any faults anywhere within EPG or anywhere in the fabric. 

I have tried setting deployment and resolution immediacy to immediate and On Demand, with no difference. I have even removed the VMM association and added it back, again with no difference. 

On the Leafs connected to our UCS devices, when I look at the zoning-rules, I can see the contracts being shown, so I assume this means they are being applied to the switches? 

Thanks

Roy

[gmonroy · Accepted solution]

Roy,

    The resolution Immediacy of both "Immediate" and "On Demand" are similar in that they both require some neighbor relation in conjunction with Access Policies to determine:

• When to program the VLAN
• Where to program the VLAN

This is information is highlight in the document I sent over previously:

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/virtualization/b_ACI_Virtualization_Guide_1_3_x/b_ACI_Virtualization_Guide_1_3_x_chapter_010.html#concept_C3EF1E36E8F6406A96A71E605AC175BD

With that said, can you clarify the following:

1. What is the server model you are trying to integrate? And are there any intermediate switches/device between the host and the ACI leaf? (For example, UCSB has FIs)
2. What vlan are you trying to use for this integration.
   1. If using Dynaimc, you can verify this from the VMM networking tab to see what VLAN is being assigned to the port group.
   2. On the directly connected leaves, run "show vlan extended" to see if that vlan exists, what local VLAN it is mapped to, and what interfaces it is on.
3. If there are any intermediate devices, have we had a chance to verify the MAC tables on them to see that the VMs in question are traversing on the expected VLAN, and that the VLAN is forwarding on said intermediate device?
   1. Example reference for UCSB scenario:
   2. https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/ACI_Best_Practices/b_ACI_Best_Practices/b_ACI_Best_Practices_chapter_0101.html#d17423e23a1635

-Gabriel

[roysm]

Gabriel

I went through your troubleshooting steps. When I logged in to the CLI on our UCS FI devices, I noticed the vlans were not being shown on the vethernet interfaces. I queried this with our Server guys and it transpires that one of them had been modifying various service templates, including the vnic templates. Therefore the vlan pool assigned for the VMs was not being trunked to the ESX servers. 

So, once we added the vlans back on the vnic templates, everything started to work again. 

Fortunately, for them, we do not have live VMs running yet! Although all this work is in preparation for the first live VMs getting set up next week. 

Thanks for the help. 

Roy