Leaf to UCS trunk ports not working, no VMM integration.

[Joshua Glenn]

Hello!

Key:

Vlan 200 - 20.1.1.0  all /24  

Vlan 201 - 20.1.2.0

Vlan 202 - 20.1.3.0

Vlan 203 - 20.1.4.0

Vlan 204 - 20.1.5.0

Just migrated some UCS, HCI SuperMicro and physical servers over from 5ks/fex to new 9k spine/leafs running Cisco ACI.  Worked fine on the 5ks, access ports where needed, trunks allowing VLANs 200-204 going to the UCS and SuperMicro stuff.  

*** Before we continue, note I am currently NOT doing any VMM integration, just trying to get the same trunk ports allowing the above VLAN range on leafs ports. ***

So, the physical servers and other access ports work perfectly.  The trunk links to the UCS and SuperMicro don't seem to work right.  Focusing on just the UCS, I can see the ARP entry for the UCS itself, but no resolved IP address.  Also, I do not see any of the MACs/IPs of the blade servers, or anything else.

For the EPG static bindings, I've tried tagged (assuming right) and untagged.  Different encap vlans (currently vlan-203) but nothing seems to work.  UCS side has both LLDP and CDP active, as do I.  

Speaking of VLANs, I built 5 EPGs for this, each matching a function of one of the 5 VLANs above.  However, the trunk links to the UCS need to allow all VLANs 200-204.  I have a jump server on 20.1.2.5 that cannot call the 20.1.2.x IPs which also live in the UCS, because there's some kind of disconnect to where the APIC cannot see down the trunk links at all.  I'm not sure if it's EPG related or Fabric config related.

I would be very happy to provide additional information since I'm dead in the water.  Many thanks.

Josh

[Joshua Glenn]

Also FYI,

The subnets are all on the bridge domain and shared by all the EPGs.  The EPGs are set to talk to each other on all ports.  From the access ports, I can successfully reach any subnet.

[Robert Burns]

Joshua,

Did you create a VLAN domain (physical)? It would be something like "UCS-VLAN-Domain" and point to the VLAN pool containing VLANs 200-204. You will need to do this and attach the physical domain to each EPG you plan on creating a static path binding with.  Without this, you will get faults.

Sounds like your VLANs aren't being programmed on the Leafs correctly.  This is usually a mis-config between the logical & concrete model.   You can check the VLAN programming on the Leaf by logging into the Leaf CLI (SSH) and running the following command:

176-Leaf1# show vlan extended

 VLAN Name                             Status    Ports
 ---- -------------------------------- --------- -------------------------------
 13   infra:default                    active    Eth1/2, Eth1/15, Eth1/16, Po1,
                                                 Po2
 16   common:default                   active    Eth1/15, Eth1/16, Po1, Po2
 17   common:default:Management-176    active    Eth1/15, Eth1/16, Po1, Po2  << My UCS facing interfaces
 18   common:default:VMotion           active    Eth1/15, Eth1/16, Po1, Po2
 19   roberbur:bd1                     active    Eth1/15, Eth1/16, Po1, Po2
 20   roberbur:ProjectExodus:Mars      active    Eth1/15, Eth1/16, Po1, Po2
 22   roberbur:ProjectExodus:Venus     active    Eth1/15, Eth1/16, Po1, Po2

 VLAN Type  Vlan-mode  Encap
 ---- ----- ---------- -------------------------------
 13   enet  CE         vxlan-16777209, vlan-4093
 16   enet  CE         vxlan-14811120
 17   enet  CE         vlan-176   <<< Goes to a UCS static path binding, non VMM-integrated.
 18   enet  CE         vxlan-8650752
 19   enet  CE         vxlan-16744306
 20   enet  CE         vxlan-8617984
 22   enet  CE         vxlan-8716288
176-Leaf1#

Don't mind all the VXLAN VNIDs, I'm also running AVS on UCS in addition to static path bindings without VMM integration.  There are two VLANs shown on each row of the lower output.  The first column is the "system" VLAN which is locally significant on each Leaf and maps up to your EPG.  The far right column shows the "encap" or "wire" VLAN which will be the same on all Leafs.  This should match your VLANs 200-204 on your UCS facing interfaces.   If you don't see the VLANs listed on the UCS interfaces, that's a config issue.   Try the fix above and see if that resolves your issue. 

Robert

[Joshua Glenn]

Robert,

Thank you for the quick reply!

Based on my output below:

Leaf-1-Lab# show vlan extended

VLAN Name Status Ports
---- -------------------------------- --------- -------------------------------
13 infra:default active --
14 Europe-POC:EU-POC-DB active Eth1/5, Eth1/6, Eth1/10,
Eth1/18, Eth1/19, Eth1/26,
Eth1/32
25 Europe-POC:EU-POC-AppProfile active Eth1/5, Eth1/26
:iSCSI-Storage
26 Europe-POC:EU-POC-AppProfile active Eth1/10
:Server-Infra
27 Europe-POC:EU-POC-AppProfile active Eth1/6, Eth1/32 (UCS)
:Host-Mgmt
28 Europe-POC:EU-POC-AppProfile active Eth1/18, Eth1/19
:IPMI-Network

VLAN Type Vlan-mode Encap
---- ----- ---------- -------------------------------
13 enet CE vxlan-16777209, vlan-4093
14 enet CE vxlan-16121791
25 enet CE vlan-200
26 enet CE vlan-203
27 enet CE vlan-201 (UCS)
28 enet CE vlan-204

++++++++++++++++++++++++++++++++++++++++++++++

I can see the VLAN encap/association seems to match, and I can see MAC/IP addresses:

Leaf-1-Lab# show endpoint | grep eth1/32
27/Europe-POC:EU-POC vlan-201 547f.ee13.db08 L eth1/32
27 vlan-201 0025.b5ca.501a L eth1/32

Europe-POC:EU-POC vlan-201 20.1.2.100 L
27 vlan-201 0050.566f.d7d2 L eth1/32

Europe-POC:EU-POC  vlan-201 20.1.2.102 L
27 vlan-201 0050.566c.d4d7 L eth1/32

Europe-POC:EU-POC   vlan-201   20.1.2.101 L
27 vlan-201 0050.5661.8c2b L eth1/32

Europe-POC:EU-POC   vlan-201  20.1.2.103 L

+++++++++++++++++++++++++++++++++++++++++++++

BUT:

1. Those endpoints cannot ping their gateway 20.1.2.254, nor can working physical servers in the same subnet ping them.

2. I'm concerned because other subnets/VLANs should also be visible through these ports too, ex. VLAN 203 (10.1.4.x network) but I don't see them.

3. To get this far, I had to use static bindings (paths) on the EPGs, so each EPG has configured, for example, Eth1/32 as UNTAGGED, with an Encap respective to that subnet.  This may be where i'm shooting myself in the foot if it's supposed to be trunk.  What do your static bindings look like?

Thanks!

Josh

[Robert Burns]

So the Leafs are programmed correctly - great.  Is the GW on the fabric (BD SVI subnet) or outside the fabric?  If the GW is off the fabric, you need to enable Flooding on the BD.  If the GW is the BD SVI, this is not required.

1.  A good test is to create an subnet SVI on the BD/EPG level and see if you can reach (ping) that from your UCS hosts.  You should be able to.  The only EPG that appears to be "Correctly" static path bound is "Europe-POC:EU-POC-AppProfile" on VLAN 201.  Notice no other VLANs showing the UCS facing interfaces.

2.  Check static path binding on the relevant EPGs.  The VLAN 201 is correctly bound, ensure you copy what you configured there to your other EPGs.  Ensure you're using "tagged" mode, and not 802.1p or native (unless using Native VLAN on UCS uplinks also).

3. Yeah you're shooting yourself in the foot here with the untagged mode.  If you use "untagged mode" there can't be ANY other VLANs using this port.   With UCS we "typically" trunk all VLANs from the FI's northbound.  I really discourage using native VLANs myself, but if you have one VLAN from UCS uplinks that is untagged, that EPG using that native VLAN would be static path bound in mode "802.1p".  Untagged mode would be the equivalent to an "access port".

As a reference

Tagged = 802.1q trunked VLAN.  Traffic will be tagged on egress with encap VLAN. (can support many static paths using different VLANs to same interface)

802.1p = "switchport trunk native vlan xxx". Traffic will be untagged on egress.  Only one VLAN can be untagged, but other tagged VLANs also supported on same interface.

untagged = access port.  Traffic will be untagged on egress. No other VLAN can be statically bound to this interface

Robert

[Joshua Glenn]

Hey Robert,

Starting to make more sense.  The trunking makes sense, but say, with VLAN 201 (20.1.2.0) I have IPs that live on the UCS AND other physical servers which use the same IP space.  In the EPG Static Bindings, it forces me to use either all Trunks or all Access Untagged.  So maybe I can get the UCS trunks working, but then the other physical servers are going to lose connectivity.  

Probably a design flaw on my part.

[Robert Burns · Accepted solution]

It depends if the non-UCS hosts are on the same Leaf or not.  You can map an EPG to different ports with some tagged, and others not.  There are limitations when you want to have multiple ports on the same leaf as untagged however.  A single VLAN ID can't be tagged & untagged on the same Leaf. Even when we set a Static path as "untagged, or 802.1p", we still need to assign a unique VLAN to that traffic (can't also be VLAN 201).  This is really an arbitrary VLAN for the fabric only.  All traffic egress or ingress from the connected device would send/receive traffic without a VLAN tag.

Ex. Let's say you have UCS connected on port 1, and a baremetal on port 2 (same Leaf).  You can set EPG-A to "tagged vlan-201" for the UCS ports, and "untagged vlan-205" for the baremetal connected ports.  Even though the encaps are different, it doesn't matter.  Both ports will land in the same EPG at the end of the day and be able to communicate.

Make sense?

Robert

Here's what the config would look like. **Note my VLAN pool extends from 200-205, since ACI needs to allocate a VLAN from the pool specifically for the untagged ports (system requirement).  Since your hosts are directly connected to the fabric, VLANs are arbitrary.  We just want them to land in the same EPG.