Infra VLAN local on Leaf different to configured Infra VLAN

[nouse4it]

Hi everybody,

I have a couple of questions regarding to understand the Infra VLAN:

1. Why do all Leafs have the same IP on their Infra-VLAN SVI?
2. Why do Spines don´t have such an Infra-VLAN SVI at all?
3. Why is the Infra-VLAN configured on the Leaf (local VLAN) different to the Infra-VLAN that was configured on the Fabric Setup? And how is it mapped to the configured Infra-VLAN?

Hope somebody here can help me to understand that?

Thanks!

Best,

Ben

[RedNectar]

Hi @nouse4it ,

Can you please give us some more context about the SVIs etc - like where are you seeing the "same IP on their infra-VLAN SVI" - give us a CLI command or menu path to show us where you see this.  

I suspect you have some confusion between the infra VLAN and something else - perhaps the pervasive SVI that every routing enabled BD has, but without more context, I can't tell.

In the meantime, here's a brief explanation of the infra VLAN.

The infra VLAN is defined during fabric setup, and is used to encapsulate traffic leaving a leaf destined to one of the following:

1. An APIC - if your infra VLAN is say 3900, you'll see there is an interface created on each APIC called bond0.3900. All management traffic between the fabric and the APIC is sent on this interface. 
2. A device that uses OPFLEX (typically a vSwitch) to communicate with the APIC
3. VXLAN traffic generated by Hypervisors (typically a Cisco AVE)

Since all of these instances describe connections to a leaf, you won't see the infra VLAN on a spine.

The infra VLAN is configured during APIC setup, never on a leaf.   This VLAN ID information is conveyed to each leaf, and each leaf will set up a forwarding VLAN to Private Internal VLAN mapping (FD_VLAN to PI_VLAN) just like every other encapsulation, so again using the example of an infra VLAN of 3900, you will see mappings on each leaf of VLAN 3900 to an PI_VLAN on each leaf, and it's possible each leaf has a different PI_VLAN.  You can see this mapping by sending the show vlan encap-id 3900 extanded command on the leaf. (The easiest place to send commands to leaf switches is from the APIC using the fabric xxxx  construction as shown below)

apic1# fabric 2201-2202 show vlan encap-id 3900 extended
----------------------------------------------------------------
 Node 2201 (Leaf2201)
----------------------------------------------------------------

 VLAN Name                             Encap            Ports
 ---- -------------------------------- ---------------- ------------------------
 7    infra:default                    vxlan-16777209,  Eth1/1
                                       vlan-3900

----------------------------------------------------------------
 Node 2202 (Leaf2202)
----------------------------------------------------------------

 VLAN Name                             Encap            Ports
 ---- -------------------------------- ---------------- ------------------------
 8    infra:default                    vxlan-16777209,  Eth1/1
                                       vlan-3900

Note that the PI VLAN on leaf 2201 is 7, but on leaf 2202 it is 8, but on both leafed the FD_VLAN is 3900 - i.e. the infra VLAN ID set up during fabric setup.

I suspect this is where your confusion lies when you ask

Why is the Infra-VLAN configured on the Leaf (local VLAN) different to the Infra-VLAN that was configured on the Fabric Setup? And how is it mapped to the configured Infra-VLAN?

[nouse4it]

Hello Chris,

first of all thank you very much for your thorough explaination.

Yes, you were right about the part with the PI-VLAN on the Leaf and the Infra VLAN of the APIC.

But what I don´t understand is, why is the Infra VLAN mapped to a PI-VLAN on the Leaf? Why is f.e. VLAN3900 not used as the Infra-VLAN on the Leafs, but instead is mapped to an PI-VLAN, that can be different on any Leaf?

Your explaination now also clears up to me why there is no Infra-VLAN SVI on the Spine But this makes me wonder now even more, how the actual configuration of spines and leafs takes place? I always thougth this type of communication is done in the Infra-VLAN?

Regarding the IP of the Infra-VLAN SVI, sorry I provided a little less context there. What I mean is, if I do a

fabric 1101-1102 show ip interface brief vrf overlay-1

I see that VLAN9 is my PI-VLAN which is mapped to the Infra-VLAN. And that SVI has the same IP on every Leaf:

I would have thougth every Leaf has it´s own IP in the Infra-VLAN, but it seems like they all have the same. But why?

Thank you again for your help!

Best

Ben

[RedNectar · Accepted solution]

Hi @nouse4it/Ben,

Ah - now I get it.

The IP address you are looking at is indeed a pervasive gateway IP address.  You'll find it configured (automatically) under Tenants > infra > Networking > Bridge Domains > default > Subnets

And just like every other IP address defined on a BD, this same IP gets pushed to every leaf as required.

Now, TBH, I have no idea when an attached device might send traffic to this IP - I suspect that it might be used when Cisco proprietary virtual switch (AVE) has been deployed, or maybe when devices use OPFLEX to communicate with "whatever leaf switch its connected to".  Although I also note there is a pre-defined BD called ave-ctrl which may get assigned an IP if using AVE.

Now to your other questions:

---

why is the Infra VLAN mapped to a PI-VLAN on the Leaf? Why is f.e. VLAN3900 not used as the Infra-VLAN on the Leafs, but instead is mapped to an PI-VLAN, that can be different on any Leaf?

I can only guess at the reasoning, but I suspect that it was just easier to do the internal mapping like every other IP on the switch.  And back in first generation switches, there actually was a physical separation between two chips (one Broadcom, one Cisco) that had to be accommodated too.

Your explaination now also clears up to me why there is no Infra-VLAN SVI on the Spine But this makes me wonder now even more, how the actual configuration of spines and leafs takes place? I always thougth this type of communication is done in the Infra-VLAN?

---

I also thought the infra VLAN was used for leaf-to-spine communications for years (I may have even stated this somewhere). But I was wrong - and I discovered this when I did some packet captures on leaf-to-spine interfaces - I've just looked for the capture but can't find it, and don't have time to to do a new one ATM.

I hope this helps.

---

Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem.

---

[nouse4it]

Thank you Chris, that indeed helps me a lot