ACI Questions

[Nitesh_A]

Hi All,

I got interviewed couple of days ago on ACI and few questions i couldn't answer. Could someone please answer my query.

Q1.

HW_vlan --> I know Leaf Switch comprises of a Broadcom ASIC that is utilised when traffic is meant to be locally switched. For eg switch knows source & dest mac, then traffic can be switched directly to that egress port. (like traditional switching), then what is the use of HW vlan here.

PI vlan --> allocates PI VLAN per EPG, Per BD and these allocation is local to leaf and is different to each Leaf. Does this mean this vlan is only used for classification of EPG/BD. If yes how does this vlan plays role in classification of EPG/BD

BD_vlan --> Known as Bridge Domain VLAN, it can map to a single or multiple encap (FD) VLANs and VXLANs. On this VLAN no direct learning occurs. How exactly it works

Q2. Do we need route reflector if there are no L3 outs?

Could someone please answer Q1 in layman's term. 

[RedNectar · Accepted solution]

Hi @Nitesh_A ,

Let's deal with Q2 first - the answer is a simple "No" - BGP is used only to convey the routes learned from external routers to other leaf switches - using MB-BGP

Great question for an interview!

But Q1.  I would have had a bit of a spew about the wording they've used.  It would seem they are targeting knowledge of a pretty specific rather obscure - even OBSOLETE command. More on that later. (My suspicion is that the person who interviewed you went through hell implementing ACI on 1st generation switches)

But back to your Q.

Traffic arrives at a switch carrying a VLAN tag - that tag is the FD VLAN, or I prefer to call it the user VLAN - the one that the user assigned. Curiously, the FD VLAN was not in the Q. But let's tell the WHOLE story.

Now is ACI is to be able to scale, there needs to be a way of using the same 802.1Q VLAN tag to represent multiple EPGs, to to make this possible, every leaf switch allocates an Internal (sometime called canonical) VLAN, or PI VLAN to every FD VLAN.

You can see these mappings using the fabric xxxx show vlan extended command from the APIC (where xxxx is id of the leaf switches you wish to view).

Now there is not strictly a 1 to 1 mapping of PI VLAN to EPGs - as you said above.  It is quite possible to have more than one 802.1Q encapsulation mapping to one VLAN, perhaps some bare metal hosts using one VLAN, and a different dynamically allocated VLAN for VMs.  Here's an example from my lab.  Note particularly the WebServer_EPG

apic1# fabric 1201-1202 show vlan extended 
----------------------------------------------------------------
 Node 1201 (Leaf1201)
----------------------------------------------------------------

 VLAN Name                             Encap            Ports
 ---- -------------------------------- ---------------- ------------------------
 1    Tenant18:2Tier_AP:AppServers_EPG vlan-1183        Eth1/28, Eth1/48,
 74   Tenant18:Production_VRF:l3out-   vxlan-15335346,  Eth1/10
 104  Tenant18:Web_BD                  vxlan-16613256   Eth1/48, Eth191/1/28,
 105  Tenant18:2Tier_AP:WebServers_EPG vlan-1386        Eth191/1/28
 106  Tenant18:App_BD                  vxlan-16056263   Eth1/28, Eth1/48,
 107  Tenant18:2Tier_AP:DBServers_EPG  vlan-1383        Eth191/1/28
 109  Tenant18:2Tier_AP:AppServers_EPG vlan-1181        Eth1/28
 110  Tenant18:2Tier_AP:WebServers_EPG vlan-1184        Eth1/48, Po3

----------------------------------------------------------------
 Node 1202 (Leaf1202)
----------------------------------------------------------------

 VLAN Name                             Encap            Ports
 ---- -------------------------------- ---------------- ------------------------
 72   Tenant18:Web_BD                  vxlan-16613256   Eth1/28, Eth1/48,
 73   Tenant18:2Tier_AP:WebServers_EPG vlan-1386        Eth1/28
 75   Tenant18:2Tier_AP:WebServers_EPG vlan-1182        Eth192/1/28
 76   Tenant18:2Tier_AP:WebServers_EPG vlan-1184        Eth1/48, Po22
 77   Tenant18:App_BD                  vxlan-16056263   Eth1/28, Eth1/48,
 78   Tenant18:2Tier_AP:AppServers_EPG vlan-1183        Eth1/28, Eth1/48,

Notice in the output above that the PI VLAN mappings (that's the first column) for the WebServers_EPG are different on each switch.  The FD VLAN is the VLAN listed under the encap column.

And also notice there is also a VLAN allocated for each BD.  Which brings us to the discussion of the 3rd item of your Q.  And for that, you need to understand that a BD has two functions in ACI

1. It provides a broadcast domain, much like a VLAN in traditional networking, except the VLAN ID of the broadcast domain is going to be a different VLAN in each switch (for the scaling reasons mentioned above)
2. It provides an anchor for an IP address - an SVI for the default gateway IP associated with the BD

So, for example, note the following two entries for the Web_BD from above.

apic1# fabric 1201-1202 show vlan extended 
----------------------------------------------------------------
 Node 1201 (Leaf1201)
----------------------------------------------------------------

 VLAN Name                             Encap            Ports
 ---- -------------------------------- ---------------- ------------------------
 1    Tenant18:2Tier_AP:AppServers_EPG vlan-1183        Eth1/28, Eth1/48,
 74   Tenant18:Production_VRF:l3out-   vxlan-15335346,  Eth1/10
 104  Tenant18:Web_BD                  vxlan-16613256   Eth1/48, Eth191/1/28,
 105  Tenant18:2Tier_AP:WebServers_EPG vlan-1386        Eth191/1/28
 106  Tenant18:App_BD                  vxlan-16056263   Eth1/28, Eth1/48,
 107  Tenant18:2Tier_AP:DBServers_EPG  vlan-1383        Eth191/1/28
 109  Tenant18:2Tier_AP:AppServers_EPG vlan-1181        Eth1/28
 110  Tenant18:2Tier_AP:WebServers_EPG vlan-1184        Eth1/48, Po3

----------------------------------------------------------------
 Node 1202 (Leaf1202)
----------------------------------------------------------------

 VLAN Name                             Encap            Ports
 ---- -------------------------------- ---------------- ------------------------
 72   Tenant18:Web_BD                  vxlan-16613256   Eth1/28, Eth1/48,
 73   Tenant18:2Tier_AP:WebServers_EPG vlan-1386        Eth1/28
 75   Tenant18:2Tier_AP:WebServers_EPG vlan-1182        Eth192/1/28
 76   Tenant18:2Tier_AP:WebServers_EPG vlan-1184        Eth1/48, Po22
 77   Tenant18:App_BD                  vxlan-16056263   Eth1/28, Eth1/48,
 78   Tenant18:2Tier_AP:AppServers_EPG vlan-1183        Eth1/28, Eth1/48,

Notice that

• the VLAN ID is different in each switch, but
• the VXLAN ID is the same for both switches

this is what enables a switch, which receives a multicast or broadcast on one switch to be able to flood it on another switch.

But also look at the following output (truncated for brevity):

apic1# fabric 1201-1202 show ip interface brief vrf Tenant18:Production_VRF
----------------------------------------------------------------
 Node 1201 (Leaf1201)
----------------------------------------------------------------
IP Interface Status for VRF "Tenant18:Production_VRF"(227)
Interface            Address              Interface Status
vlan104              10.118.12.1/24       protocol-up/link-up/admin-up


----------------------------------------------------------------
 Node 1202 (Leaf1202)
----------------------------------------------------------------
IP Interface Status for VRF "Tenant18:Production_VRF"(16)
Interface            Address              Interface Status
vlan72               10.118.12.1/24       protocol-up/link-up/admin-up

Note how the same VLANs used to define the broadcast domain on each switch (vlan104 and vlan72) also hold the IP addresses for the gateway IP address for that BD.

So, hopefully that has covered the PI VLAN, the BD VLAN, and as a bonus, the FD VLAN

---

And finally - back to the obscure HW VLAN

To answer it in the way they probably expected, you'd need to understand the rather obscure command show system internal eltmc info vlan brief 

But this command is not entered at the APIC.  Oh no.  It's not even entered at the switch level.  You have to dig down line-card level to issue this!  I'll use my leaf1202 for illustration

apic1# ssh leaf1202
<snip>
Leaf1202# vsh_lc
<snip>
module-1# show system internal eltmc info vlan brief
<big selective snip>
VlanId  HW_VlanId               Type  Access_enc_type Access_enc Access_in_enc_type Access_in_enc  Fabric_enc_type Fabric_enc  BDVlan
=====================================================================================================================================
73           30            FD_VLAN           802.1q       1386          Unknown          0            VXLAN      10398         72
75           28            FD_VLAN           802.1q       1182          Unknown          0            VXLAN      12394         72
76           42            FD_VLAN           802.1q       1184          Unknown          0            VXLAN      12396         72

Note particularly the HW_VlanId column.

And to save you scrolling back, let me repeat section of an earlier section, highlighting the relevant output for leaf 1202

apic1# fabric 1202 show vlan extended 

----------------------------------------------------------------
 Node 1202 (Leaf1202)
----------------------------------------------------------------

 VLAN Name                             Encap            Ports
 ---- -------------------------------- ---------------- ------------------------
 72   Tenant18:Web_BD                  vxlan-16613256   Eth1/28, Eth1/48,
 73   Tenant18:2Tier_AP:WebServers_EPG vlan-1386        Eth1/28
 75   Tenant18:2Tier_AP:WebServers_EPG vlan-1182        Eth192/1/28
 76   Tenant18:2Tier_AP:WebServers_EPG vlan-1184        Eth1/48, Po22
 77   Tenant18:App_BD                  vxlan-16056263   Eth1/28, Eth1/48,
 78   Tenant18:2Tier_AP:AppServers_EPG vlan-1183        Eth1/28, Eth1/48,

Notice that the relationship between the VlanId and the Access_enc of the line card with VLAN and Encap of the leaf.

So what in the hell is the HW_VlanId column showing in the line card output?

Well, to see that, you have to dig EVEN DEEPER. Honestly, unless you are looking for a job with Cisco TAC, you shouldn't need to dive this deep. (I don't think you CAN dive that deep without assistance from the CISCO TAC).  Definitely beyond what could be expected for a normal person looking for a normal ACI career.

BUT the short story is that there is yet another mapping between the FD VLANs (and BD VLANs - but I didn't show that) and the silicon ASIC chips, similar to the to the FD VLANs and the PI VLANs.  I have a sneaking suspicion that this harks back to the days when Cisco used Broadcom chips to do the heavy lifting high-speed switching, and is only there today as a hang-over.  I hope someone with better knowledge than I can confirm or clarify that bit.

I've attached BRKACI-2101 and BRKACI-2102 from 2017 (6 years ago!) which I had to dig out to even answer this question. They may help. 

[Nitesh_A]

@RedNectar Thankyou so much for taking your time out and giving this detailed explanation. I have gone through this post & watched cisco live videos and seems doubts are getting cleared. 

Now is ACI is to be able to scale, there needs to be a way of using the same 802.1Q VLAN tag to represent multiple EPGs, to to make this possible, every leaf switch allocates an Internal (sometime called canonical) VLAN, or PI VLAN to every FD VLAN. :  Understood, like the one shown in below snap. Also in EPG we may have more than one access encap vlan & it could be vxlan as shown below.

And also notice there is also a VLAN allocated for each BD.  Which brings us to the discussion of the 3rd item of your Q.  And for that, you need to understand that a BD has two functions in ACI : Understood. So we have one PI vlan for BD just to create SVI. This is just to route from one vlan to another (In ACI it is pervasive gateway)

Q. As we can see from above snap, Although both leaf switches has different PI vlan but it generates same VNID. How does it generates same vnid. is there any type of conversion happening in background ?

[RedNectar]

Hi @Nitesh_A ,

I've seen that graphic before - it is a fantastic picture.  But to answer your question:

Although both leaf switches has different PI vlan but it generates same VNID. How does it generates same vnid. is there any type of conversion happening in background ?

you need to understand that in ACI you build a model using the APIC GUI.  The model is nothing more than a model until the APIC pushes the config to a Leaf Switch, which it does at the point in time where you create a static mapping to a leaf switch port for a particularVLAN for a particular EPG.

Now part of this model that is managed by the APIC is the VNIDs.  You can easily see the VNID for any BD.  Tenants > YourTenant >> Networking > Bridge Domains > Your_BD >| [Advanced/Troubleshooting] tab.

Or even easier: Just go to  Tenants > YourTenant >> Networking > Bridge Domains and look at the Segment ID

So when the APIC needs to instantiate a new BD on a Leaf, it tells the leaf what VNID to use for the BD, and the Leaf figures our what spare VLANs it needs to allocate for that BD.  This is of course reported back to the APIC - so that is the conversation that happens.  Next leaf gets told the same VNID and that leaf reports back its VLAN allocations - but these VLAN allocations are really just for display purposes. The APIC doesn't NEED to know the PI VLAN for any other reason.

[Nitesh_A]

@RedNectar Thankyou. I would love to know more deeper on this. Let me know if you have any link/books etc where i can study this. I have been giving interviews on ACI and getting questions for which i am unable to find the answers easily.

[RedNectar]

Hi @Nitesh_A ,

Maybe I should write a book!  

There are a couple of really good Cisco Live presentations, and some of CIsco's documentation is top notch. The best of these is Cisco's ACI Fabric Endpoint Learning White Paper written mostly by Takuya Kishida who also presented BRKACI-3545 and did this video too.

A google search for Cisco ACI Takuya Kishida will probably lead you to some more great stuff.

Even a google search for Cisco ACI RedNectar may throw up some interesting stuff, or look at my replies on my Cisco Community profile page (click on my name next to my avatar above)