Packet transferring in aci fabric

[Leftz]

Hi Please see below picture. That is diagrame of aci packet routing. Anyone can explain the relation between the numbers in green box and blue box? Thanks

[M02@rt37]

Hello @Leftz

As I know, PI-vlan is a VLAN dedicated to infrastructure-related communication within a specific tenant. It facilitates communication between various ACI fabric components, such as leaf switches, spine switches, and controllers. The PI-vlan is mainly used for control plane and management plane communication between fabric elements, helping in the exchange of configuration, status, and other essential information.

On the other hand, VNIDs are used in the VXLAN encapsulation header to tag traffic belonging to a particular Bridge Domain. Each BD in a tenant has a unique VNID, enabling the ACI fabric to correctly forward traffic between endpoints in different network segments.

[RedNectar]

Hi @Leftz & M02@rt37 ,

There seems to be some confusion here ( M02@rt37 - great to see you contributing to the community, but I need to point out a couple of misconceptions you have)

So lets start by showing the diagram full size so we can read it.

The first misconception is from @Leftz 

---

That is diagrame of aci packet routing.  

---

No - it is NOT. It has nothing to do with routing. If you look at the title of the slide where this diagram came from I think you'll see it is labelled VLAN Types in ACI and I did a whole video explaining this for you back in November 2022

The numbers in the BLUE box are the Private Internal VLAN IDs - these are PRIVATE TO THE LEAF

Sorry M02@rt37 , a PI-vlan is NOT "a VLAN dedicated to infrastructure-related communication within a specific tenant".  Nor is used for "control plane and management plane communication between fabric elements".  Your VNID explanation is closer to the mark.

So, in the diagram above, when a frame arrives at LEAF1 with an Access Encap VLAN of 5, it is allocated a Private Internal VLAN ID of 19, but when a frame arrives at LEAF2 with an Access Encap VLAN of 5, it is allocated a Private Internal VLAN ID of 30

The reason PI-VLANs are used on a per-switch (or per-leaf) basis is because it would be impossible to scale ACI beyond 4095 VLANs across multiple switches without doing this. remember each switch/leaf is still limited to 4095 PI-VLANs just like the rest of the world.

Now ACI still has to be able to take frames that arrive with VLAN ID = 5 on LEAF1 and get them to LEAF2 and have them exit with VLAN 5.  Now in 90% of ACI designs, communication between an endpoint on VLAN 5/LEAF1 and an endpoint on VLAN 5/LEAF2  is going to be bridged, because we expect each endpoint to be on the same subnet.  But ACI does not force endpoints in the same EPG to be in the same subnet - it is indeed possible and desirable in some designs to have endpoints in different subnets in the same EPG.

Scenario 1

I'll start with two endpoints in different subnets.  Let's say there is an endpoint connected to LEAF1 with IP of 10.10.10.10 and an endpoint connected to LEAF2 with IP of 11.11.11.11 (as described at about the 6:20 mark in that video).

And let's say 10.10.10.10 sends a packet to 11.11.11.11. For this packet to get from LEAF1 to LEAF2 it must be routed, so ACI will send the routed packet in VXLAN encapsulation using the VNID of the VRF - 2293760 in the diagram.

Scenario 2

Next, I'll deal with two endpoints in the same subnet.  Let's stick with an endpoint connected to LEAF1 with IP of 10.10.10.10 but an give the endpoint connected to LEAF2 an IP of 10.10.10.11 (as described at about the 4:40 mark in that video).

Again, let's say 10.10.10.10 sends a packet to 10.10.10.11. For this packet to get from LEAF1 to LEAF2 it must be switched/bridged, so ACI will send the routed packet in VXLAN encapsulation using the VNID of the BD - 15826915 in the diagram.

Scenario 3

You'll notice that there is also endpoints on both LEAF1 and LEAF2 labelled with VXLAN 8388608. This is to cater for the case where the endpoints are connected to ACI via Virtual Machine Management Domain that uses VXLAN encapsulation rather than VLAN encapsulation to separate endpoints into port-groups or networks - such as when using Cisco's AVE vSwitch.  Just like the other scenarios, each switch allocates a PI-VLAN to the incoming traffic, but that is more for historical reasons (to do with the internal design of 1st generation leaf switches). 

Scenario 4

You'll notice VXLAN/VNID 8388608 also is shown in green. Now, you didn't ask about green, but I'll mention that this VXLAN is used to distribute Cisco proprietary Spanning Tree BPDUs that arrive at either switch carrying a VLAN tag of 5

And one more thing

You'll also notice VXLAN/VNID 9492 in the diagram. THIS IS WRONG - 9492 is actually a pcTag, or EPG class ID, and is NOT global, but local to the VRF.

Remember you can see the whole presentation delivered by Takuya Kishida at Cisco Live as BRKACI-3545 - if you have Cisco Live access, you can view it here.

[M02@rt37]

Thanks @RedNectar,

Bad understanding as I concerned of that thread :

https://community.cisco.com/t5/application-centric-infrastructure/internal-vlan-aci-pi/td-p/4544074

Thanks again.

[Leftz]

Thanks! ACI use vnid to expand vlan number from 4095 to 16 million, why PI-vlan take the role? Please see below

"   The reason PI-VLANs are used on a per-switch (or per-leaf) basis is because it would be impossible to scale ACI beyond 4095 VLANs across multiple switches without doing this. remember each switch/leaf is still limited to 4095 PI-VLANs just like the rest of the world.  "

[RedNectar · Accepted solution]

Hi @Leftz ,

You are correct in saying that "ACI use vnid to expand vlan number from 4095 to 16 million" but that is 16 million across the whole ACI fabric.  Each switch is still limited to 4095 VLANs - there is no way ACI can support 16 million VLANs on any particular switch. 

REMEMBER: VNIDs are ONLY used when traffic is forwarded to the fabric - they have no local significance internally to the switch which needs to operate like a ordinary L2/L3 switch for local (on the same switch) traffic.

So without PI-VLANs, it would not be possible to have the same VLAN ID represent two different EPGs on the same switch.

Note:

To be able to map the same VLAN to two different VLANs on the same switch, you have to make sure each EPG is in a different BD and has the L2 Interface Policy VLAN Scope set to Per Port Local scope.

For a simple example, imagine this.  You have two tenants, Tenant1 and Tenant2, both with two interfaces connected to a switch.

Tenant1 uses interfaces 1/1 and 1/2.  Tenant2 uses interfaces 1/3 & 1/4.

Tenant1 has statically mapped EPG1 to VLAN 5 on interfaces 1/2 and 1/2 - and ACI has mapped VLAN 5 on interfaces 1/1 and 1/2 to PI-VLAN 51

Tenant2 has statically mapped EPG2 to VLAN 5 on interfaces 1/3 and 1/4 - and ACI has mapped VLAN 5 on interfaces 1/1 and 1/2 to PI-VLAN 52

To keep it simple, let's assume

1. All endpoints in EPG1 are in the same subnet
2. All endpoints in EPG2 are in the same subnet
3. ARP flooding is enabled in all cases

Now imagine an ARP request arriving with a VLAN tag of 5 in interface 1/1. 

What will happen is the ACI switch maps that broadcast to PI-VLAN 51 and floods it out interface 1/2 - BUT NOT 1/3 or 1/4 because they were mapped to PI-VLAN 52.

Now, if ACI didn't do this, it would NOT be possible to have the same VLAN mapped to two different EPGs.

Please keep asking questions if this is still not clear.

[Leftz]

Great, Thank you!