ACI Spine-leaf interface config

[xh_liu]

Hi,

ACI underlay is OVERLAY-1 vrf, so i think spine and leaf connection should be trunk config, and use subinterface for overlay-1 vrf and use 802.1q config, like:

spine:

int ex/x/x.xx

describtion connect to leaf

ip vrf forwarding overlay-1

enc dot1q xx

ip add xx

!

and i know all these config are generated by aci, how i can check these interface config?

thanks for help!

[RedNectar]

Hi @xh_liu ,

ACI underlay is OVERLAY-1 vrf,

Correct

so i think spine and leaf connection should be trunk config,

Not necessarily, leaves and spines could send all traffic to each other untagged. In fact, even after 5 years of working with ACI I've never needed to know or care exactly what encapsulation is used between the leaves and the spines, just as you don't need to know how traffic is encapsulated on any chassis back-plane. It may use proprietary encapsulation. Indeed, I don't know what encapsulation is used.

and use subinterface for overlay-1 vrf

It does indeed use sub-interfaces, as well as loopback interfaces. Probably the best view to see how these are configured internally is to issue the command fabric node_list show isis interface brief level-1 vrf overlay-1 from the APIC.  Leave out the worf brief for more detail. I've put an example below that show how sub-interfaces are used and you can see clearly that the sub-interface numbers used do NOT refer to VLAN encapulations, because each end is numbered differently.

and use 802.1q config,

Now this is the part of the question that makes me think you have not grasped a fundamental ACI concept. And that is that when user traffic arrives at a leaf switch, the VLAN encapsulation is used to identify the source EPG, and then the VLAN enacpsulation is discarded.  It is NOT carried across the fabric to the spine.

---

Example output of the command fabric node_list show isis interface brief level-1 vrf overlay-1 from the APIC issued on a simple 2-leaf one-spine topology: viz:

[Leaf1201] [eth1/50] <-->[eth 1/1] [Spine1101] [eth1/2]<-->[eth 1/1] [Leaf1202]

apic1# fabric 1101,1201,1202 show isis interface brief level-1 vrf overlay-1
----------------------------------------------------------------
 Node 1201 (Leaf1201)
----------------------------------------------------------------
IS-IS process: isis_infra VRF: overlay-1
Interface    Type  Idx State        Circuit   MTU  Metric  Priority  Adjs/AdjsUp
                                                   L1  L2  L1  L2    L1    L2
Ethernet1/50.42 P2P   3    Up//Ready  0x01/L1  9366 1 1 64 64 1/1 0/0

----------------------------------------------------------------
 Node 1202 (Leaf1202)
----------------------------------------------------------------
IS-IS process: isis_infra VRF: overlay-1
Interface    Type  Idx State        Circuit   MTU  Metric  Priority  Adjs/AdjsUp
                                                   L1  L2  L1  L2    L1    L2
Ethernet1/50.7 P2P   3    Up//Ready  0x01/L1  9366 1 1 64 64 1/1 0/0

----------------------------------------------------------------
 Node 1101 (Spine1101)
----------------------------------------------------------------
IS-IS process: isis_infra VRF: overlay-1
Interface    Type  Idx State        Circuit   MTU  Metric  Priority  Adjs/AdjsUp
                                                   L1  L2  L1  L2    L1    L2
Ethernet1/1.38 P2P   3    Up//Ready  0x01/L1  9366 1 1 64 64 1/1 0/0
Ethernet1/2.37 P2P   4    Up//Ready  0x01/L1  9366 1 1 64 64 1/1 0/0

I hope this helps.

---

Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem.

---

[xh_liu]

Dear sir:

Thanks for your detail reply.

I think I know user vlan traffic will be changed into vxlan(l2vni/l3vni) traffic based on traffic type.

I just want to know how ovlay-1 traffic walk between spine and leaf which you say it may use proprietary encapsulation. 

I hope get clear answer by check aci config or aci document.

I draw a picture based on your reply show information:

because aci use sub-interface for overlay-1 vrf, I think spine-leaf link should be trunk, but I check apic gui, it tell me it is routed interface(i use cisco dcloud aci simulator get these information):

So I am comfused.

Hope get more help, thank you.

[RedNectar · Accepted solution]

Hi @xh_liu ,

I find your persistence inspiring. Please NEVER give up.

And thank you for creating a diagram for me to illustrate my topology.

As I said, for 5 years I've never needed to know the encapsulation between the leaves and the spine.  Although I have wondered about this. I have always assumed that traffic accross the infra connections would be encapsulated in the infra VLAN as defined during setup.

But thanks to your persistence I've had to dig a little deeper.  It took me some time to find a command that would reveal the actual VLAN encapsulation on the sub-interfaces. And I was a little surprised at what I saw.

Traffic is encapsulated on VLAN 2 between the leaves and the spines

The command that revealed this to me was the show interface brief command.  Here is an edited version of the output of that command on one of my lab topologies - my other lab topology is exactly the same and also uses VLAN 2, and I've even check our production ACI Fabric - it also uses VLAN 2!

apic1# fabric 1101,1201,1202 show interface brief
----------------------------------------------------------------
 Node 1201 (Leaf1201)
----------------------------------------------------------------
<Some lines deleted>
Ethernet         VLAN    Type Mode    Status Reason                   Speed     Port
Interface                                                                       Ch #
--------------------------------------------------------------------------------
<many lines deleted>
Eth1/50          --      eth  routed  up     none                     40G(D)    --
Eth1/50.42       2       eth  routed  up     none                     40G(D)    --
<many lines deleted>
----------------------------------------------------------------
 Node 1202 (Leaf1202)
----------------------------------------------------------------
<Some lines deleted>
Ethernet         VLAN    Type Mode    Status Reason                   Speed     Port
Interface                                                                       Ch #
--------------------------------------------------------------------------------
<many lines deleted>
Eth1/50          --      eth  routed  up     none                     40G(D)    --
Eth1/50.7        2       eth  routed  up     none                     40G(D)    --
<many lines deleted>
----------------------------------------------------------------
 Node 1101 (Spine1101)
----------------------------------------------------------------
<Some lines deleted>
Ethernet         VLAN    Type Mode    Status Reason                   Speed     Port
Interface                                                                       Ch #
--------------------------------------------------------------------------------
Eth1/1           --      eth  routed  up     none                     40G(D)    --
Eth1/1.38        2       eth  routed  up     none                     40G(D)    --
Eth1/2           --      eth  routed  up     none                     40G(D)    --
Eth1/2.37        2       eth  routed  up     none                     40G(D)    --
Eth1/3           --      eth  routed  down   sfp-missing              inherit(D --
<many lines deleted>

So now you have your answer! And thank you for inspiring me to find it for you!

---

Footnote:

I know traffic to the APICs is encapsualted on the infrastructure VLAN, so I also investigated the interfaces connected to our APIC.  The APIC connections are switchports, and configured as trunk ports and map the infra VLAN to an internal VLAN.  On the lab I used in the example above, the internal VLAN is VLAN 7, and the infrastructure VLAN is 3962.  Here is my output - the APIC is connected to interface Ethernet1/1 on leaf 2201 and 2202 (the output is the same for both)

apic1# fabric 2201 show interface ethernet 1/1 switchport
----------------------------------------------------------------
 Node 2201 (Leaf2201)
----------------------------------------------------------------
Name: Ethernet1/1
  Switchport: Enabled
  Switchport Monitor: not-a-span-dest
  Operational Mode: trunk
  Access Mode Vlan: unknown (default)
  Trunking Native Mode VLAN: unknown (default)
  Trunking VLANs Allowed: 7

which shows that it is VLAN 7 internally, but a show vlan extended should show that vlan-7 is mapped to the infrastructure VLAN 3962

apic1# fabric 2201 show vlan extended
----------------------------------------------------------------
 Node 2201 (Leaf2201)
----------------------------------------------------------------

 VLAN Name                             Encap            Ports
 ---- -------------------------------- ---------------- ------------------------
 7    infra:default                    vxlan-16777209,  Eth1/1
                                       vlan-3962

And sure enough - the above shows that vlan-7 is mapped to the infrastructure VLAN 3962

[xh_liu]

Thanks a lot!