ACI Fabric Discovery

[Amin89]

Hello team,

I have 2 questions (confusion) regarding ACI Fabric discovery:

1- When the Leaf-1 sends a DHCP Discover, the offered IP is assigned to the Infra interface VLAN, right? (and then the Lo0 interface is configured from the bootstrap file)?

(I mean which interface gets the first DHCP)

2- When the Spine gets DHCP OFFER from APIC (IP for its Lo0), does it include a default/static route via leaf to reach the APIC in order to download its bootstrap file?

Thanks,

[RedNectar]

Hi @Amin89 ,

These are the slides I use to explain the process:

Or 

As to your specific questions about the infra VLAN

1- When the Leaf-1 sends a DHCP Discover, the offered IP is assigned to the Infra interface VLAN, right?

The IP address is assigned to a loopback as a /32 - the infra VLAN is mapped to an internal vlan on each switch and given a different IP address.

You can see the ip address by issuing a show ip interface lo0 command

Leaf2201# show ip interface lo0
IP Interface Status for VRF "overlay-1"
lo0, Interface status: protocol-up/link-up/admin-up, iod: 4, mode: ptep
  IP address: 10.2.176.66, IP subnet: 10.2.176.66/32
  IP broadcast address: 255.255.255.255
  IP primary address route-preference: 0, tag: 0

[Note: following has been edited: Originally I suggested a show vlan internal info command]

You can find the infra vlan mapping by issuing a show vlan encap-id <infa-vlan-id> command on the switch. My Infra VLAN is 3962

Leaf2201# show vlan encap-id 3962

 VLAN Name                             Status    Ports
 ---- -------------------------------- --------- -------------------------------
 8    infra:default                    active    Eth1/1

 VLAN Type  Vlan-mode
 ---- ----- ----------
 8    enet  CE

so on my Leaf2201 switch, the infra vlan is mapped to VLAN 8

I can see VLAN 8's ip address with show ip interface vlan 8 

Leaf2201# show ip interface vlan 8
IP Interface Status for VRF "overlay-1"
vlan8, Interface status: protocol-up/link-up/admin-up, iod: 66, mode: unspecified
  IP address: 10.2.0.30, IP subnet: 10.2.0.0/27
  IP broadcast address: 255.255.255.255
  IP primary address route-preference: 0, tag: 0

Note that both the infra vlan IP address and lo0 are in VRF overlay-1

(and then the Lo0 interface is configured from the bootstrap file)?

Well, the lo0 interface gets its IP from the DHCP offer

(I mean which interface gets the first DHCP)

I assume it is the first DHCP offered - but haven't tested.

2- When the Spine gets DHCP OFFER from APIC (IP for its Lo0), does it include a default/static route via leaf to reach the APIC in order to download its bootstrap file?

For a single pod, It doesn't need to. The IP address assigned will be on the same subnet as the APIC's IP address.  In multi-pod, you need to set up DHCP relay so that the DHCP requests reach the APIC.

Once the IP addresses are assigned, ISIS starts running and routes are learned via ISIS, so there is no need for a default/static route.

You can see the routes learned via ISIS using the command show isis route vrf overlay-1 

Leaf2201# show isis route vrf overlay-1
IS-IS process: isis_infra VRF: overlay-1
IS-IS IPv4 routing table
10.2.0.33/32
    *via 10.2.176.65, eth1/51.7, metric 2
10.2.0.34/32
    *via 10.2.176.65, eth1/51.7, metric 2
10.2.0.35/32
    *via 10.2.176.65, eth1/51.7, metric 2
10.2.24.64/32
    *via 10.2.176.65, eth1/51.7, metric 2
10.2.24.65/32
    *via 10.2.176.65, eth1/51.7, metric 2
10.2.24.66/32
    *via 10.2.176.65, eth1/51.7, metric 2
10.2.112.65/32
    *via 10.2.176.65, eth1/51.7, metric 2
10.2.112.66/32
    *via 10.2.176.65, eth1/51.7, metric 2
10.2.112.67/32
    *via 10.2.176.65, eth1/51.7, metric 2
10.2.176.64/32
    *via 10.2.176.65, eth1/51.7, metric 3
10.2.176.65/32
    *via 10.2.176.65, eth1/51.7, metric 2

Note that the next hop for all these routes is 10.1.176.65 - which is the Spine IP address (my lab has only one spine - your output will likely be different).  You can find the IPv4 anycast gateway address using the command show isis dteps vrf overlay-1 

Leaf2201# show isis dteps vrf overlay-1

IS-IS Dynamic Tunnel End Point (DTEP) database:
DTEP-Address       Role    Encapsulation   Type
10.2.176.65        SPINE   N/A             PHYSICAL
10.2.24.65         SPINE   N/A             PHYSICAL,PROXY-ACAST-MAC
10.2.24.64         SPINE   N/A             PHYSICAL,PROXY-ACAST-V4
10.2.24.66         SPINE   N/A             PHYSICAL,PROXY-ACAST-V6
10.2.176.64        LEAF    N/A             PHYSICAL
10.2.24.67         LEAF    N/A             PHYSICAL

[Amin89]

Hello @RedNectar ,

Thank you for your answer, really informative, but I need some details:

- As for my understanding, when the Leaf receives LLDP packets from APIC it will program the Infra VLAN on its interface and start sending DHCP Discover. so, the DHCP Discover is sent from the sub-interfaces (infra VLAN) on Leaf?

and therefore, After registering the leaf, the Offered IP from APIC is assigned to this sub-interface. Later, a bootstrap file is downloaded from APIC, which will make the TEP address configured on the lo0 and the interface facing Spine unnumbered, ISIS config, etc.

And for the spines and other leafs, LLDP will program Infra Vlan on their interfaces, and for the first leaf will DHCP relay to the APIC.

What I'm saying is probably not accurate, so please correct me.

Thanks

---- 

edit: for some reason, I wasn't able to see your full answer before, I only saw the pic and video, so, I replied with the previous message! , I edited below:

- Maybe, it is a dump question, but, can we configure a Loopback interface to get an Ip address via DHCP, like a normal interface? (I wasn't able to achieve it on N9K, NX-OS)

- What I understand from your answer is that the lo0 interface is created as a result of the DHCP Offer which is received on the actual leaf interface.

"Once the IP addresses are assigned, ISIS starts running and routes are learned via ISIS, so there is no need for a default/static route." -> I thought ISIS is configured from APIC via the bootstrap file, is it preconfigured on the switches os?

Br.

[RedNectar]

Hi @Amin89 ,

Bit of cross posting there - I've was updating my original answer while you were responding.

But back to your comments:

Thank you for your answer, really informative, but I need some details:

- As for my understanding, when the Leaf receives LLDP packets from APIC it will program the Infra VLAN on its interface and start sending DHCP Discover. so, the DHCP Discover is sent from the sub-interfaces (infra VLAN) on Leaf?

That is probably right, but remember the infra vlan will also be mapped to a different internal vlan.

However, I'm not 100% sure if DHCP packets are sent BEFORE the registration process, but I do know that the APIC learns the leaf serial number, which it could learn for the 1st leaf via LLDP or DHCP - but for the spines and other leaves, the APIC can't learn these serial numbers via LLDP, so probably the DHCP requests are sent and have the serial number embedded somewhere.

However, the APIC won't assign an IP until the leaf has been registered.

and therefore, After registering the leaf, the Offered IP from APIC is assigned to this sub-interface. Later, a bootstrap file is downloaded from APIC, which will make the TEP address configured on the lo0 and the interface facing Spine unnumbered, ISIS config, etc.

Well, the loopback gets the VTEP IP address, and sure, the vlan sub-interface ALSO gets an IP address, and both addresses are advertised via ISIS.

I believe all this happens BEFORE the bootstrap file is downloaded - but I haven't set up a packet capture to examine this.

From Cisco Application Centric Infrastructure Fundamentals, Release 5.0(x)

The ACI fabric bootstrap sequence begins when the fabric is booted with factory-installed images on all the switches. The Cisco Nexus 9000 Series switches that run the ACI firmware and APICs use a reserved overlay for the boot process. This infrastructure space is hard-coded on the switches. The APIC can connect to a leaf through the default overlay, or it can use a locally significant identifier.

And for the spines and other leafs, LLDP will program Infra Vlan on their interfaces, and for the first leaf will DHCP relay to the APIC.

There is a bit of mystery here that I've never bothered to completely research, but my spine has only one vlan - vlan-1 mgmt:inb and a show interface trunkcommand shows all interfaces trunking with vlan-1 being the native VLAN but NO allowed vlans on any interface.

But yes, the other leaf switches will map a VLAN to the infra VLAN and send DHCP requests which will be flooded to reach the APIC on the infra VLAN.

Sorry that there a few gaps in my explanation

[RedNectar]

Hi @Amin89 ,

There's been a bit of cross posting and editing, so I'll add a new response rather than editing an old one.

New points from your edit

but, can we configure a Loopback interface to get an Ip address via DHCP, like a normal interface? (I wasn't able to achieve it on N9K, NX-OS

Hmm.  Well, the lo0 IP is assigned by DCHP, so I guess it can be done. But perhaps not in normal config

I thought ISIS is configured from APIC via the bootstrap file, is it preconfigured on the switches os?

Yep.  That's what I understand from Cisco Application Centric Infrastructure Fundamentals, Release 5.0(x)

The ACI fabric bootstrap sequence begins when the fabric is booted with factory-installed images on all the switches. The Cisco Nexus 9000 Series switches that run the ACI firmware and APICs use a reserved overlay for the boot process. This infrastructure space is hard-coded on the switches. The APIC can connect to a leaf through the default overlay, or it can use a locally significant identifier.

And FYI - I've found a more detailed breakdown of the process in the Cisco_TroubleshootingApplicationCentricInfrastructure.pdf book p 14 - pretty much the first thing in the book.

The ACI fabric discovery process follows a specific sequence of events. The basic steps are as follows.
1 Connect to the KVM console of the first APIC and complete the setup script by inputting values such as fabric name, APIC cluster size, and tunnel endpoint (TEP) address pool.
2 Once completed, APIC1 will begin sending LLDP via its fabric ports. The LLDP packets contain special TLVs with information such as the infra VLAN and its role as an APIC (also referred to as the controller).
3 On reception of these LLDP packets from APIC1 the leaf will program the infra VLAN on all ports where an APIC is detected.
4 The leaf begins sending DHCP Discovers on the now-known infra VLAN.
5 The user logs into the OOB IP of APIC1 via HTTPS and registers the first leaf
node in the Fabric Membership submenu.
6 Once the leaf is given a Node ID, APIC1 will respond with an IP address from the
configured TEP address pool and the DHCP process completes.
7 The registered leaf relays DHCP Discovers from other directly connected spines which were discovered via LLDP to APIC1. 
8 The user will see those dynamically discovered spines appear in the Fabric Membership submenu and can register them.
9 Once the spines are registered, APIC1 responds with an IP address from the TEP pool and DHCP completes for those nodes.
10 The spines relay DHCP Discovers from all other nodes of pod1. (This is assuming there is a full-mesh between spines and leaf switches as is advised and is the typical architecture).
11 Once the leaf nodes connected to the other APICs are registered, the APIC cluster can be established via TCP communication amongst themselves. Make sure to complete the setup dialog on APIC2 and APIC3.
12 Confirm all APICs have formed a cluster and are fully fit. If this is the case, fabric discovery is complete.