Overlapping or non-overlapping VTEP pool

[apache_le]

The question is when asking for a VTEP pool to configure ACI fabric, do you ask for non-overlapping IP pool or overlapping is OK?  I understand that the VTEP pool is only used within the fabric (for now) but with the direction of Multi-pod, multi-pod site, GOLF, the Vxlan boundary will be extended out the edge of the network thus VTEP IP will be advertised into IGP for the underlay.  That will required using a non-overlapping VTEP pool if you don't want to rebuilt your fabric.  Is this correct? 

[RedNectar]

The logic explained below is my logic and the way I explain it, there may be a different official Cisco answer.

The problem with VTEP pools is the APICs.  You see, the APICs can't handle

1. having a management IP address that overlaps with the VTEP address space, (it can't figure out which interface to send management responses on) or
2. being accessed from a workstation that is using an IP address that overlaps with the VTEP address space.

Since it is conceivable that any internal IP address may need to access the APIC for some reason sometime, I would recommend that you don't overlap VTEP addresses with any currently used internal addresses.

Below is an example of the routing table from an APIC:

apic1# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 172.16.11.1 0.0.0.0 UG 0 0 0 oobmgmt
10.0.0.0 10.0.0.30 255.255.0.0 UG 0 0 0 bond0.3967
10.0.0.30 0.0.0.0 255.255.255.255 UH 0 0 0 bond0.3967
10.0.32.64 10.0.0.30 255.255.255.255 UGH 0 0 0 bond0.3967
10.0.32.65 10.0.0.30 255.255.255.255 UGH 0 0 0 bond0.3967
169.254.1.0 0.0.0.0 255.255.255.0 U 0 0 0 teplo-1
169.254.254.0 0.0.0.0 255.255.255.0 U 0 0 0 lxcbr0
172.16.11.0 0.0.0.0 255.255.255.0 U 0 0 0 oobmgmt

In this case, the management interface is an OOB management interface, and the APIC sees the OOB management interface route as 172.16.11.0/24.  Now imagine for a minute I had used 10.0.11.0/24 as my OOB Management subnet.  Since that overlaps with my VTEP range (10.0.0.0/16) there is potential that an IP address of say 10.0.11.11 could be allocated to a VTEP somewhere - and if that happened my APIC would be unable to communicate with it because that address overlaps with my management address range.

HTH

[apache_le]

Excellent example of why the VTEP pool should be non-overlapping.  Thanks.

[RedNectar]

apache_le - don't forget to mark your question as Answered if you are satisfied with the answer given.  It helps anyone searching the forum to find unanswered questions, and helps others find the answer if they have the same question.

RedNectar

aka Chris Welsh

[suryakant.chavan]

Hi Chris,

Recently started learning ACI and per my understanding VTEP address falls under Infra/overlay-1 VRF and management address in management VRF and both should be maintaining different routing table and there should not be any issue while using same subnet IP.

Here also in netstat output VTEP pool IP use interface bond0.3967 in overlay-1 VRF and Management subnet use interface oobmgmt in mgmt VRF.

Then why both VTEP pool and management subnet should be different?

Thanks,

Surya

[richmond]

Because the APIC sees both networks and is not VRF aware.

[suryakant.chavan]

Thanks Rich, but Per my understanding APIC is VRF aware as we are using different VRF with Tenant.

Also Overlay-1 Vrf is part of Infra Tenant which includes VTEP IP range and Managemet ip are in Management tenant VRF, then how they will overlap.

[Robert Burns]

The APIC itself is NOT VRF-aware.  This is why the APIC management subnet and Infra subnet can't overlap.  

Robert

[suryakant.chavan]

That means we will face issue with APIC management only or for all other nodes like Spine and Leaf. I think VRF will be present on Leaf switches.

[john.dejesus]

Hi Chris,

Sorry for asking as this might sound simple to you, I am still trying to understand this concept. Since ACI is multi-tenant capable, I assume that overlapping IP is supported (just like L3 MPLS VPN). But Cisco states otherwise or I guess it's not very clear to me. So from my understanding, as long as the ACI Fabric TEP pool, ACI OOB and to the Virtualized Environment Servers that are directly attached to the fabric are non-overlapping, does this mean its still possible to have an overlapping IP addresses for other things? lets say for a scenario below.

ACI OOB: 10.0.0.0/24

TEP Pool (POD 1): 10.1.0.0/16

TEP Pool (POD 2): 10.2.0.0/16

Tenant A: Server Tenant (172.16.0.0/24)

User: 10.1.10.0/24

I have a server (physical or virtual) attached to the leaf/s in Tenant A, and a user outside the fabric (from Branch office through a Private WAN) that has an overlapping IP with the TEP pool and wants to access the Server in Tenant A. Will this pose an issue on packet forwarding inside the ACI infra vrf?

[RedNectar]

Hi @john.dejesus ,

---

lets say for a scenario below.

ACI OOB: 10.0.0.0/24

TEP Pool (POD 1): 10.1.0.0/16

TEP Pool (POD 2): 10.2.0.0/16

Tenant A: Server Tenant (172.16.0.0/24)

User: 10.1.10.0/24

 That will be fine, although I did have a problem with the APIC OOB IP overlapping with the TEP Pool once as I explained above " Now imagine for a minute I had used 10.0.11.0/24 as my OOB Management subnet.  Since that overlaps with my VTEP range (10.0.0.0/16) there is potential that an IP address of say 10.0.11.11 could be allocated to a VTEP somewhere - and if that happened my APIC would be unable to communicate with it because that address overlaps with my management address range." 

In your case, imagine the APIC allocates 10.1.10.11 via DHCP to the VTEP of one of the switches. Now you have a user who just happens to have an IP address of (you guessed it) 10.1.10.11 and is trying to log into the APIC. A packet with a source IP of 10.1.10.11 arrives at the APIC. Now, unless the APIC is smart enough to send the reply back on the same interface it arrived on, the APIC will have a routing problem, and since it will have a 10.1.10.11/32 route via the fabric, it will send it to the VTEP rather than to your user's PC

.

I have a server (physical or virtual) attached to the leaf/s in Tenant A, and a user outside the fabric (from Branch office through a Private WAN) that has an overlapping IP with the TEP pool and wants to access the Server in Tenant A. Will this pose an issue on packet forwarding inside the ACI infra vrf?

---

This will not be a problem. The ACI Tenant VRF IPs are completely separated from the VTEPs.  The problem ONLY exists if you want to log into the APIC management from a source IP that overlaps with the VTEP address space.

[john.dejesus]

Thanks for your clarification

[Robert Burns]

Just to further add to RedNectar's excellent response, the reason that Tenant traffic would never interfere with VTEP subnets comes from the encapsulation.  When a leaf receives tenant traffic, it does a lookup against the VRF for the endpoint, and determines (via COOP query) where the destination endpoint resides.  That original packet is then encasuplated with a VXLAN header with the destination address of the remote leaf (assuming the src & dst EPs aren't on the same physical leaf).  When the destination leaf receives the traffic it removes the encap VXLAN headers, and forwards the traffic to the destination endpoint. 

Let's examine the headers on a typical packet that would be sent from a host on one Leaf (TEP1) to an endpoint on another leaf (TEP2).

All the fabric forwarding decisions in this case would be based on the outer Source & Destination IP headers (sIPo & dIPo).  When the remote TEP receives this packet the Leaf determines the VRF/BD the traffic is destine for (also found in the VXLAN Header), then the headers are removed, and forwarding then done within the VRF/BD based on the original packets inner MAC/IP (shown as the dMACi/sMACi or sIPi/dIPi).

The explanation for not overlapping the APICs INB/OOB IP subnets with the Infra TEP Subnet not only includes the fact that an APIC is not VRF aware (can't distinuish between it's own Management traffic & infra traffic), but this also includes the Use Case when deploying the AVE/Opflex Extension - which requires extending the Infra Subnet outside of the fabric.  If the Infra Subnet conflicted with existing subnets within your environment, you would run into issues with this scenario. 

Summary:

• ACI's Infra Subnet should be unique and unused eslewhere in your environment
• APICs Management subnets should never ovlerap with the Infra Subnet
• The Infra VLAN should be dedicated and never be used elsewhere in your environment

Regards,

Robert