07-30-2020 03:15 AM
Greetings, sorry for the basic questions
can you please confirm whether in ACI we strip the incoming encap header while forwarding to the destination leaf... so for example
HostA --------LEAF1--------SPINE1--------LEAF2-----HostB
Assume HostA is sending a VXLAN packet (native).. When it hits LEAF1
a) strip the native vxlan header
b) encapsulate with ivxlan header
c) at Leaf2 strip the ivxlan header
d) add a native vxlan header and send it to HOSTB
is the above right? or is it that a) encapsute the original packet with ivxlan and send it to leaf2...?
If we do not strip the encap at the ingress then what happens if the HostA to LEAF1 is vxlan encap and HostB to LEAF2 is nvgre encap....?
Thanks
J
07-30-2020 12:58 PM
Hi @Jayashanker warrier ,
You have nailed it. Your explanation is spot on.
HostA --------LEAF1--------SPINE1--------LEAF2-----HostB
Assume HostA is sending a VXLAN packet (native).. When it hits LEAF1
a) strip the native vxlan header
b) encapsulate with ivxlan header
c) at Leaf2 strip the ivxlan header
d) add a native vxlan header and send it to HOSTB
One of the advantages of this approach is that HostB doesn't even have to use the same ecapsulation. LEAF2 learns what encapsulation HostB uses when sending traffic, and then re-uses that encapsulation when sending from the leaf to the host. So it could be that HostB is using VLAN encapsulation, while HostA uses VXLAN.
ACI doesn't care
ACI uses the incoming encapsualtion to identify the Source End Point Group.
...and then discards that encapsulation. It has done its job.
I hope this helps
Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem
07-30-2020 05:16 PM
Hi Chris, thanks a lot, a follow-up question...
as per the previous steps
d) when the LEAF2 (at the destination end) adds the nvgre/vxlan header(according to the destination leg encap), he has to construct ip header (src and dst)... from where will he get that info? (the src and dst would be the HOSTA HOSTB right)?.... does he get the info from the ivxlan header?
thanks
J
07-30-2020 09:12 PM
Hi @Jayashanker warrier ,
The leaf switches learn just about everything by observing incoming packets.
So when the host connected to LEAF2 sent its first frame, LEAF2 would have recorded
So when a frame coming through the fabric from Host A on LEAF1 reaches LEAF2, LEAF2 will recognise the destination (MAC if switched, IP if routed) and encapsulate the frame with the encapsulation learned and associated with that MAC/IP
I hope this helps
Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem
07-31-2020 08:51 AM
thanks Chris
Jayashanker
08-05-2020 12:23 AM
Hi, just a quick follow up question:
08-05-2020 02:52 AM
Hi @ThiloS ,
I've added some comments inline
- So I define a static binding for an EPG A with encap VXLAN VNI-1234 on Port 1/1 LEAF 1 and Port 1/1 on LEAF 2 as well.
- EPG A to EPG A, two host on two different leafs, same VXLAN.
- VXLAN Packet arrives at Port1/1 Leaf 1. ACI strips off Outer MAC/IP and VXLAN Header and learns Endpoint based on INNER MAC/IP?
Correct. And the leaf remembers that INNER MAC/IP to Outer MAC/IP and VXLAN Header
- Same on Leaf 2?
Precisely. Leaf 2 also remembers that INNER MAC/IP to Outer MAC/IP and VXLAN Header
- Endpoints are learnt eventually. ACI has Mapping table in COOP to knwo which endpoints are behind which LEAFs. Also ACI knows because of the static binding which encap to expect and to apply on outgoing packets.
I think you just answered your own question...."ACI knows because of the static binding which encap to expect and to apply on outgoing packets."
- Packet from Host A is now sent to Leaf 2 and then encapsulated in VXLAN-Header VNI 1234 when beeing sent out the serverport. This is fine. But as ACI stripped off the OUTER MAC/IP Headers how will ACI be able to readd those on LEAF 2?
This information MUST be learned by LEAF2 after receiving a packet from the target MAC/IP on port 1/1. If it has never received a packet from LEAF2 1/1 revealing the inner MAC/IP, then the frame/packet will not have got to LEAF2 in the first place - it would have been dropped way before this stage.
- Where is this mapping table stored?
Each leaf stores its own local mapping table, and the contents of each leaf local mapping table is reported to the COOP Proxy, so the Spines have the complete global table
- This is completely unclear to me and I never saw any documentation explaining this.
I hope this helps
Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem
@Jayashanker warrier - Did I solve your original problem? If so, please mark the answer as correct to help others
08-05-2020 03:55 AM - edited 08-05-2020 04:01 AM
Hi @RedNectar,
thank you for the detailed response.
So ACI can learn endpoints from third party vxlan encapsulated packets and not just the "VTEPs" (=outer IPs)?
When two docker hosts using VXLAN communicate over ACI fabric the LEAFs would learn all containers that are communicating as endpoints and not just the two docker hosts with their VTEP addresses?
Is there a command to show the "outer" and "inner" mac/IPs?
08-05-2020 12:06 PM
Hi @ThiloS
So ACI can learn endpoints from third party vxlan encapsulated packets and not just the "VTEPs" (=outer IPs)?
Correct, but ACI needs knowledge of the outer VTEPs. This achieved via VMM integration.
When two docker hosts using VXLAN communicate over ACI fabric the LEAFs would learn all containers that are communicating as endpoints and not just the two docker hosts with their VTEP addresses?
That's the idea - but just like you can't plug two ESXi hosts into ACI and expect their guests to communicate over the ACI fabric without some configuration, the Docker Hosts will need to be configured.
At this point you have hit my limit of knowledge of Docker integration with ACI - there are articles online to help, and Cisco provides a free Container Network Interface for ACI https://hub.docker.com/_/acicni
Is there a command to show the "outer" and "inner" mac/IPs?
I believe the show endpoint command will show what you want - I don't have a relevant setup configured to show you exactly
apic1# fabric 1201-1202 show endpoint ---------------------------------------------------------------- Node 1201 (Leaf1201) ---------------------------------------------------------------- Legend: s - arp H - vtep V - vpc-attached p - peer-aged R - peer-attached-rl B - bounce S - static M - span D - bounce-to-proxy O - peer-attached a - local-aged m - svc-mgr L - local E - shared-service +-----------------------------------+---------------+-----------------+--------------+-------------+ VLAN/ Encap MAC Address MAC Info/ Interface Domain VLAN IP Address IP Info +-----------------------------------+---------------+-----------------+--------------+-------------+ 109 vlan-1001 0050.56b9.193f L eth1/9 common:SharedServices_VRF vlan-1001 10.100.0.5 L eth1/9 <snip>
Tenant18:Production_VRF vlan-1183 10.118.11.200 LV po11 116 vlan-1184 0050.56b9.d238 LV po11 Tenant18:Production_VRF vlan-1184 10.118.12.200 LV po11 overlay-1 10.1.80.64 L lo0 overlay-1 10.1.192.67 L lo1 8/overlay-1 vxlan-16777209 34ed.1b8b.5a2b L eth1/1 ---------------------------------------------------------------- Node 1202 (Leaf1202) ---------------------------------------------------------------- Legend: s - arp H - vtep V - vpc-attached p - peer-aged R - peer-attached-rl B - bounce S - static M - span D - bounce-to-proxy O - peer-attached a - local-aged m - svc-mgr L - local E - shared-service +-----------------------------------+---------------+-----------------+--------------+-------------+ VLAN/ Encap MAC Address MAC Info/ Interface Domain VLAN IP Address IP Info +-----------------------------------+---------------+-----------------+--------------+-------------+ 43 vlan-1022 b496.9162.988b L eth192/1/32 Tenant02:Production_VRF vlan-1022 10.102.12.10 L eth192/1/32 <snip>
Tenant18:Production_VRF vlan-1184 10.118.12.200 LV po14 27 vlan-1183 0050.56b9.c75b LaV po14 Tenant18:Production_VRF vlan-1183 10.118.11.200 LV po14 overlay-1 10.1.80.66 L lo0 overlay-1 10.1.192.67 L lo1
I hope this helps
Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem
08-05-2020 12:51 PM
Hi Chris, Thanks for the explanation...
Assuming native vxlan encap from the host
So is it right to say that if there is VMM integration (incase of vmware) then the endpoint will be learned (via opflex?) and so the native vxlan encap will stripped at the ingress
If there is no integration then only the vtep address (at the host end) will be learned and for this scenario the native vxlan header will not be stripped at the ingress
Thanks
J
08-05-2020 09:51 PM - edited 08-05-2020 09:52 PM
I did watch some ciscolive sessions about Kubernetes/Openshift integration in ACI.
There you see that an open Vswitch is installed into the hosts and this switch talks opflex to ACI and is configured just like a VMM integration. This way you see the docker hosts as endpoints with vlan-tags in their "mgmt-vlan-EPG" AND single containers as endpoints with their VXLAN encap in an Application-EPG which are learned via VMM. So the OVS is part of the ACI fabric as an own VTEP (need to extend INFRA vlan to host!) from my understanding (just like the AVS was in the past).
I think without vmm integration aci would only learn the outer vteps as endpoints (if you just do static binding with vxlan-encap). So you could have an endpoint with vxlan and one with nvgre talk to each other in theory but these hosts would need to have an own control plane to be able to know of each other and exchange encapsulated packets.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide