cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements
3375
Views
0
Helpful
10
Replies

Encap of the Endpoint in ACI

Greetings, sorry for the basic questions

can you please confirm whether in ACI we strip the incoming encap header while forwarding to the destination leaf... so for example

HostA --------LEAF1--------SPINE1--------LEAF2-----HostB

Assume HostA is sending a VXLAN packet (native).. When it hits LEAF1

a) strip the native vxlan header

b) encapsulate with ivxlan header

c) at Leaf2 strip the ivxlan header

d) add a native vxlan header and send it to HOSTB

is the above right? or is it that a) encapsute the original packet with ivxlan and send it to leaf2...?

 

If we do not strip the encap at the ingress then what happens if the HostA to LEAF1 is vxlan encap and HostB to LEAF2 is nvgre encap....?

Thanks

J

10 REPLIES 10
RedNectar
Advocate

Hi @Jayashanker warrier ,

 

You have nailed it. Your explanation is spot on.

HostA --------LEAF1--------SPINE1--------LEAF2-----HostB

Assume HostA is sending a VXLAN packet (native).. When it hits LEAF1

a) strip the native vxlan header

b) encapsulate with ivxlan header

c) at Leaf2 strip the ivxlan header

d) add a native vxlan header and send it to HOSTB

One of the advantages of this approach is that HostB doesn't even have to use the same ecapsulation. LEAF2 learns what encapsulation HostB uses when sending traffic, and then re-uses that encapsulation when sending from the leaf to the host. So it could be that HostB is using VLAN encapsulation, while HostA uses VXLAN.

ACI doesn't care

The key point about encapsulation in ACI is that:

ACI uses the incoming encapsualtion to identify the Source End Point Group.

...and then discards that encapsulation. It has done its job.

I hope this helps

 



Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem


RedNectar
aka Chris Welsh


Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem

Hi Chris, thanks a lot, a follow-up question...

as per the previous steps

d) when the LEAF2 (at the destination end) adds the nvgre/vxlan header(according to the destination leg encap), he has to construct ip header (src and dst)... from where will he get that info? (the src and dst would be the HOSTA  HOSTB right)?....  does he get the info from the ivxlan header?

 

thanks

J

Hi @Jayashanker warrier ,

The leaf switches learn just about everything by observing incoming packets.

So when the host connected to LEAF2 sent its first frame, LEAF2 would have recorded

  1. The encapsulation type (VLAN/VXLAN)
  2. The encapsulation identified (VLAN ID/VNID)
  3. The source MAC address - in the case of VXLAN encapulated packets, this will be the inner MAC
  4. The source IP address - in the case of VXLAN encapulated packets, this will be the inner IP

So when a frame coming through the fabric from Host A on LEAF1 reaches LEAF2, LEAF2 will recognise the destination (MAC if switched, IP if routed) and encapsulate the frame with the encapsulation learned and associated with that MAC/IP

I hope this helps

 



Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem


RedNectar
aka Chris Welsh


Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem

thanks Chris

 

Jayashanker

Hi, just a quick follow up question:

 

  • So I define a static binding for an EPG A with encap VXLAN VNI-1234 on Port 1/1 LEAF 1 and Port 1/1 on LEAF 2 as well.
  • EPG A to EPG A, two host on two different leafs, same VXLAN.
  • VXLAN Packet arrives at Port1/1 Leaf 1. ACI strips off Outer MAC/IP and VXLAN Header and learns Endpoint based on INNER MAC/IP? Same on Leaf 2? Endpoints are learnt eventually. ACI has Mapping table in COOP to knwo which endpoints are behind which LEAFs. Also ACI knows because of the static binding which encap to expect and to apply on outgoing packets. 
  • Packet from Host A is now sent to Leaf 2 and then encapsulated in VXLAN-Header VNI 1234 when beeing sent out the serverport. This is fine. But as ACI stripped off the OUTER MAC/IP Headers how will ACI be able to readd those on LEAF 2? Where is this mapping table stored? This is completely unclear to me and I never saw any documentation explaining this.

Hi @ThiloS ,

I've added some comments inline

  • So I define a static binding for an EPG A with encap VXLAN VNI-1234 on Port 1/1 LEAF 1 and Port 1/1 on LEAF 2 as well.
  • EPG A to EPG A, two host on two different leafs, same VXLAN.
  • VXLAN Packet arrives at Port1/1 Leaf 1. ACI strips off Outer MAC/IP and VXLAN Header and learns Endpoint based on INNER MAC/IP?

Correct. And the leaf remembers that INNER MAC/IP to Outer MAC/IP and VXLAN Header

  • Same on Leaf 2?

Precisely. Leaf 2 also remembers that INNER MAC/IP to Outer MAC/IP and VXLAN Header

  • Endpoints are learnt eventually. ACI has Mapping table in COOP to knwo which endpoints are behind which LEAFs. Also ACI knows because of the static binding which encap to expect and to apply on outgoing packets. 

I think you just answered your own question...."ACI knows because of the static binding which encap to expect and to apply on outgoing packets."

  • Packet from Host A is now sent to Leaf 2 and then encapsulated in VXLAN-Header VNI 1234 when beeing sent out the serverport. This is fine. But as ACI stripped off the OUTER MAC/IP Headers how will ACI be able to readd those on LEAF 2?

This information MUST be learned by LEAF2 after receiving a packet from the target MAC/IP on port 1/1.  If it has never received a packet from LEAF2 1/1 revealing the inner MAC/IP, then the frame/packet will not have got to LEAF2 in the first place - it would have been dropped way before this stage.

  • Where is this mapping table stored?

Each leaf stores its own local mapping table, and the contents of each leaf local mapping table is reported to the COOP Proxy, so the Spines have the complete global table

  • This is completely unclear to me and I never saw any documentation explaining this.

I hope this helps

 



Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem

@Jayashanker warrier - Did I solve your original problem? If so, please mark the answer as correct to help others


RedNectar
aka Chris Welsh


Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem

Hi @RedNectar,

 

thank you for the detailed response.

 

So ACI can learn endpoints from third party vxlan encapsulated packets and not just the "VTEPs" (=outer IPs)?

When two docker hosts using VXLAN communicate over ACI fabric the LEAFs would learn all containers that are communicating as endpoints and not just the two docker hosts with their VTEP addresses?

 

Is there a command to show the "outer" and "inner" mac/IPs? 

 

Hi @ThiloS 

So ACI can learn endpoints from third party vxlan encapsulated packets and not just the "VTEPs" (=outer IPs)?

Correct, but ACI needs knowledge of the outer VTEPs. This achieved via VMM integration.

When two docker hosts using VXLAN communicate over ACI fabric the LEAFs would learn all containers that are communicating as endpoints and not just the two docker hosts with their VTEP addresses?

That's the idea - but just like you can't plug two ESXi hosts into ACI and expect their guests to communicate over the ACI fabric without some configuration, the Docker Hosts will need to be configured. 

At this point you have hit my limit of knowledge of Docker integration with ACI - there are articles online to help, and Cisco provides a free Container Network Interface for ACI https://hub.docker.com/_/acicni

Is there a command to show the "outer" and "inner" mac/IPs? 

I believe the show endpoint command will show what you want - I don't have a relevant setup configured to show you exactly

apic1# fabric 1201-1202 show endpoint
----------------------------------------------------------------
 Node 1201 (Leaf1201)
----------------------------------------------------------------
Legend:
 s - arp              H - vtep             V - vpc-attached     p - peer-aged
 R - peer-attached-rl B - bounce           S - static           M - span
 D - bounce-to-proxy  O - peer-attached    a - local-aged       m - svc-mgr
 L - local            E - shared-service
+-----------------------------------+---------------+-----------------+--------------+-------------+
      VLAN/                           Encap           MAC Address       MAC Info/       Interface
      Domain                          VLAN            IP Address        IP Info
+-----------------------------------+---------------+-----------------+--------------+-------------+
109                                       vlan-1001    0050.56b9.193f L                      eth1/9
common:SharedServices_VRF                 vlan-1001        10.100.0.5 L                      eth1/9
<snip>
Tenant18:Production_VRF vlan-1183 10.118.11.200 LV po11 116 vlan-1184 0050.56b9.d238 LV po11 Tenant18:Production_VRF vlan-1184 10.118.12.200 LV po11 overlay-1 10.1.80.64 L lo0 overlay-1 10.1.192.67 L lo1 8/overlay-1 vxlan-16777209 34ed.1b8b.5a2b L eth1/1 ---------------------------------------------------------------- Node 1202 (Leaf1202) ---------------------------------------------------------------- Legend: s - arp H - vtep V - vpc-attached p - peer-aged R - peer-attached-rl B - bounce S - static M - span D - bounce-to-proxy O - peer-attached a - local-aged m - svc-mgr L - local E - shared-service +-----------------------------------+---------------+-----------------+--------------+-------------+ VLAN/ Encap MAC Address MAC Info/ Interface Domain VLAN IP Address IP Info +-----------------------------------+---------------+-----------------+--------------+-------------+ 43 vlan-1022 b496.9162.988b L eth192/1/32 Tenant02:Production_VRF vlan-1022 10.102.12.10 L eth192/1/32 <snip>
Tenant18:Production_VRF vlan-1184 10.118.12.200 LV po14 27 vlan-1183 0050.56b9.c75b LaV po14 Tenant18:Production_VRF vlan-1183 10.118.11.200 LV po14 overlay-1 10.1.80.66 L lo0 overlay-1 10.1.192.67 L lo1

I hope this helps

 



Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem


RedNectar
aka Chris Welsh


Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem

Hi Chris, Thanks for the explanation...

Assuming native vxlan encap from the host

So is it right  to say that if there is VMM integration (incase of vmware) then the endpoint will be learned (via opflex?)  and so the native vxlan encap will stripped at the ingress

If there is no integration then only the vtep address (at the host end) will be learned and for this scenario the native vxlan header will not be stripped at the ingress

 

Thanks

J

I did watch some ciscolive sessions about Kubernetes/Openshift integration in ACI.

 

There you see that an open Vswitch is installed into the hosts and this switch talks opflex to ACI and is configured just like a VMM integration. This way you see the docker hosts as endpoints with vlan-tags in their "mgmt-vlan-EPG" AND single containers as endpoints with their VXLAN encap in an Application-EPG which are learned via VMM. So the OVS is part of the ACI fabric as an own VTEP (need to extend INFRA vlan to host!) from my understanding (just like the AVS was in the past).

 

I think without vmm integration aci would only learn the outer vteps as endpoints (if you just do static binding with vxlan-encap). So you could have an endpoint with vxlan and one with nvgre talk to each other in theory but these hosts would need to have an own control plane to be able to know of each other and exchange encapsulated packets.