Re: Abt Ingress policy enforcement mode for VRF

CaoMinhThinh4782 · ‎07-04-2019

Dear all,

I'm currently learning about ACI, forcus on Endpoint Learning firstly. I've read this page for details information.

When I learning the L3Out Endpoint Learning Considerations, I did not understand the explain of vrf ingress policy enforcement in the attached image.

Based on the document, they said that when the packet from non-border leaf to border leaf to go out L3Out, and the VRF mode is set to ingress policy enforcement, no source MAC or IP address is learned as a new remote endpoint at border leaf by a packet to the L3Out connection => I don't understand, why the border leaf doesn't learn new remote endpoint in this case?

Thank you for your spending time to read my trouble, hope that you could support me in this scenario!

richmond · ‎07-05-2019

The border still learns remote IP endpoints from the ingress leaves (edit: when the traffic is to an endpoint attached to the border and not to the L3 Out).

The prefix to EPG mapping is programmed on the ingress leaves so policy enforcement can happen as soon as the packet enters the fabric. With the mode set to egress the prefixes are only programmed on the border leaves and enforcement happens at the border.

There is also an option to stop IP endpoint learns on the borders altogether (Diasable Remote EP Learn) but this is a separate setting.

CaoMinhThinh4782 · ‎07-07-2019

Thank you for spending time to response my case.

As I understand from your post, the egress leaf will still learn Endpoint in case of traffic is sent to L3Out during using Ingress Policy Enforcement.

But, refer to the Cisco mentioned in the attached image, they said in the reverse. Or it's an excepted case?

Hope that you could make it sense to me!

Thanks & Brgs,

richmond · ‎07-08-2019

This might be an optimization to stop L3 Out data flows from keeping endpoints alive at the borders. I'd have to test it to see the behaviour with second gen gear. Where is the reference you've included documented?

CaoMinhThinh4782 · ‎07-08-2019

Please let me know about the test result, I also want to test but acctually I do not have the test equipments.

This is a link that I used to learn Endpoint Learning: https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-739989.html#Figure8Staleendpointafterendpoin

Thank you for your support!

richmond · ‎07-08-2019

So after reading the whitepaper I think you're correct. Traffic from endpoints on the ingress leaf does not trigger and endpoint learn at the border leaf (egress) when the traffic is to an L3 Out and the enforcement is set to ingress. Often the border leaves are also compute leaves in smaller fabrics, so the border leaves will have endpoint learns present due to endpoint to endpoint traffic flows.

richmond · ‎07-08-2019

I have tested this now. My ingress leaves were second generation and the egress (border) leaves were first generation. My ingress leaf was a Remote Leaf, which may also change things.

The behaviour I saw was that the endpoint on the ingress leaf was learnt at the border when the traffic was to an L3 Out destination.

Here is the ingress leaf route:

Rleaf202# show ip route vrf common:Lab
10.0.0.1/32, ubest/mbest: 2/0
*via 10.220.240.32%overlay-1, [200/41], 01w03d, bgp-65000, internal, tag 65000
*via 10.220.240.35%overlay-1, [200/41], 01w03d, bgp-65000, internal, tag 65000

These are the TEP IPs of the two border leaves:

Leaf101# show ip int brief vrf overlay-1 | egrep "10.220.240.3[2|5]"
lo0 10.220.240.32/32 protocol-up/link-up/admin-up

Leaf102# show ip int brief vrf overlay-1 | egrep "10.220.240.3[2|5]"
lo0 10.220.240.35/32 protocol-up/link-up/admin-up

I pinged from a switch that was acting as a client in the fabric:

Client_SW# ping 10.0.0.1 source 10.33.33.33
PING 10.0.0.1 (10.0.0.1) from 10.33.33.33: 56 data bytes
64 bytes from 10.0.0.1: icmp_seq=0 ttl=251 time=0.877 ms
64 bytes from 10.0.0.1: icmp_seq=1 ttl=251 time=0.628 ms
64 bytes from 10.0.0.1: icmp_seq=2 ttl=251 time=0.641 ms
64 bytes from 10.0.0.1: icmp_seq=3 ttl=251 time=1.793 ms
64 bytes from 10.0.0.1: icmp_seq=4 ttl=251 time=0.605 ms

--- 10.0.0.1 ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 0.605/0.908/1.793 ms

IP address: 10.220.240.35, IP subnet: 10.220.240.35/32

Now we check the endpoint learning:

Rleaf202# show endpoint ip 10.33.33.33
Legend:
s - arp H - vtep V - vpc-attached p - peer-aged
R - peer-attached-rl B - bounce S - static M - span
D - bounce-to-proxy O - peer-attached a - local-aged m - svc-mgr
L - local E - shared-service
+-----------------------------------+---------------+-----------------+--------------+-------------+
VLAN/ Encap MAC Address MAC Info/ Interface
Domain VLAN IP Address IP Info
+-----------------------------------+---------------+-----------------+--------------+-------------+
4 vlan-30 00b7.6666.7777 L eth1/52
common:Lab vlan-30 10.33.33.33 L eth1/52

The leaf attached to the client machine has learned the IP.

One of the border leaves has learned the IP.

Leaf101# show endpoint ip 10.33.33.33
Legend:
s - arp H - vtep V - vpc-attached p - peer-aged
R - peer-attached-rl B - bounce S - static M - span
D - bounce-to-proxy O - peer-attached a - local-aged m - svc-mgr
L - local E - shared-service
+-----------------------------------+---------------+-----------------+--------------+-------------+
VLAN/ Encap MAC Address MAC Info/ Interface
Domain VLAN IP Address IP Info
+-----------------------------------+---------------+-----------------+--------------+-------------+
common:Lab 10.33.33.33 tunnel18

So with generation 2 ingress and generation 1 egress the L3Out traffic does trigger a learn.

CaoMinhThinh4782 · ‎07-09-2019

Wow, thank you so much, I have seen the real result, it's quite useful to me. I need to note this case and may be I'll ask the cisco's tranner in my ACI tranning course.

Appreciate for all your support!

micgarc2 · ‎07-14-2019

Just a recommendation if you have GEN1 BLs and also EPs connected to this leaf with ingress VRF enforcement, I recommend you disable remote EP learning on the BL if you are running a version prior to 3.2.

See https://community.cisco.com/t5/application-centric/l3out-stale-endpoint-by-remote-endpoint-learning/m-p/3889525

I explained potential stale EP issues you can run into with GEN1 HW and this setup.

Thank you for participating in the Cisco Support Forum for ACI! If you have other questions related to this post, please let us know. If this response answers your questions, please mark this post "answered" and assign a rating to the response(s) provided. This will help notify other viewers that your question(s) is answered and this helps us provide better responses for this and future questions.

Regards,

Michael G.