SD Access inter VN route leaking

[stevieoliver]

Hi

I have a test SD Access deployment.  2 Edge switches 2 Border/Control Planes and a Fusion router.

The Fusion router sees all internal fabric networks in various vrf's via BGP.  As recommended I have a Shared services vrf where I have things like DHCP, DNA Center, ISE, WLC but this is also my exit to the internet.  So I have a default route in that vrf.

When I create multiple VN's in my SDA Fabric I want to route leak the Shared services subnets and the default route into those VN's but I don't necessarily want to use my Fusion router as a transit between fabric VNs unless I specifically route leak subnets between each fabric VN.  I don't currently do that.  I leak Shared services subnets to each VN and VN subnets to Shared services.

However, when I route leak the default route into each fabric VN then even though I don't leak each VN's routes into the other VN I still create a path from VN1 to VN2.  VN1 follows the default route because it doesn't know VN2 subnets.  It gets to the Fusion router and it can reach both VN1 and VN2 subnets via route leaking into and from the Shared services vrf.  So the traffic hits the Fusion router and rather than going further via the default route the Fusion router routes to the other VN as it knows its subnets.

This is to be expected.  However, what is the recommended way to stop inter-VN communications via a transit through the Fusion router?

Both Borders are configured as Default Borders by ticking the Default to all Virtual Networks box when I define my Transit and both borders have the Do not import external routes box unticked.

I actually watched an online Cisco course where they don't even use vrf's on the Fusion router.  They link Border vrf's to the Fusion router's Global table and the idea is that eBGP loop avoidance via the same AS number will stop routes from one VN propagating into the other via the other Border.  I see that will be the case but if the default route is on the Fusion router this still creates a transit path between fabric VN's.  Traffic follows the default to the Fusion then picks up a specific route into the other VN.

Thanks for any ideas, Stephen.

[ChuckMcF]

So you're most likely not learning the route from a device outside of your network via the DFR. As you stated earlier, eBGP should stop you from being able to learn that you since it's the same AS.

My assumption would be that you didn't create import maps when you route leaked. If that is the case then you should add them to your vrf section in the Fusion Routers I created this example to help explain how to set it up.

Import Sales to Engineers
vrf definition ENGINEERS
rd 1:4101
!
address-family ipv4
import map ENGINEERS_IMPORT_MAP
route-target export 1:4101
route-target import 1:4101
route-target import 1:4102
exit-address-family

ip prefix-list SALES_PREFIX seq 10 permit 10.0.2.0/24

route-map ENGINEERS_IMPORT_MAP permit 10
match ip address prefix-list SALES_PREFIX

Import Engineers to Sales
vrf definition SALES
rd 1:4102
!
address-family ipv4
import map SALES_IMPORT_MAP
route-target export 1:4102
route-target import 1:4102
route-target import 1:4101
exit-address-family

ip prefix-list ENGINEERS_PREFIX seq 10 permit 10.0.1.0/24

route-map SALES_IMPORT_MAP permit 10
match ip address prefix-list ENGINEERS_PREFIX

Also, from past experience, if you have any control over purchasing you really should have two External Border Nodes and two Fusion Routers. Connections between them as follows: EBN1 to EBN2 (iBGP);  EBN1 to FR1 (eBGP); EBN1 to FR2 (eBGP); EBN2 to FR1 (eBGP); EBN2 to FR2 (eBGP) (essentially connect all together except the two FRs).

I hope this is helpful

Chuck McFadden

--please mark helpful posts as helpful--

[stevieoliver]

Thanks for the input Chuck

However, I am route leaking the required routes.  What is happening is

1. I route leak subnet A to Shared Services vrf from Fabric VN A.

2. I route leak subnet B to Shared Services vrf from Fabric VN B.

3.  I do NOT leak between VN A and VN B.

4. I then leak the Shared Services default route into both VN A and VN B.

So VN A and VN B have a default route but Shared Services has visibility of Subnet A and B.

So although VN A and VN B do not have routes to each other's subnet if they know the address they can get there by following the default route to Shared services and into the other VN.

So effectively by Shared Services knowing routes into each VN then by it's default route being leaked into those VN's it creates a transit through Shared services into each VN.

Hopefully I explained that correctly.

Regards, Stephen.

[ChuckMcF]

Glad to help. I love this stuff.

By using the import map on your VRF you are controlling what routes are learned by each leaked VRF. I've modified my example and placed it below to model what I think is what you are describing. In this example ENGINEERS and SALES are leaking to SHARED and SHARED is leaking back to both. There is no leaking between ENGINEERS or SALES.

By using the import map function in each VRF we are ensuring that the only route that ENGINEERS is learning from SHARED is 10.0.254.0/24, SALES is also only learning 10.0.254./24, and SHARED is only learning 10.0.1.0/24 and 10.0.2.0/24. Each VRF learns their DFR from their legacy connected VRF VLAN interface IP, none are learned via the leaking.

One additional point that just came to mind, ENGINEERS, for example, could have several networks in it's VRF. Let's say it actually has two SGT IP Pools: 10.0.0.0/24 and 10.0.1.0/24. The example I provided would exclude the SHARED VRF from learning about the 10.0.0.0/24 network as well as the DFR.

vrf definition ENGINEERS
rd 1:4101
!
address-family ipv4
import map ENGINEERS_IMPORT_MAP
route-target export 1:4101
route-target import 1:4101
route-target import 1:4112
exit-address-family
!
vrf definition SALES
rd 1:4102
!
address-family ipv4
import map SALES_IMPORT_MAP
route-target export 1:4102
route-target import 1:4102
route-target import 1:4112
exit-address-family
!
vrf definition SHARED
rd 1:4112
!
address-family ipv4
import map SHARED_IMPORT_MAP
route-target export 1:4112
route-target import 1:4112
route-target import 1:4101
route-target import 1:4102
exit-address-family
!
ip prefix-list ENGINEERS_PREFIX seq 10 permit 10.0.1.0/24
ip prefix-list SALES_PREFIX seq 10 permit 10.0.2.0/24
ip prefix-list SHARED_PREFIX seq 10 permit 10.0.254.0/24
!
route-map SHARED_IMPORT_MAP permit 10
match ip address prefix-list SALES_PREFIX
route-map SHARED_IMPORT_MAP permit 20
match ip address prefix-list ENGINEERS_PREFIX
!
route-map ENGINEERS_IMPORT_MAP permit 10
match ip address prefix-list SHARED_PREFIX
!
route-map SALES_IMPORT_MAP permit 10
match ip address prefix-list SHARED_PREFIX
!
ip route vrf SHARED 0.0.0.0 0.0.0.0 <external fabric VRF IP> name DFR
ip route vrf SHARED 10.0.1.0 255.255.255.0 <ENGINEERS vrf VLAN IP> name SHARED_TO_ENGINEERS
ip route vrf SHARED 10.0.2.0 255.255.255.0 <SALES vrf VLAN IP> name SHARED_TO_SALES
!
ip route vrf ENGINEERS 0.0.0.0 0.0.0.0 <external fabric VRF IP> name DFR
ip route vrf ENGINEERS 10.0.254.0 255.255.255.0 <SHARED vrf VLAN IP> name ENGINEERS_TO_SHARED
!
ip route vrf SALES 0.0.0.0 0.0.0.0 <external fabric VRF IP> name DFR
ip route vrf SALES 10.0.254.0 255.255.255.0 <SHARED vrf VLAN IP> name SALES_TO_SHARED

HTH,

Chuck McFadden

--please mark helpful posts as helpful--

[stevieoliver]

Chuck

Thanks again for your input.

You're doing pretty much what I'm doing but with a different method.

You use import maps to import routes from another vrf to shared services.  I use export maps to export from the original vrf.  I think it has the same effect.

You then define a static default in each vrf pointing to the Shared services vrf next how.  I actually leak that default from Shared services into the other two vrfs.  They need it for internet access.  So not sure if that's a big difference but it has the same desired outcome.  Each fabric vrf has a route leaked into Shared services and shared services sends the default route to them.

When I do a trace route from one fabric VN via my Border router to another fabric VN it has the following path

Border VN-A ->

Fusion VN-A

Fusion Shared Services IP address in default route subnet

Border VN-B IP address

So the strange thing is after the trace hits the Fusion in VN-A (or vrf A as the Fusion sees it) there is a further hop within the Fusion router via a Shared Services IP address.  It then goes back to the Border in VN-B

So the Fusion is acting as a transit because it is the default route for both VN-A and VN-B.  I found someone on Reddit who has described the same issue.

This makes sense to me.  Once the traffic hits the Fusion then a VN-B route not explicitly known by the Border in VN-A is suddenly reachable because the Fusion knows it.  So it passes the traffic on rather than trying to send it via my default route where it would get dropped.

I'll need to find some time to try some other solutions to this.

Thanks again, Stephen.

[ChuckMcF]

AHHH - the problem is that you are exporting the DRF from Shared Services to A and B. The problem with that is that A & B's networks are included in the DFR. So by exporting the DFR from Shared Services to A & B you are sending all routes between all VRFs. This effectively eliminates the fact that you are using exports maps (or import maps for that matter).

To fix the issue, remove the DFR from all export maps and create an individual SR for each VRF to the DFR. This way only specif traffic will be leaked and the rest will need to go outside of the fabric. BGP rules will stop the routes from being learned from outside of the fabric due to same AS.

Please try this and let me know if it works for you.

HTH,

Chuck McFadden

--please mark helpful posts as helpful--

[jedolphi]

Because the shared services network also contains the exit to internet, every SDA VN will need to have a default route to the shared services network, which means this accidental inter-VN connectivity you describe will continue to happen - default routes attract all traffic, very tricky in multi-VN networks that merge into a common routing table - not an SDA specific problem, also a problem for traditional MPLS L3VPN designs too. Some ways to solve:

-Put ACLs on the fusion to block inter-VN traffic, this can quickly become complex and unwieldy however

-Use a firewall as a fusion instead of a switch, that way you can block inter-VN traffic using a simple firewall construct such as zones (zone 1 connects to SDA-VN1 and zone 2 connects to SDA-VN2), I have this setup in BRKCRS-2819 from Barcelona 2020, check it on www.ciscolive.com if you wish

-Move the internet exit out of the shared services VN and create an per-SDA-VN internet exit e.g. if you have a firewall, connect sub-interfaces of the firewall to SDA VNs and route each SDA VN to a different firewall sub-interface

-You preserve VN1 SGTs in data plane into VN2, and then use TrustSec to block e.g. VN1-SGTx not allowed to talk to VN2-SGTy

Probably there's other ways too, but that's what comes to mind right now,

Best regards, Jerome

[stevieoliver]

Jerome, Chuck

Thanks again for your input.  As I suspected there is no simple answer to this.  The Firewall as a Fusion router seems to be the most suitable method with each VN terminating on a different firewall interface or sub-interface.  Or we still use a fusion router and give each VN a default route via their own firewall interface.  Then inter-VN routing could still be done on the Fusion router with route leaking but the default route must go via a firewall interface to stop inadvertent inter-VN communications.

When I watch some Cisco online training videos they don't really deal with this issue. 

Thanks again, Stephen,

[jedolphi]

I agree with your summary there Stephen regarding potential routing design solutions. Once traffic leaves the border it's no longer SD-Access, it's just standard multi-VRF routing. There's many many valid permutations of fusion configurations. I do touch in this in BRKCRS-3493 on www.ciscolive.com , but I also agree, it potentially could be more documented, and if it helps, the team writing the SD-Access Cisco Validated Designs are always working on new and better content.

[ChuckMcF]

The reason we went with dual 9500-16X as the FRs was the throughput and cost. The cost of a FW that does full 10G throughput on each port is substantially higher than a 9500-16X. As you say, though, there are many methods to make it happen. At the end of the day you are just getting the info out of the BGP AFs and moving it into whatever legacy format that you have.

If you think it would be helpful I can post a generic version of our set up. Please let me know if it would be helpful and I'll get started on it.

Chuck McFadden

--please mark helpful posts as helpful--