VRF and BGP on Catalyst 9k

[ericdavishop]

I have 2 Catalyst 9k's being used an WAN routers to terminate links to our other offices into our data center.  Upstream from the Cat9k's is a firewall running BGP.  The 9k's are running BGP and I want to force the routes to go through the firewall so that each site cannot reach the other directly without having to go through the firewall policy.  I was thinking VRF might be the way to go but I am not sure I understand how to setup VRF and BGP on the 9k to make this work. I was also thinking I can add SVI's on the 9k's for each site/VRF and that way each site will only be able to share routes with the firewall.  I have setup something similar on a NX9k but the Cat9k seems a bit different with respect to VRF and BGP. Anyone have any ideas that may help?   I attached a diagram for reference.

[M02@rt37]

Hello @ericdavishop,

VRF is a suitable approach.

General approach:

--VRF Configuration

Create a VRF for each remote site on your C9K Switches. You can do this with the `vrf definition` command. 

vrf definition SITE1
rd 1:1
!
vrf definition SITE2
rd 2:2
Assign interfaces (SVIs or physical interfaces) to the appropriate VRFs. This separates the traffic for each site into their respective VRFs.

For example, if you have SVIs for SITE1 and SITE2:

interface Vlan10
description SITE1
vrf forwarding SITE1
ip address 192.168.10.1 255.255.255.0
!
interface Vlan20
description SITE2
vrf forwarding SITE2
ip address 192.168.20.1 255.255.255.0

--BGP Configuration

Set up BGP for each VRF to peer with the firewall. Each VRF will have its own BGP instance.

!
router bgp 65000
address-family ipv4 vrf SITE1
neighbor <Firewall-IP> remote-as <Firewall-AS>
redistribute connected
exit-address-family
!
router bgp 65001
address-family ipv4 vrf SITE2
neighbor <Firewall-IP> remote-as <Firewall-AS>
redistribute connected
exit-address-family

Use BGP route redistribution to advertise routes from the VRFs to the firewall and receive routes from the firewall.

--Firewall Configuration

On the Fw, configure BGP peerings with the C9K switches. The firewall will be responsible for controlling the routing between the remote sites.

--Access Control on the FW

Implement firewall policies to control traffic between the remote sites. Ensure that the firewall enforces the desired routing policies and security rules.

[ericdavishop]

M02@rt37,

Thanks for the quick response. Just to clarify, my sites will have Layer3 links to the 9k WAN switches.  I am using SVIs on the 9k to the firewall.   I have to run multiple instances of BGP on the Cat9k?  Is there any way to do this within the same BGP instance? 

[Giuseppe Larosa]

Hello @ericdavishop ,

you run a single instance of BGP router process on a device but you need a specific configuration for each VRF you want to configure.

Running multiple BGP router processes is not supported.

Edit:

you need to provide an end to end logical L3 connecivity within a VRF for each site so also the WAN link has to be associated to the same VRF and then there is an SVI that points to the Firewall.

Hope to help

Giuseppe

[ericdavishop]

@Giuseppe Larosa 
Thank you for the response.  Do I need to use RD and RT?  I have a similar setup on a NX 9k but did not use RD and RT for the VRF.  Seems like the Cat9k VRF is a little different.  I am pasting what I think it should look like on one of my WAN switches.  I just am unsure how to properly configure BGP for these VRFs.  Do i need to add separate peers in the firewall for each VRF?  How do the RD and RT work?

WAN01

vrf definition NYC-P2P
!
address-family ipv4
exit-address-family

vrf definition CRC-P2P
!
address-family ipv4
exit-address-family

interface Vlan3571
description SITE1-to-Firewall
vrf forwarding NYC-P2P
ip address 10.163.6.59 255.255.255.248

interface Vlan3572
description SITE2-to-Firewall
vrf forwarding CRC-P2P
ip address 10.163.6.66 255.255.255.248

interface TenGigabitEthernet1/1/1
description HOPNYCCORE1-CC-179289
no switchport
vrf forwarding NYC-P2P
ip address 10.163.201.2 255.255.255.252

interface TenGigabitEthernet1/1/2
description CRC-P2P
no switchport
vrf forwarding CRC-P2P
ip address 10.164.49.225 255.255.255.248

[M02@rt37]

@ericdavishop,

In your setup, you're using VRFs on the C9k switches to segregate traffic from different sites.

RD and RT are typically used in MPLS VPN scenarios to ensure unique VPN route identification and control route distribution within a VPN. They are not always necessary for basic VRF configurations like yours, especially if you don't have overlapping IP addresses between sites. We call that VRF lite, so you can omit the RD and RT configuration if your setup doesn't require MPLS VPN functionality or complex inter-VRF route sharing.

It's common to use a single BGP instance with multiple address families within each VRF context for this type of setup. Your configuration for BGP within VRFs looks correct.

As concerned the Firewall configuration, you will need to configure the firewall to peer with each VRF separately. The firewall should have separate BGP peer configurations for each VRF to establish BGP sessions.

[ericdavishop]

M02@rt37 
Thank you.  My only problem then is that when I try to add an address family for the VRF it complains that i do not have RD configured on the VRF

(config-router)#address-family ipv4 vrf CRC-P2P
% VRF CRC-P2P does not have an RD configured.

[M02@rt37]

Hello @ericdavishop,

ip vrf CRC-P2P
rd auto

This should allow you to configure the BGP address family for the VRF without encountering the RD error. It's a practical solution when you don't need complex MPLS VPN functionality but still want to utilize VRFs to segregate your network.

[Giuseppe Larosa]

Hello @ericdavishop ,

I would expect configuration of RD to be required within each VRF definition unless there is an easy rd auto alternative.

RT are needed only if you would like to import/export routes in MP BGP but it is not your scenario.

On each VRF you need to peer with a different IP address on the FW, the FW has to send a default route on each eBGP session so that traffic is attracted to it.

Then on the FW you configure FW policies to decide who can talk with whom.

Edit:

I see your Cat9K is complaining about missing RD so yes you need to configure it with a unique value in each VRF definition or VRF def /address family ipv4 level

Hope to help

Giuseppe

[ericdavishop]

@Giuseppe Larosa M02@rt37 

Does this look right then?  I currently have 1 global router and 1 VRF that I want to deploy(CRC-P2P).  Will there be any leaked routes between the global router and the VRF?  My goal is to force the CRC-P2P site to route through the firewall to be able to reach the NYC-P2P site

WAN01

vrf definition NYC-P2P
!
address-family ipv4
exit-address-family

vrf definition CRC-P2P
!
address-family ipv4
exit-address-family

interface Vlan3571
description SITE1-to-Firewall
vrf forwarding NYC-P2P
ip address 10.163.6.59 255.255.255.248

interface Vlan3572
description SITE2-to-Firewall
vrf forwarding CRC-P2P
ip address 10.163.6.66 255.255.255.248

interface TenGigabitEthernet1/1/1
description HOPNYCCORE1-CC-179289
no switchport
vrf forwarding NYC-P2P
ip address 10.163.201.2 255.255.255.252

interface TenGigabitEthernet1/1/2
description CRC-P2P
no switchport
vrf forwarding CRC-P2P
ip address 10.164.49.226 255.255.255.248

router bgp 65046
bgp router-id 10.163.6.58
bgp log-neighbor-changes
neighbor 10.163.200.6 remote-as 65040
neighbor 10.163.200.11 remote-as 65045
neighbor 10.163.201.1 remote-as 65043
!
address-family ipv4
neighbor 10.163.200.6 activate
neighbor 10.163.200.11 activate
neighbor 10.163.201.1 activate
exit-address-family
!
address-family ipv4 vrf CRC-P2P
bgp router-id 10.163.6.66
neighbor 10.163.6.65 remote-as 65040
neighbor 10.163.6.65 activate
redistribute static

ip route vrf CRC-P2P 10.164.49.0 255.255.255.0 10.164.49.225
ip route vrf CRC-P2P 10.165.49.0 255.255.255.0 10.164.49.225

[M02@rt37]

@ericdavishop,

Your configuration appears to be on the right track for your intended setup.

In your configuration, routes within the NYC-P2P VRF should not be leaked into the CRC-P2P VRF, and vice versa, as long as you maintain separate VRFs and do not configure any route redistribution between them.

To force the CRC-P2P site to route through the firewall to reach the NYC-P2P site, you should configure BGP sessions with the firewall separately within each VRF. Your BGP configuration for the CRC-P2P VRF appears to be set up for this.

Just ensure that the firewall is configured to peer with the Catalyst 9k switch for the CRC-P2P VRF as a BGP neighbor and that appropriate policies are in place on the firewall to control the traffic flow between these VRFs.

Your IP route commands within the CRC-P2P VRF for routes to the firewall's interfaces (e.g., 10.164.49.0/24) look correct and should guide traffic through the firewall.

=> Verify the firewall's BGP configuration and routing policies to ensure that traffic from the CRC-P2P site is correctly routed through the firewall to reach the NYC-P2P site.

[ericdavishop]

Great.  I have not actually setup the NYC-P2P vrf yet.  That one is currently using the global router.  Will there be any leakage there or do I need to do anything to prevent leakage?

[M02@rt37]

@ericdavishop,

If you haven't set up the NYC-P2P VRF yet and it's currently using the global routing table, there won't be any route leakage between the NYC-P2P VRF and the CRC-P2P VRF unless you explicitly configure route redistribution between them or with the global routing table.

Keep the NYC-P2P VRF and the CRC-P2P VRF as separate entities. Do not configure route redistribution between them or with the global routing table unless you have specific requirements to do so.

When configuring BGP within the NYC-P2P VRF, maintain separate BGP sessions, separate address families, and specific route redistribution policies within that VRF. This separation ensures that BGP routes in one VRF are not shared with routes in the CRC-P2P VRF.

[ericdavishop]

M02@rt37 
I tested this and realized that the routes will not get passed since EBGP loop prevention blocks the route from passing an AS more than 1 time.  So I have been looking at EBGP-Multihop as an alternative solution.

[M02@rt37]

OK @ericdavishop,

Do you see the BGP feature allow-as-in ?

BGP Allow-as-in feature is used to allow the BGP speaker to accept the BGP updates even if its own BGP AS number is in the AS-Path attribute. By default EBGP loop prevention is, if any BGP speaker sees its own AS Number in the BGP update, then the update is rejected, thus the advertisement cannot be accepted...as you see.

https://www.cisco.com/c/en/us/support/docs/ip/border-gateway-protocol-bgp/112236-allowas-in-bgp-config-example.html