NX-OS Switch BGP Peer with Itself in different VRF

[j.a.m.e.s]

Dear All,

I am implementing an inter-tenant firewall and I would like to know what the most efficient way of setting up the routing would be. Basically, I need to route between a main VRF and several protected VRFs using a firewall to hop between them. Each protected VRF would be advertised a default only. There is an EVPN fabric behind all these so the tenants are kept separate.

My current thinking is to run iBGP across the firewall and allow the switch to peer with itself inside a different VRF. I heard that this is possible by changing the MAC address used for the peering? Does anyone know the configuration for this, or could point out a document? My platform is N9k.

I have shown the target setup in a diagram.

Regards

James.

[Giuseppe Larosa]

Hello James,

looking at your network diagram, we see that the inter mediate firewall is not working in transparent mode but it is a L3 device (and above) with different IP subnets towards the Nexus 9000 different VRFs.

You don't need to change any MAC address at SVI interface level on the Nexus device as there is not direct L2 communication.

The trick of changing the MAC address on one side is needed only if the FW is in transparent mode allowing L2 direct communication.

Using iBGP of a device in main default VRF with the same device in a VRF has the following challenge:

BGP uses the BGP router-id concept, that is chosen for the whole node (at least in IOS, IOS XE devices).

If no command is available to change the BGP router-id on the VRF side, BGP may not be able to setup seeing the same BGP router-id. (This is something to be tested)

In that case you should use a different routing protocol an IGP like OSPF or EIGRP that have this capability.

To be noted, being the FW in routed or NAT mode ( the naming depends on FW vendor) it should take part in the routing.

So if the FW is an ASA EIGRP could be considered.

If the FW is a third party device  OSPF might be your only choice.

I understand you are already using BGP , MP BGP for the EVPN and this would make BGP attractive.

If there is a command that allows to set a different BGP router-id for each VRF on the Nexus you could use iBGP also on the FW and make the FW a route reflector server between the main default VRF and each VRF.

Using the FW as iBGP RRS might even be a workaround for the same BGP router-id issue.

Edit:

according to the following configuration guide Advanced BGP for Nexus 9000 NX OS 9.3, it is possible to set a BGP router-id in each VRF

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/93x/unicast/configuration/guide/b-cisco-nexus-9000-series-nx-os-unicast-routing-configuration-guide-93x/b-cisco-nexus-9000-series-nx-os-unicast-routing-configuration-guide-93x_cha...

Using the FW as iBGP RRS removes the need to setup static routes on the FW, so it should be a better solution.

Hope to help

Giuseppe

[j.a.m.e.s]

Hi Giuseppe,

Thank you for taking the time to reply. Presumably with iBGP though, I would need 2x peerings to the firewall for each VRF? The firewall is a Fortigate which can run as a RR server. It sounds like a lot of manual neighbour configuration would be needed.

Do you think an alternative would be to run the firewall in transparent mode? Does this allow for sub-interfaces to be setup on the switch vrf at either side, and would these have unique mac addresses?

Regards

James.

[j.a.m.e.s]

Giuseppe

I have thought about this in more detail and I think running the FW in transparent mode is the best option. This would then allow iBGP peerings to run through. There are a few tweaks though which I believe are necessary:

• Next-Hop-Self
• Set ibgp update-source to the egress interface (does this need to be hardcoded?)
• Set the bgp router-id per vrf
• Change the mac-address on the sub-interfaces

As you rightly pointed out, the BGP router-id can be set per-vrf on nx-os:

n9k-bdrlf01(config)# router bgp 65000
n9k-bdrlf01(config-router)#   vrf vf-protected
n9k-bdrlf01(config-router-vrf)# router-id ?
  A.B.C.D  Manually configured router identifier

And the sub-interfaces appear to allow the mac-address to be hardcoded and to configure overlapping subnets (overlapping, but in different VRFs):

interface Ethernet1/1.411
  mac-address 0000.0411.0001
  vrf member vf-protected
  ip address 10.0.0.3/24
interface Ethernet1/1.511
  mac-address 0000.0511.0001
  vrf member vf-outsideworld
  ip address 10.0.0.1/24

I have never tried peering through a transparent FW like this. Could you let me know if you think this would work please (updated diagram attached)?

Many thanks

James.

[j.a.m.e.s]

Just to close this thread down...I didn't get the opportunity to test the above iBGP peering across the firewall. It should work as I think it's the approach recommended at CIscoLive BRKDCN-2304 (page 82), but it's still a lot of peerings and configuration when you consider the full-mesh and failover scenarios. Since I'm using Fortinet for the FW, I went with the option of stacked vdoms, which meant L3 peerings to the root (parent) of the firewall then using inter-vdom links to propagate routes through to the other vdom/vrfs.