Uplinks on Catalyst 9000 series to multiple VRFs

[pehi030670]

Hello all, I need to propose a design to a customer that is demanding segregation of LAN traffic into compartments that contain several VLANs that are then policed by a firewall before traffic is permitted to enter/leave that compartment.

VRFs seem to be the obvious solution to this, and from the perspective of the L3 core switches this is simple enough.

However I have to get the traffic - physically - to and from the core switch. My reading indicates that an interface can only belong to one VRF, so if I have multiple VRFs, does this mean I need multiple physical interfaces to carry this traffic between access/distribution switches and the core and up to the firewall(s), or can I do any configuration with sub-interfaces to reduce and rationalise the physical interlinks?

All tips appreciated.

[Sergey Lisitsin · Accepted solution]

pehi030670,

The interface can only belong to a single VRF. However, don't forget that we are talking only layer 3 interfaces. So, you could have a trunk port with multiple VLANs in it and then a bunch of VLAN interfaces, all belonging to different VRFs. That would achieve separation between the VRFs, but within the VRF, VLANs would be able to communicate unrestricted.

[balaji.bandi]

You can use Port-channel (sub interface for the different VRF) for each VLAN or comparment traffic to seperate.

[Reza Sharifi]

Hi BB,

As far as I know, the 9ks don't support subinterfaces on physical or logical interfaces.

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9300/software/release/16-6/configuration_guide/int_and_hw/b_166_int_and_hw_9300_cg/b_166_int_and_hw_9300_cg_chapter_00.html

HTH

[Scott Hodgdon]

Reza,

Subinterface support for the 9500s that support 100G was added in IOS XE 16.10 : https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9500/software/release/16-10/release_notes/ol-16-10-9500.html#concept_lld_yb1_r2b__sw-16101-9500H 

Also added in the  same release for 9400:

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9400/software/release/16-10/release_notes/ol-16-10-9400.html#concept_pnn_xb1_r2b

Cheers,

Scott

P.S. Hope all is well. Not sure if you remember, but we met many years ago :-) .

[Reza Sharifi · Accepted solution]

Hi,

does this mean I need multiple physical interfaces to carry this traffic between access/distribution switches and the core and up to the firewall(s), or can I do any configuration with sub-interfaces to reduce and rationalise the physical interlinks?

So, you can do this with logical interfaces (SVIs). If you want to carry the traffic from the access switch all the way to the firewall using VRF you would have to configure the SVIs on the access switches as layer-2 vlan can't be in a separate vrf by itself.

The easiest way to do this is by using vlan separation on the access and distro (assuming both are layer-2 only) and then configure all SVIs on the core only and put them each in a separate VFP. From there, you need to configure the connection between the core and the firewall with multiple transit vlans each in a separate VRF. So, for example, vlan 10s SVI terminates at the core and belong to vrf test, you need a transit vlan (vlan100) for vrf test to carry the traffic from core to the firewall in vrf test as well.

the same logic applies to all the other vrfs

HTH

[chandlerbr]

Works with IOS-XE 17.x and straight forward configuration.  Uplinks can be port-channels or not.  Depends on your BW and link redundancy requirements.

Working scenario

9500-16X pair core/distribution with StackWise Virtual

Multiple Catalyst 9300 access switches.  Nexus works too.

Uplinks between 9300s and 9500s are MEC port-channels configured as dot1q trunks.

9500 VLANs and SVIs are strictly for multiple P2P /30 uplinks from the access switches.  Each SVI is in a different VRF.

9300 VLANs and SVIs are other end of the /30 P2P links plus client VLANs/subnets.

VRFs RED, BLUE, GREEN

All VRFs defined on the core

Necessary VRF(s) defined on the access switches

P2P link SVIs are /30

Client VLAN SVIs /24 or whatever you need

Core switch

VLANs 101 and 102

P2P from core to access switch 1. VLAN 101 with SVI 101 forwarding in VRF RED

P2P from core to access switch 2. VLAN 102 with SVI 102 forwarding in VRF RED

Static or dynamic routing needed for core to know where the client subnets are.

IP route for client VLAN 201 added to VRF RED, pointing to the RED P2P interface on access switch 1

IP route for client VLAN 202 added to VRF RED, pointing to the RED P2P interface on access switch 2

Access switch 1

VLANs 101 and 201

P2P from access switch 1 to core. VLAN 101 with SVI 101 forwarding in VRF RED

Client VLAN 201 with SVI 201 forwarding in VRF RED. The client GW

0.0.0.0/0 route in RED VRF pointing to the core RED P2P SVI 101.  Route learned from core through EIGRP or other, or static

Access switch 2

VLANs 102 and 202

P2P from access switch 2 to core. VLAN 102 with SVI 102 forwarding in VRF RED

Client VLAN 202 with SVI 202 forwarding in VRF RED.  The client GW

0.0.0.0/0 route in RED VRF pointing to the core RED P2P SVI 102.  Route learned from core through EIGRP or ther, or static

Repeat for the other VRFs.  With trunks for the uplinks, you can have one or all VRFs communicating to and through the core

VRF-Lite route leaking works as well if you need to communicate between VRFs.  Static or dynamic routing.

Say your NTP servers are in the "NTP" VRF and clients in the other VRFs need to use a common NTP source.

Add (leak) the NTP and Client routes to the global table on the core.  Routes need to point to the outgoing local SVI, not the next-hop IP.

Add the NTP LAN route from the global table to the RED VRF.  Route points to the local outgoing SVI, the next-hop, and the global table. 

Add the RED LAN route from the global table to the NTP VRF.  Route points to the local outgoing SVI, the next-hop, and the global table.

Example

!Static routes added to core switch

!No route modification needed on the access switches.

!Enables routing between RED and NTP VRFs

!10.0.0.0/30 and 10.0.0.4/30 are used for the P2P links between switches.

ip route vrf RED 10.10.1.0 255.255.255.0 10.0.0.2 name RED-LAN              ![to access switch 1 Vlan101 P2P]

ip route vrf NTP 10.10.2.0 255.255.255.0 10.0.0.6 name NTP-LAN              ![to access switch 2 Vlan102 P2P]

ip route 10.10.1 255.255.255.0 Vlan101 name RED-CLIENTS                          ![Leak RED to global]

ip route 10.10.2.0 255.255.255.0 Vlan102 name NTP-SERVERS    ![Leak NTP to global]

ip route vrf RED 10.10.2.0 255.255.255.0 Vlan102 10.0.0.2 global name NTP-into-RED

ip route vrf NTP 10.10.1.0 255.255.248.0 Vlan101 10.0.0.6 global name RED-into-NTPE

Show routes on the core.

show ip route vrf NTP.  You will see the RED route in the NTP VRF routing table.

show ip route vrf RED.  You will see the NTP route in the RED routing table.

Add ACLs to the mix and limit cross-VRF communication to UDP/123 and or specific hosts.

[balaji.bandi]

sure and good to know, appreciated your feedback.