Virtual connection between VRF's

wj343 · ‎08-06-2019

Is it possible to setup two virtual interfaces in different VRF's that are connected to each other? This would need to be equivalent to having two physical interfaces that are physically connected to each other using fiber, with one interface being in a different VRF.

My use case is as follows (feel free to suggest alternatives):

I have traffic from external sources (transit & peering) that ingresses into my core router (ASR9k), then egresses through a bundle-ether into a custom filtering appliance (transparent layer 2), the exits that appliance and enters back into my core for distribution within the network.

Here's a sample configuration that I'm using:

interface Bundle-Ether1
  description Appliance In
  vrf dirty
  ipv4 address 10.0.0.0 255.255.255.254

interface Bundle-Ether2
  description Appliance Out
  ipv4 address 10.0.0.1 255.255.255.254
  ipv4 access-group ipv4-external-ingress ingress

interface Bundle-Ether10
  description IP Transit In
  ipv4 access-group ipv4-transit-ingress ingress

ipv4 access-list ipv4-external-ingress
  < some ACE's that needs to be applied to all extenal ingress traffic >

ipv4 access-list ipv4-transit-ingress
  10 permit ipv4 any any nexthop1 vrf dirty ipv4 10.0.0.1
  20 permit ipv4 any any

be1 is connected to the appliance's input, while be2 is connected to the output. Since the appliance is transparent/L2, it's basically like connecting a direct fiber cable between the interfaces of be1 and be2.

The issue with this is that if the appliance were to be taken offline, entry 10 of ipv4-transit-ingress would stop moving traffic from the transit interface to the dirty VRF to be sent through the appliance. While this would save the network from an outage, any ingress traffic will also end up skipping the ipv4-external-ingress ACL.

I need ipv4-external-ingress to apply to all ingress traffic, regardless of whether the appliance is active. However, these ACE's need to take place after the traffic has exited the appliance, as the appliance requires the full unfiltered external traffic forwarded.

I was planning on setting up a similar two-interface pair within different VRF's, except somehow virtualized (more reliable and doesn't take up physical ports), then using nexthop2 to move traffic through the VRF's/to the virtual interface so that ipv4-external-ingress can be applied if the route for nexthop1 doesn't exist (which it won't if the appliance is offline).

f00z · ‎08-08-2019

Usually in this case I set up the appliance to accept traffic on vlan X and return traffic on vlan Y so you only need to use one bundle interface with two subinterfaces one for egress one for ingress, and that way it only uses one set of physical ports. The traffic is unidirectional so this works out.

Inside the 'dirty' vrf you can set a default route that points back to the other vrf or leak the routes from the other vrf, a few options there.

The way mine is currently set up is similar to this however it's backwards, there's a clean vrf instead of a dirty, and the routes are injected via bgp, so they can be removed/rerouted if the appliance has issues or even the appliance itself can do the BGP injection and if it goes down so does the BGP.

IP SLA another option.

Last resort you could create a tunnel interface to itself (I've done this on other platforms for some internal hacks to get something working, although it requires recirculation). Haven't tested it on ASR9k though.

wj343 · ‎08-10-2019

Unfortunately it doesn't look like IOS-XR supports tunneling to itself, I'm getting an error message when attempting to setup two tunnels pointing to each other:

tunl_gre_ma[1170]: %FORWARDING-IP_TUNNEL-3-DEST_LOCAL : Tunnel-IP MA: Interface tunnel-ip 11 uses destination 192.168.1.10, which is a local IP address on interface Loopback10. This is not allowed and tunnel interface state will remain DOWN.

Configuration:

interface Loopback10
 ipv4 address 192.168.1.10 255.255.255.255
!
interface Loopback11
 ipv4 address 192.168.1.11 255.255.255.255
!

interface tunnel-ip10
 vrf dirty
 ipv4 address 192.168.2.10 255.255.255.0
 tunnel mode gre ipv4
 tunnel source Loopback10
 tunnel destination 192.168.1.11
!
interface tunnel-ip11
 ipv4 address 192.168.2.11 255.255.255.0
 tunnel mode gre ipv4
 tunnel source Loopback11
 tunnel destination 192.168.1.10
!

Ideally there would be some part of an access-list that allows other access lists to be applied in the middle of an access-list (sort of like RPL), but I don't believe that feature is available in XR.

Example:

ipv4 access-list ipv4-external-ingress
  10 permit <something>
  20 deny <something else>
  30 <ipv4 abf statement with nexthop1 configured>
  50 <apply all ACE's from ipv4-external-sub-ingress>
  100 ipv4 permit any any

ipv4 access-list ipv4-external-sub-ingress
  <some ACE's here>

f00z · ‎08-10-2019

I see what you are trying to do now. Ours is set up a bit differently using BGP to reroute and can reroute to other appliances if one fails.

One thing you could do is put a switch and use subinterfaces and set up a routing adjacency, so subinterface 50 (tag 50) goes through 50.ASR -> 50switch ->(untag) appliance (untag)-> (tag51)switch -> ASR.51, and then subinterface 100 goes 100.ASR -> switch -> ASR.100. Of course this requires adding a switch.

The metric on the routing adjaceny determines if it goes thorugh the appliance or not. For the nexthop1 in the ACL you'd set it to drop it into the vrf and not to a specific nexthop. The routing adjacencies would default originate. So if the appliance dies, the adjacency will go down and it will pick up to the backup route on another subinterface. I have done this before in some other circumstances, using IOS of course. IOS-XR has some strange limitations, like the tunnel to itself work fine on 6500/7600 for example, although it recirculates. Hopefully it will allow routing adjacencies to itself. On IOS this works but you have to manually set the MAC addr on the interface.

You'd think since ASR9K is more NPU than ASIC it would be much more flexible.