Using PBR to override a (loopback) directly connected network

[tramseyG4S]

I been using a workaround to overcome the limitation of a non-cisco network router that I have (which can't do OSPF). It connects to a Cisco based network which uses OSPF. But, I can't use that workaround anymore. So I need a new workaround or hack. The working idea is the thread topic. I don't know if it is viable though. So I'd appreciate a look over, once-over, or double check on my thought process, logic, and implementation.

I have a Ubiquiti Dream Machine Pro (UDMP) this services my household's networking needs. It talks to the extensive Cisco basement lab network (2 5505 ASA, 2 3825 ISR, 2 3560 L3/MLS, 3 L2-only 2950). Unfortunately, the UDMP is only capable of static routes. For whatever reason Ubiquiti has not see it fit to implement OSPF or even RIP on it.

I didn't want to take the admin hit of using static routes throughout the basement Cisco network - so as a quick and dirty way - how I solved the problem, was that I connected (to the basement Cisco network) a Ubiquiti EdgeRouter which can do OSPF. I implemented the networks that the UDMP uses on the EdgeRouter. I have the ER inject the networks into OSPF. This points traffic from the basement Cisco equipment that is destined to the UDMP, in the right direction. Then, when that traffic destined for the UDMP, reaches the right L3 router I use a single static route to override the OSPF route (from the ER) and to redirect the traffic to the UDMP which actually has the networks. This way if I make any network changes on the UDMP, then, I don't have to rework a bunch of static routes on the basement Cisco gear. Instead, I make one change on the EdgeRouter to implement any network changes I've made on the UDMP.

However, I now need to repurpose that EdgeRouter so I can't use this workaround anymore. And since the AD of directly connected networks trumps static routes - I can't just use a static route anymore to override a loopback network.

So I went looking and found Policy Based Routing. This How does Internet Work site states definitively that, yes, PBR will override directly connected networks. So, on the Cisco L3 switch connected to the UDMP, I created loopback interfaces for the UDMP networks and made those loopbacks /24 networks. Advertise them into OSPF. Setup an ACL to identify UDMP destine traffic, create the route-map, and then apply the route-map to on the 3650 L3 switch interfaces and VLANs. What I'm trying to achieve is a sort of, "Hey, if you get traffic for the UDMP networks - don't even look at the routing table, the next hop IP for that traffic is x.x.x.x"

The (/24 loopback network) routes for the UDMP networks appear in OSPF and, using traceroute, all the other devices can correctly route traffic for the UDMP to the correct L3 switch. Unfortunately, it just isn't working.

"show route-map <routemap>" doesn't show any hits. Neither does "debug ip policy" show any activity. So either I'm missing something and this isn't a usable workaround or I've incorrectly configured the PBR.

Presuming this is possible I can then provide config from the L3 switch.

[Richard Burts]

I find your explanation confusing and admit that I have a very incomplete understanding of your environment. You do seem to have an impressive basement lab network. So let me start with a question. Is there a single L3 device that forwards traffic to UDMP or are there multiple L3 devices that communicate directly with UDMP?

[tramseyG4S]

Thank you. Yes, I do take pride in the extensive basement lab I created. It's around 10 years old at this point. I used monies I had acquired from my early tech support positions to support the development of my networking knowledge. As I learned about networking though Cisco Academy classes, I decided that I wanted to practically (but permanently) implement the things that I was learning about. This lab basement isn't about solving any problem. It's just been a playground for me to learn, experiment, and (in a safe non-impacting way) learn from problems. It's design is over-kill, non-sensical, janky even. But, the knowledge I've gained from building and maintaining it has been my golden ticket to helping businesses and individuals and my paycheck.

Mew (the c3560) in the basement network communicates to the UDMP. I setup an individual /30 network (10.255.255.220) between them. Mew is 10.255.255.221 and the UDMP is 10.255.255.222

As it stands right now - the EdgeRouter covers the UDMPs lack of OSPF support. That is, I implement the networks the UDMP has (a data lan, IoT lan, Guest lan, Workbench lan, and Surveillance camera lan) and injects them into the basement Cisco network. So from the home network, when I want to communicate to something in the basement, I have routing back! The basement traffic would otherwise end up being routed to the EdgeRouter - but on Mew I use static routing to override OSPF, or provide a more trustworthy route, to the UDMP which is 10.255.255.222. So the EdgeRouter gets UDMP traffic going in the correct direction (towards Mew), but once it gets to Mew - I told Mew to just redirect it to the UDMP (instead of the EdgeRouter).

So here's the change and why I can't (or don't want to) use the EdgeRouter for this workaround. I got a new townhome and I want to use that EdgeRouter now to create a new network at the townhome. (and perhaps create a site-to-site IPSec VPN to connect it to the Cisco Lab and Home network).

Therefore ... I need to (or would like to) find a way to still cover for the UDMPs lack of OSPF. The idea being - create loopbacks on Mew itself. And then, in similar fashion, just override those loopback interface / networks and redirect to the UDMP.

Of course, yes - I could just buy another EdgeRouter, or any OSPF capable device and continue "the hack". But, I thought it be a good thought experiment and excuse to implement PBR into my basement network.

[paul driver]

Hello
TBH, I don’t see a problem redesigning this network for full commutation, From that diagram you don’t need the edge router to route for the UDMP device, you have enough L3 devices already to perform all the routing for your basement network.

I believe in the KISS approach and to me if the UDMP device has L3 ip address and is able to route (even if static routing) then why not make the 3825s perform the inter-vlan routing (running a FHRP such as HSRP/VRRP for your basement network/ subnets (include the UDMP network and whatever resides behind it) then advertise those networks between your ASA's/ external wan device for internet/NAT access, Lastly have the 3560s trunks carry your basement vlans around your LAN with that UDMP being assigned to one of those vlans.

[tramseyG4S]

Hm.. so right now the UDMP controls / owns the household networks (10.0.1.0/24, 10.0.(10, 20, 30, 40, 50).0.1/24). If I'm following you - correct me if not - the idea could be to move the UDMP networks off the UDMP and into the basement -- then L3 trunk them (by subinterfaces) up to the house behind HSRP? That could work ... but it'd 1) make the household depend on the basement 2) Obviate the need for the residential internet.

(Mind you, I didn't explain that the basement network was a backup / failover for the house if the residential internet service fails. But, this is the peril of asking for assistance on just a small portion or individual topic of a larger complex network...)

I want to keep the basement and household networks separate. The household is supposed to be the pristine non-jank "production" network of sorts. The basement network can be jank and jury-rigged and nonsensical, because it's my lab network where I experiment and play. If I break something - that only affects the basement network - and the only outage it creates is my access to the basement from my bedroom. It'd be really ideal if the Ubiquiti would just add OSPF to the UDMP. It's already pretty much an ASA otherwise... But, arguing that is beside the point.

That's why I was I was using the EdgeRouter to cover for the UDMP. Why I was thinking loopbacks + PBR could cover for the EdgeRouter. But, in the pinch of right now - supersizing the loopback networks and maintaining the (more specific) static routes works. Par for the course in my basement lab.

I'll work on PBR again, in the lab, a bit later. I need to get a proper network at the townhome built (not the comcast rental modem wi-fi)

[Richard Burts]

Thanks for the update. Glad that you have something that works. It will be interesting when you get around to it to hear what your experience is with PBR. Good luck. Keep us updated.

[tramseyG4S]

Well... It's been a while. So much has happened since I first started this thread. Unfortunately, it's sad to say, I've never got around to trying to work on the PBR solution to the quandary laid out. The hack of supersizing the loopback networks with more specific static routes to override became the de facto solution.

That said, I rediscovered this thread and it helped me again to re-solve pretty much the same problem. Last time it was getting traffic from the basement lab up to the residential home network. This time I needed the reverse, getting traffic from a new network in the home network into the basement lab.

The townhome has been great and the EdgeRouter, that you all helped me to extract, has worked great. I was able to build an IKEv2 site-to-site VPN between it and the 5505 ASA documented here. Unfortunately, Steven - the 5505 in charge at the time experienced some hardware failure (won't power on) and had to be replaced with Wallace. But, over the past couple of weeks, my employer threw a pair of 5515-X ASA as discards at me - so I replaced both Steven and Wallace (the 5505) with the 5515-X. But, this caused major trouble with my basement ISP provider, though, those issues has just been ironed out over this past weekend. Along the way, Ubiquiti updated the software / firmware of the UDMP firewall / appliance to provide over Wireguard as a VPN server option, OpenVPN support for Client VPN connections, as well as, IKEv2 site-to-site VPN capability. But, no OSPF routing support yet (boo).

All that to say, I was experimenting with UDMP's new Wireguard VPN server support and I wanted traffic from clients using that Wireguard VPN to be able to talk to the basement equipment. But, as the UDMP doesn't have OSPF support I needed to figure out routing for that. Which, huzza, I was able to use this thread again to figure out the magic solution again.

Now, things are perhaps in better place for me (personally and professionally) for me to persue study for CCNP. Part of that would be studying and learning more about PBR. But, it has been about 10 years since I got my CCNA so I'll have to probably start with re-certifying there, figuring out what has been added to that since then, then moving on to CCNP.

[Richard Burts]

Thanks for the update. Glad that you have been able to get things working. Best wishes as you resume your networking studies.

[paul driver]

Hello

---

@tramseyG4S wrote:

but it'd 1) make the household depend on the basement 2) Obviate the need for the residential internet.

I want to keep the basement and household networks separate

---

Thank you for clarifying.
Just like to point out as you mention segregation another possible option would to create an DMZ on the FW and attach the UDMP, this would provide you the separation but only logically not physically im afriad!.

[Jon Marshall]

In theory it should work so difficult to say why it isn't but your setup does sound somewhat complex. 

A possible workaround if you can't get this working is to remember that more specific routes are always preferred so you could use 2 x /25 static routes per /24 subnet to point to the UDMP and that should work. 

Not a great solution but it would allow you to use statics as you did on your previous router. 

Jon

[tramseyG4S]

... that's a great idea! I forgot about that precedent part of routing. If I can't get the PBR functional, then, yea the UDMP networks are separated enough I could make the loopback networks larger than necessary and just let the more specific static routes override. I'm going to try to implement that for one UDMP network, right now!

[tramseyG4S]

I was able to implement this, btw. I still want to work on getting the PBR functional. But, for right now this works wonderfully. Thank you.

Let me know if the diagram and writeup above is helpful or still confusing. I'll return later (probably tomorrow) with a copy of the early PBR config I did.

[Georg Pauwen]

Hello,

I agree that it is kind of difficult to get a good overview of what you are trying to achieve, from just reading your extensive post. What usually helps (a lot) is to see a visual diagram of your topology, so if you have that, or can create that, post it here.

PBR overrides everything and takes precedence....

[tramseyG4S]

Thank you for confirming that PBR overrides everything even loopback networks - I'll likely just have to look at my config again and see where I made a mistake or typo'd something. If I still can't get it working later - i'll post back some config. But for right now, supersizing the loopback network is a functional hack to allow the (more specific) static routing to precedence again.

Thank you everyone for your input on my quandary!

[paul driver]

Hello

---

@tramseyG4S wrote:

Thank you for confirming that PBR overrides everything even loopback networks

@Georg Pauwen PBR overrides everything and takes precedence..

This isnt true, it depends on what set statment you apply to the PBR stanza

Example1:
route-map PBR
set ip default next-hop x.x.x
or
set default interface xx

With both of the above commands the rtr will perfrom normal routing  and will ONLY policy route if/when no route entry for the destination exists in its rib table.

Example2:
route-map PBR
set ip next-hop x.x.x
or
set interface xx

With both of the above commands the rtr will policy route no matter if or not an entry for the destination exists in its rib table .