topic Thanks so much for taking an in Network Security

PBR on ASA to Interface Without Directly Connected Next Hop

Steve Gaede — Tue, 12 Mar 2019 07:45:02 GMT

I have an ASA on which I'm trying to use PBR to route to one of two ISPs which

I'll call "slow" and "fast."

The interface to the slow ISP is connected to a subnet on which the next-hop

address is clearly in the subnet and it would count as "directly connected."

The interface to the fast ISP is connected via pppoe. The interface address

is on a different subnet than the next-hop address so it would not be directly

connected.

The default route is to the slow ISP.

When I create route maps to send traffic to the slow ISP, I see the next-hop

address being selected and the egress interface selected in the first phase

of the packet trace. That tells me that my rules are working.

When I switch the map's next-hop address to be the next-hop address of the fast

ISP interface, PBR is selecting the right next-hop address, but it leaves the

egress interface decision to the next processing step, which always selects

the slow ISP interface. Using the recursive next-hop address selection in

the route map doesn't correct the problem.

Any suggestions on how to fix this? The only thing I can think of is to set

the default route to the fast ISP and use PBR to route to the exceptions

that need to go over the slow ISP instead of now where the exceptions

are to route to the fast ISP.

Hi

Francesco Molino — Sat, 14 May 2016 01:08:01 GMT

I assume that you're using ASA version 9.4 minimum (http://www.cisco.com/c/en/us/td/docs/security/asa/asa94/config-guides/cli/general/asa-94-general-config/route-policy-based.html)

the ip next-hop is working only if next-hop IP is directly connected.

You might try with set ip default next-hop or set interface.

could you give the output of debug ip policy?

thanks

Yes, ASA 9.5(1) on a 5506-X.

Steve Gaede — Sat, 14 May 2016 18:01:01 GMT

Yes, ASA 9.5(1) on a 5506-X. I have tried ip next-hop, default next-hop, recursive next-hop, and set interface. I think the debug output confirms that the problem is with this ppoe configured interface, since my default route is out the slow interface, there is no route to the outside world on the fast interface but I thought that PBR would fix that. Here's the output:

pbr: policy based route lookup called for 192.168.20.22/45321 to 206.168.118.24/0 proto 1 sub_proto 8 received on interface inside
pbr: First matching rule from ACL(15)
pbr: route map SingleHostTest, sequence 50, permit; proceed with policy routing
pbr: evaluating recursive next-hop 207.225.112.15
pbr: no route to next-hop 207.225.112.15 found
pbr: evaluating interface fast
pbr: no route to 206.168.118.24 found on interface fast
pbr: policy based routing could not be applied; proceeding with normal route lookup

Here is the policy map with recursive next hop and set interface configured as was the case when I tested:

route-map SingleHostTest, permit, sequence 50
Match clauses:
ip address (access-lists): test_route_maps
Set clauses:
ip next-hop recursive 207.225.112.15
interface fast

I'm missing something on your

Francesco Molino — Sat, 14 May 2016 19:10:29 GMT

I'm missing something on your explanation.

Could you send a quick sketch of your infrastructure? And sending out interfaces, routes and route-map configuration?

PBR is a simple thing that's working quite in the same way as router, I've never done any testing with pppoe interface but I'm certainly missing something on your explanation but with a design and some config output it would be clear.

Would you mind to add the output of sh int ip brief ?

Thanks

Here it is, addresses have

Steve Gaede — Sat, 14 May 2016 19:44:00 GMT

Here it is, addresses have been changed for anonymity.

I am convinced that PBR is working fine but that the problem is basic routing. I don't know how to set a route that would direct any traffic reaching the fast interface out to any IP address. And that is complicated by it being a pppoe-defined interface without a directly connected next-hop IP address.

So no matter what I do in PBR, the route lookup will always send out the default (slow) interface.

Thanks so much for helping… this is not my "day job" so I'm over my head.

Here is the output you wanted

Steve Gaede — Sat, 14 May 2016 19:51:29 GMT

Here is the output you wanted to see:

# show int ip brief
Interface IP-Address OK? Method Status Protocol
Virtual0 127.1.0.1 YES unset up up
GigabitEthernet1/1 xxx.xxx.xxx.74 YES CONFIG up upGigabitEthernet1/2 yyy.yyy.yyy.209 YES manual up up
GigabitEthernet1/3 dmz YES CONFIG up up
GigabitEthernet1/4 192.168.21.4 YES CONFIG down down
GigabitEthernet1/5 unassigned YES unset down down
GigabitEthernet1/6 unassigned YES unset down down
GigabitEthernet1/7 unassigned YES unset down down
GigabitEthernet1/8 inside YES CONFIG up up
Internal-Control1/1 127.0.1.1 YES unset up up
Internal-Data1/1 unassigned YES unset up up
Internal-Data1/2 unassigned YES unset up up
Internal-Data1/3 unassigned YES unset up up
Management1/1 unassigned YES unset up up
~

Thanks,

Francesco Molino — Sun, 15 May 2016 00:49:02 GMT

Thanks,

I don't have any pppoe connection right here. I'm trying to mount my Cisco Virl lab up and simulate a pppoe connection in order to test.

Maybe some issues with ASA and pppoe. Have you raised a TAC case?

I've never tried with pppoe connection. and even if you are doing a set route for the pppoe, as you already have a default route it will not be installed.

Try to see with TAC and in the mean time, I'll try to build up my lab soon.

I've an issue with my virl server. Waiting for Cisco feedback and then I'll build up the lab to test. In the mean time, are you able to configure a fix IP for test pppoe connection? You can set a old router as pppoe server.

Thanks

Thanks so much for taking an

Steve Gaede — Sun, 15 May 2016 19:13:42 GMT

Thanks so much for taking an interest in this. I'm starting think that this is a problem with pppoe and routing and not PBR.

I trie switching the default route to the fast ISP / pppoe connection and found problems with the traffic PBR routed out the slow interface, so I tried switching back. ASDM would not let me set the default route to the slow interface because of a "conflict with existing routes." The state was no static route at all so I essentially had a brick but I can configure it inbound over the fast interface. I think that there is an implicit default route out the fast/pppoe interface now but it doesn't show up in the static IP list (doing this through ASDM)

I'll raise a TAC case if nothing in this situation sounds familiar to anyone.

Hi

Francesco Molino — Sun, 15 May 2016 23:41:50 GMT

I've tried it in my virtual lab (Cisco VIRL) and I'm facing the exact same issue when my interface out is configured as PPPOE. Be careful, virtual is not the same as reality, however as we have exact same issue and we've not copied our configs, I think you need to raise a TAC case.

Please come back to us with TAC solution if possible.

Thanks so much

Wow, thanks for re-creating

Steve Gaede — Sun, 15 May 2016 23:51:53 GMT

Wow, thanks for re-creating the problem.

I see what happened when I removed the default route to the slow ISP, the route out the pppoe interface became the default route because of "ip address pppoe set route." The other thing that happened is that I believe (and I didn't watch for long before I tried restoring the default route) that inbound traffic to the DMZ over the slow interface was being routed out (and dropped) on the fast interface.

To your knowledge, should PBR be able to override the default route? It seems like if you are setting the next-hop address you are declaring that you know what route to use and the ASA shouldn't reevaluate what route the packet takes. Clearly if I can get packets out to the next-hop addresses defined by PBR, the upstream routers will know what to do with them.

The goal of pbr is to forward

Francesco Molino — Mon, 16 May 2016 00:45:22 GMT

The goal of pbr is to forward specific traffic out to a different next hop without taking into account the default route. However, the next hop has to be reachable.

PBR will not erase the default route because others will use it if not matching pbr acl.

in your case asa seems to not be able to get out to pppoe interface and lokking towards the routing table. If you test with 2 fixed ip addresses (no pppoe) it will work.

generally we use pbr to forward specific traffic based on source ip and destination port (https, https,....) out to a specific next hop. For example, you want specific traffic to be redirected out to a proxy; in this case pbr is usefull. This is the test I've done but asa don't work as expected, maybe a limitation (but not seen on release notes). TAC should debug it in details if a bug or they will say that's a limitation.

It's along time I'm using pbr but never with pppoe ISP. I never gate such specific case.

Keep us informed about TAC solution.

Thanks

First word back from TAC is

Steve Gaede — Mon, 16 May 2016 15:42:04 GMT

First word back from TAC is that PBR+pppoe as the secondary route is a known limitation but not documented as such.

We are supposed to be able to allow the pppoe interface to have the primary route and use the static (slow in this example) be a backup route with a higher metric.

When I get this going I will report back.

Ok thanks. This is what I've

Francesco Molino — Mon, 16 May 2016 17:02:23 GMT

Ok thanks. This is what I've configured yesterday but not yet tested (run out of time, sorry).

But quite sure that it will works in that way

I have it configured and

Steve Gaede — Wed, 18 May 2016 16:52:57 GMT

I have it configured and running with great help from TAC.

As I noted earlier, the pppoe interface cannot be the secondary interface, most likely because it sets a default route with metric 1 when it finds out what the next hop is. This also makes some engineering sense to me because if I configure PBR to set the next hop out pppoe, and the upstream provider changes the next hop address, my configuration would be broken.

The solution is to allow pppoe (fast in the diagram) to set the default route (ip address pppoe setroute), and use PBR to send any exceptions to the static (slow) interface. This is straightforward because the rules you're creating are just the inverse of using PBR to route out the fast interface. With the fast interface as the default, I just don't bother assigning a next hop for anything I want to go out that route.

The non-intuitive part for me is this: I would think that by setting next hop in PBR, the ASA would leave the route alone because clearly if I set the next hop I know where the packet should be routed. But after setting the next hop to be the slow interface, and the next-hop address is correctly set, the routing tables are still consulted and the packet is sent out the fast interface because it has the default route.

So you need to have a secondary default route with a higher metric to allow packets with next-hop set through PBR to go out the slow interface. So based on the updated diagram attached, I have:

route slow 0.0.0.0 0.0.0.0 xxx.xxx.xxx.73 2

Hi

Francesco Molino — Wed, 18 May 2016 19:29:32 GMT

Thanks for your comment. This last configs is for sure the one that was working and tested yesterday on my lab.

However, does Cisco TAC talked about roadmap and/or best practices?

Because in some specific cases, we wouldn't have the pppoe as primary and just as backup. In that design, PBR will not work. Workaround is to do what you've done today to make it works,

Thanks

No talk about roadmap but

Steve Gaede — Wed, 18 May 2016 19:46:36 GMT

No talk about roadmap but best practices are basically what we implemented.

I think that what you are attempting to do can work just fine, if you want to treat the pppoe as a backup then use PBR to route everything out the non-pppoe interface. I haven't experimented with this, but it seems that you could use "configure next hop verify ability" under Edit Route Map in ASDM to verify that your next hop is available and if it isn't the next hop would not be set and your traffic would fall back to the pppoe interface.

I can see that there are pitfalls in trying to make pppoe primary: you would have to use a fixed address for the next hop based on the routing provided by the ISP. If that routing changes, the pppoe configuration would change but your router would not, so your next-hop IP address would take you nowhere.

A future study question is also what to do if you have two pppoe interfaces. It seems that the best you could do with PBR is to set the next hop interface. But I'm not sure if that would work because you might have two interfaces competing to set a default route with metric 1. Beyond where I'm going with my configuration.

The default route is to the

Mon, 17 Jul 2017 02:24:43 GMT

The default route is to the slow ISP,not use 0.0.0.0 0.0.0.0;

change to this two route 0.0.0.0 128.0.0.0 and 128.0.0.0 128.0.0.0

Re: First word back from TAC is

eduard.hoffmann — Wed, 12 Jan 2022 09:59:47 GMT

>> First word back from TAC is that PBR+pppoe as the secondary route is a known limitation but not documented as such.

TAC was wrong. It works with PPPoE.

For Example:

route-map RM-NAME permit 10
match ip address ACL_NAME
set interface outside_2

Re: First word back from TAC is

robertschols — Tue, 23 Apr 2024 18:40:28 GMT

@eduard.hoffmann wrote:
TAC was wrong. It works with PPPoE.
For Example:
route-map RM-NAME permit 10
match ip address ACL_NAME
set interface outside_2

This worked beautifully.

(B.t.w. I was not able to find an equivalent for this in ASDM, even though the ASDM allows you to configure many other aspects of PBR. I had to run this "set interface outside_etc" from the CLI.)