policy based route failover & back slow or never

[will]

Using cisco isr4300 routers. Have a simple 3-way triangle between 3 routers:

RtrA--RtrB
 |        /
 |      /
RtrC

Static route on RtrA & RtrB pointing to a RtrC loopback, via their adjacent links.
Default route on RtrC point to RtrA.
No routes exist on RtrC through RtrB.

Now I want to force RtrA to RtrC loopback traffic through RtrB.

1. RtrC: configure PBR (policy based route) with "set ip next-hop <RtrB>" for all routes sourced from RtrC loopback
2. RtrA: change RtrA static to point to RtrB, so that it directs there for RtrC loopback traffic.

What I find is if I have a continuous ping from RtrA to RtrC, sometimes PBR never kicks in, or never dies.

I ping from RtrA and remove the PBR statement but the RtrC keeps wanting to use the path through RtrB for some reason.

I think there is some sort of CEF or flow caching going on, which is forcing the PBR to stay alive even if not configured.

Anyone have an idea why caching keeps PBR active even after it is removed - especially for an active flow.

If I stop the ping and start it back up after a few seconds, it changes as expected to the different path.

It seems like the actual exit interface for the PBR needs to go down for the PBR to truly die.

[MHM Cisco World]

this is lab or real ?
if it real 
share 

debug ip policy <<- when the PBR is not work correctly 

MHM

[will]

hi, thx for quick reply., I may have mis-diagrammed the environment, assuming the problem is on the cisco-side. I have a firewall in the middle of one of the segments and I think that is potentially causing part of my problem. I will know more a bit later after doing some more testing and upgrades. and thx for the "debug ip policy" piece!

[Elliot Dierksen]

@MHM Cisco World asks a good question about whether this is a lab or production. If it is production, what is the type of routing you are trying to achieve and why? I can't say I never use PBR, but it is never my first choice because of the baggage that it carries. There may be a better way to do it than PBR.

[David Ruess]

Hello,

How do you have the PBR applied? This can help narrow down the issue. PBR can be applied in 2 ways:

Interface level - this is for traffic coming into the interface (not leaving because PBR would be ineffective as its already on the exit path)

Control-Plane - this is for traffic originated by the router that the PBR is configured on. 

It looks like you may have a loop following the logic of your configurations. You have 2 scenarios.

1. Without any PBR configured, Ping RTR-C Loopback from RTR-A loopback. No PBR is applied so it will use the static route to RTR-B and then from RTR-B use the static route to RTR-C. For return traffic RTR-C will use its default route to RTR-A

2. You apply PBR to RTR-C. Assuming you didn't use an interface PBR configuration since your sourcing the traffic from the local router. Pinging from A->C will have 0 effect on this PBR and will route as you configure statically. 

Secondly from this when you ping from C->A using the loopbacks, the PBR kicks in and routes to B...however you don't specify how RTR-B can reach the loopback of RTR-A. Either you have a route for A's loopback on B or RTR-B has a default back to C. If that's the case then B will send it back to C and since its not a PBR anymore as its not locally sourced then RTR-C will use its default to RTR-A.

Can you provide the configs of all 3 routers please?

Edit: About your last sentence stating the exit interface needing to go down before it dies: PBR routes indiscriminately. It doesn't know the next hop is down (unless you track it in the route-map which you can do). I believe as long as the route is still in the routing table (as in exit interface still up) then it will use PBR.

-David

[will]

hi folks, many thx to those chiming in. a quick update on this problem. it was in production, and related to a firewall cutover. so there was a firewall somewhat in the middle. in the end, the problem seemed to be an old IOS on one of the 43xx routers. i upgraded that and the PBR started working as expected. My problem also dealt with the traffic transiting the routers in the model (in/out 2 interfaces) and originating from the routers (i.e., loopbacks). the PBR gets applied differently in those two scenarios. in the end, i gave up on the traffic originating from the router (loopback) and just focused on transit traffic, using a PBR policy applied to the inbound router interfaces. I also created a route map which "denied" transit traffic from a router subnet to another local router subnet (1-hop routes). then i applied an allow route map to engage PBR to route everything else (default gateway), to go according to the PBR to a different path.

So in summary, i think an IOS update fixed it and for transit traffic, its working. it probably fixed it for router loopback traffic, but i didn't test that, as I deemed that traffic not critical during the cutover.