Policy Routing failure (order of operation?)

[Jay47110]

 

Hi,

This question is relating the order of operation in Cisco IOS-XE for an ingress packet.

I am currently troubleshooting an issue where a Host1 with IP 192.168.1.1 is trying to ping a remote Host2 192.168.2.2 and the router in question R1 is the transit router.

The traffic is PBR'd on R1 on the ingress interface based on source IP 192.168.1.1 and destination IP 192.168.2.2 and an IP next-hop is set.

Here is the trouble, 192.168.2.2 also happens to be a directly connected interface IP address on R1 which is completely separate and has nothing to do with Host2's 192.168.2.2 that host1 is trying to ping. And Although the traffic is PBR'd on the ingress interface, it never gets Policy based routed to the correct IP next-hop specified in the PBR. Here is the "debug ip packet detail" output of the failed routing:

*Dec 15 19:22:22.190: IP: s=192.168.1.1 (GigabitEthernet0/1), d=192.168.2.2, len 100, input feature
*Dec 15 19:22:22.191: ICMP type=0, code=0, Policy Routing(103), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Dec 15 19:22:22.191: IP: s=192.168.1.1 (GigabitEthernet0/1), d=192.168.2.2, len 100, input feature
*Dec 15 19:22:22.192: ICMP type=0, code=0, MCI Check(109), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Dec 15 19:22:22.192: FIBipv4-packet-proc: route packet from GigabitEthernet0/1 src 192.168.1.1 dst 192.168.2.2
*Dec 15 19:22:22.193: FIBfwd-proc: Default:192.168.2.2/32 receive entry
*Dec 15 19:22:22.193: FIBipv4-packet-proc: packet routing failed
*Dec 15 19:22:22.193: IP: tableid=0, s=192.168.1.1 (GigabitEthernet0/1), d=192.168.2.2 (GigabitEthernet0/2), routed via RIB
*Dec 15 19:22:22.194: IP: s=192.168.1.1 (GigabitEthernet0/1), d=192.168.2.2, len 100, rcvd 4
*Dec 15 19:22:22.194: ICMP type=0, code=0
*Dec 15 19:22:22.195: IP: s=192.168.1.1 (GigabitEthernet0/1), d=192.168.2.2, len 100, stop process pak for forus packet
*Dec 15 19:22:22.195: ICMP type=0, code=0

However, once the directly connected interface with IP 192.168.2.2 is shutdown, PBR kicks in fine and routes traffic to the correct next-hop interface. Here is the "debug ip packet detail" output for successful routing:

*Dec 15 19:21:25.785: IP: s=192.168.1.1 (GigabitEthernet0/1), d=192.168.2.2, len 100, FIB policy match
*Dec 15 19:21:25.785: IP: s=192.168.1.1 (GigabitEthernet0/1), d=192.168.2.2, len 100, PBR Counted
*Dec 15 19:21:25.785: IP: s=192.168.1.1 (GigabitEthernet0/1), d=192.168.2.2, g=192.168.12.2, len 100, FIB policy routed

What am I missing here, isn't Policy based routing triggered before RIB or CEF in the ingress order of operations? Or is the fact that the destination is also a directly connected prefix takes precedence over the PBR?

Kind regards

 

[MHM Cisco World]

are you config PBR with default keyword ?

[Jay47110]

Hi MHM,

Can you kindly explain what the default keyword is for? what it achieves?

Kind regards

[Georg Pauwen]

Hello,

post the full running configuration of R1.  You are right, according to the Cisco IOS Order of Operations, policy routing always happens BEFORE regular routing. Not sure how that works when the next hop is the same as a directly connected interface.

[Jay47110]

Hi George,

Yes that's what I expected to see i.e., PBR taking precedence over regular routing. I've attached a topology diagram for ease of understanding, it also mentions the route-map statement.

Config wise there is nothing special at all. just IP addresses on interfaces and a route-map.

[Georg Pauwen]

Hello,

I just lab tested this, and PBR works as expected (see 'debug ip pbr' output below), so I suspect your result is platform and/or configuration specific. Post the running config, maybe we can spot something.

R1#debug ip policy
Policy routing debugging is on
R1#
*Dec 16 11:00:17.246: IP: s=192.168.1.1 (GigabitEthernet0/1), d=192.168.2.2, len 84, FIB policy match
*Dec 16 11:00:17.246: IP: s=192.168.1.1 (GigabitEthernet0/1), d=192.168.2.2, len 84, PBR Counted
*Dec 16 11:00:17.246: IP: s=192.168.1.1 (GigabitEthernet0/1), d=192.168.2.2, g=10.12.12.2, len 84, FIB policy routed
R1#
*Dec 16 11:00:18.275: IP: s=192.168.1.1 (GigabitEthernet0/1), d=192.168.2.2, len 84, FIB policy match
*Dec 16 11:00:18.275: IP: s=192.168.1.1 (GigabitEthernet0/1), d=192.168.2.2, len 84, PBR Counted
*Dec 16 11:00:18.275: IP: s=192.168.1.1 (GigabitEthernet0/1), d=192.168.2.2, g=10.12.12.2, len 84, FIB policy routed
*Dec 16 11:00:19.195: IP: s=192.168.1.1 (GigabitEthernet0/1), d=192.168.2.2, len 84, FIB policy match
*Dec 16 11:00:19.195: IP: s=192.168.1.1 (GigabitEthernet0/1), d=192.168.2.2, len 84, PBR Counted
*Dec 16 11:00:19.195: IP: s=192.168.1.1 (GigabitEthernet0/1), d=192.168.2.2, g=10.12.12.2, len 84, FIB policy routed
R1#
*Dec 16 11:00:20.040: IP: s=192.168.1.1 (GigabitEthernet0/1), d=192.168.2.2, len 84, FIB policy match
*Dec 16 11:00:20.040: IP: s=192.168.1.1 (GigabitEthernet0/1), d=192.168.2.2, len 84, PBR Counted
*Dec 16 11:00:20.040: IP: s=192.168.1.1 (GigabitEthernet0/1), d=192.168.2.2, g=10.12.12.2, len 84, FIB policy routed
*Dec 16 11:00:21.028: IP: s=192.168.1.1 (GigabitEthernet0/1), d=192.168.2.2, len 84, FIB policy match
*Dec 16 11:00:21.028: IP: s=192.168.1.1 (GigabitEthernet0/1), d=192.168.2.2, len 84, PBR Counted
*Dec 16 11:00:21.028: IP: s=192.168.1.1 (GigabitEthernet0/1), d=192.168.2.2, g=10.12.12.2, len 84, FIB policy routed

[Jay47110]

Hi George,

I've attached the running configs of R1 and R2 in the original post.

In your lab, do you also have a directly connected interface hosting the IP 192.168.2.2?

What I see happening in my lab is that, when I ping from Host1 to Host2, Pings are successful but only because R1's directly connected interface is responding to them and not Host2. And when I ping from Host2 to Host1, pings fail since the R1 is not PBRing traffic towards Host2. Once when I shut down the directly connected 192.168.2.2 interface does the PBR actually work.

Kind regards

[Georg Pauwen]

Hello,

--> In your lab, do you also have a directly connected interface hosting the IP 192.168.2.2?

Yes.

I'll have a look at the configs...

[Georg Pauwen]

Hello,

looking at the configuration of R1, it looks like you have applied a non-existing policy map to the interface ?

interface GigabitEthernet0/1
description Linkt to Host1
ip address 192.168.1.254 255.255.255.0
ip policy route-map test
duplex auto
speed auto
media-type rj45

!

route-map RM permit 10
match ip address ACL
set ip next-hop 10.12.12.2

What you need is:

interface GigabitEthernet0/1
description Linkt to Host1
ip address 192.168.1.254 255.255.255.0
ip policy route-map RM
duplex auto
speed auto
media-type rj45

[Jay47110]

Hi George,

Apologies, that was a typo made when I was posting the config to this thread. I had to rename stuff in the copied config to match the topology I mentioned in this post.

The actual config on R1 has the correct route-map name applied to the interface.

[Georg Pauwen]

Actually, the only thing that is different in my setup is that I have static default routes:

R1

ip route 0.0.0.0 0.0.0.0 10.12.12.2

R2

ip route 0.0.0.0 0.0.0.0 10.12.12.1

Can you ad these and check the results ?

[Jay47110]

Thanks for the response George.

I'm still having the same issue: here is the latest debug ip packet output when I ping from host2 to host1.

*Dec 16 12:56:44.876: IP: s=192.168.1.1 (GigabitEthernet0/1), d=192.168.2.2, len 100, input feature, Policy Routing(103), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Dec 16 12:56:44.877: IP: s=192.168.1.1 (GigabitEthernet0/1), d=192.168.2.2, len 100, input feature, MCI Check(109), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Dec 16 12:56:44.878: IP: tableid=0, s=192.168.1.1 (GigabitEthernet0/1), d=192.168.2.2 (GigabitEthernet0/2), routed via RIB

You can see that the router completely disregards the PBR on gig0/1 and routes packets via RIB to gig0/2 where 192.168.2.2 IP resides instead of forwarding it over to R2 via PBR.

Have you tried pinging from your host2 to host1 see if that yeils a different result.

btw I'm using Cisco IOSv Software (VIOS-ADVENTERPRISEK9-M), Version 15.6(2)T, RELEASE SOFTWARE (fc2) for the lab.

[Georg Pauwen]

Hello,

really odd. I have compared your and my config line by line, the only difference is the flow monitor. Maybe you can remove that, not sure what, if any, impact that has on the PBR...

flow record test
match flow direction
match ipv4 source address
!
flow monitor test
record test

[Jay47110]

Thanks for your help so far George. I put netflow on the interface just to see the traffic I was receiving on that interface. It should have no impact on the actual routing.

[Georg Pauwen]

Hello,

I am out of options ! The last thing I can think of is to wr erase your configs, reboot the routers, and put the configs in from scratch.