ASR1001X PBR doesnt work

[andy!doesnt!like!uucp]

Hello,

recently i implemented some traffic distribution over two tunnels (Inet & MPLS) to decrease load on MPLS path.

Deployment is pretty simple: spok with data susbnet 172.21.44/24 attached has 2 connections (MPLS & Inet). HUB & spok has per 1 DMVPN tunnel over each underlay. EIGRP is running over tunnels. So routing solution was to setup static route on the HUB router toward data subnet in the SPOK via Inet-tunnel (mainly it was important on HUB side as 2 proxies 192.168.2.100 & 105 there was generating considerable amount of traffic toward data subnet in the SPOK). It works perfectly, but i've found too much bandwidth unutilized on MPLS-tunnel with use of static route. So further tunning was to implement PBR on the HUB router to route into Inet-tunnel only replies from proxies to data subnet in SPOK. Seemed to be a piece of cake. IP SLA, track object , access list & route-map was implemented & installed on router's interfaces where proxies appear to be behind. sho route-map confirms traffic hits... but no route is policed. traffic takes path over MPLS-tunnel :/

It was confirmed by traffic capture applied to the Inet-tunnel interface. With PBR traffic from proxies gets to MPLS-tunnel instead of Inet-tunnel. So i returned to unfair but working static route. Investigations on possible restrictions/limitations didnt help. So i wonder did anybody have the similar behavior?

Below is pieces of config:

PBR (doesnt work)

access-list 101 remark PBR
access-list 101 permit ip host 192.168.2.100 172.21.44.0 0.0.0.255
access-list 101 permit ip host 192.168.2.105 172.21.44.0 0.0.0.255

route-map PBR_4_ULM permit 10
 match ip address 101
 set ip next-hop verify-availability 172.31.12.17 1 track 101

rtr-01#sho ip policy
Interface Route map
Po1.2088 PBR
Po1.2089 PBR

rtr-01#sho route-map PBR
route-map PBR, permit, sequence 10
Match clauses:
ip address (access-lists): 101
Set clauses:
ip next-hop verify-availability 172.31.12.17 1 track 101 [up]
Policy routing matches: 4416660 packets, 2085890679 bytes

Static route (working):

ip route 172.21.44.0 255.255.255.0 172.31.12.17 track 101

[Georg Pauwen]

Hello,

weird indeed. Looking at your original post, and maybe I have looked at it for too long, I see:

route-map PBR_4_ULM permit 10
match ip address 101
set ip next-hop verify-availability 172.31.12.17 1 track 101

rtr-01#sho ip policy
Interface Route map
Po1.2088 PBR
Po1.2089 PBR

rtr-01#sho route-map PBR
route-map PBR, permit, sequence 10
Match clauses:
ip address (access-lists): 101
Set clauses:
ip next-hop verify-availability 172.31.12.17 1 track 101 [up]
Policy routing matches: 4416660 packets, 2085890679 bytes

Are these supposed to be the same route maps, PBR and PBR_4_ULM ?

[Georg Pauwen]

Also, post the output of 'show ip route' when the static route is not configured.

[andy!doesnt!like!uucp]

I made a notion above ~this piece. There was from different map. Effective config is 
route-map PBR permit 10
match ip address 101
set ip next-hop verify-availability 172.31.12.17 1 track 101

Below is route with removed static
Routing entry for 172.21.44.0/22
Known via "eigrp 10", distance 90, metric 76800640, type internal
Redistributing via nhrp, eigrp 10
Last update from 172.31.8.17 on Tunnel20, 00:12:52 ago
Routing Descriptor Blocks:
* 172.31.8.17, from 172.31.8.17, 00:12:52 ago, via Tunnel20
Route metric is 76800640, traffic share count is 1
Total delay is 50001 microseconds, minimum bandwidth is 100 Kbit
Reliability 255/255, minimum MTU 1400 bytes
Loading 255/255, Hops 1

[Georg Pauwen]

Hello,

what I meant was to show the entire routing table, not only for that specific entry. If you look at the entire routing table, is there an entry at all for 172.31.12.17, that is, is that next hop known in the routing table ?

[andy!doesnt!like!uucp]

Sure:

#sho ip route 172.31.12.17
Routing entry for 172.31.12.0/24
Known via "connected", distance 0, metric 0 (connected, via interface)
Redistributing via nhrp, eigrp 10
Routing Descriptor Blocks:
* directly connected, via Tunnel30
Route metric is 0, traffic share count is 1

I wonder currently what new command
debug platform hardware qfp active feature pbr datapath ipv4 interface port-channel 1.2089 can bring for me? i see it 1st time. do anybody have idea about potencial impacts? cannot find relevant info 

[andy!doesnt!like!uucp]

& sorry : links between rtrs & FW are marked with mistake. Po1.2088 & Po1.2089 label must be shifted. Fixed scheme is in attach.

[andy!doesnt!like!uucp]

Hi Guys,

i managed to debug pbr with debug platform hardware qfp active feature pbr datapath ipv4 interface port-channel 1.2089 command

output shows interesting traffic has its route policed (see below). But it make situation even worst, because as soon as i remove static route traffic takes not desired path :(((

#more flash:tracelogs/cpp_cp_F0-0.log.24983.20170719130154
07/19 13:01:54.062 : btrace continued for process ID 24983 with 159 modules

07/19 13:01:54.062 [cpp-dp]: (verbose): QFP:0.0 Thread:021 TS:00017236539991968334 PBR: intf Port-channel1.2089, src 192.168.2.100, dst 172.21.44.53, ipv4 next-hop 172.31.12.17 lookup okay, table_id 0
07/19 13:01:54.062 [cpp-dp]: (verbose): QFP:0.0 Thread:021 TS:00017236539991977126 PBR: intf Port-channel1.2089, src 192.168.2.100, dst 172.21.44.53, ipv4 set next-hop 172.31.12.17 routed, table_id 0, PBR exit
07/19 13:01:54.062 [cpp-dp]: (verbose): QFP:0.0 Thread:013 TS:00017236539991879474 PBR: intf Port-channel1.2089, src 192.168.2.100, dst 172.21.44.53, TCAM hit, result_addr 28dec2c0
07/19 13:01:54.063 [cpp-dp]: (verbose): QFP:0.0 Thread:013 TS:00017236539991904808 PBR: intf Port-channel1.2089, src 192.168.2.100, dst 172.21.44.53, flags 42, tos 0, prec 0 nh_or_adj ac1f0c11 (172.31.12.17), new_table_id 0, stats_addr 3447c610, color_list 0, num_color 0
07/19 13:01:54.063 [cpp-dp]: (verbose): QFP:0.0 Thread:013 TS:00017236539991956108 PBR: intf Port-channel1.2089, src 192.168.2.100, dst 172.21.44.53, ipv4 next-hop lookup 172.31.12.17, table_id 0
07/19 13:01:54.063 [cpp-dp]: (verbose): QFP:0.0 Thread:013 TS:00017236539991966524 PBR: intf Port-channel1.2089, src 192.168.2.100, dst 172.21.44.53, ipv4 next-hop 172.31.12.17 lookup okay, table_id 0
07/19 13:01:54.063 [cpp-dp]: (verbose): QFP:0.0 Thread:013 TS:00017236539991976252 PBR: intf Port-channel1.2089, src 192.168.2.100, dst 172.21.44.53, ipv4 set next-hop 172.31.12.17 routed, table_id 0, PBR exit

[Georg Pauwen]

Is Port-channel1.2089 configured for HSRP as well ? Either way, if you apply policy routing on those interfaces, their IP addresses (or the standby IP address) are/is not in the same subnet as 172.31.12.17, so 172.31.12.17 cannot be the next hop.

On which (of the two) device(s) did you configure the static route ?

[andy!doesnt!like!uucp]

No HSRP there, po1.2089 is for interconnect purpose between rtr-left & rtr-right only. EIGRP is running on it as well. So rtr-left learns route to target subnet from rtr-right & calculates it as best.

I'm not clear about your statement "Either way, if you apply policy routing on those interfaces, their IP addresses (or the standby IP address) are/is not in the same subnet as 172.31.12.17, so 172.31.12.17 cannot be the next hop". Could u pls rephrase?

& finally: static route is on the rtr-right. rtr-right is terminating tu20 & tu30 (routes learned via tu20 have better metric vs those learned over tu30). 

But why r u interesting in static? when i apply PBR i remove static, to allow PBR to make his granular work.