ASR1001X PBR doesnt work

[andy!doesnt!like!uucp]

Hello,

recently i implemented some traffic distribution over two tunnels (Inet & MPLS) to decrease load on MPLS path.

Deployment is pretty simple: spok with data susbnet 172.21.44/24 attached has 2 connections (MPLS & Inet). HUB & spok has per 1 DMVPN tunnel over each underlay. EIGRP is running over tunnels. So routing solution was to setup static route on the HUB router toward data subnet in the SPOK via Inet-tunnel (mainly it was important on HUB side as 2 proxies 192.168.2.100 & 105 there was generating considerable amount of traffic toward data subnet in the SPOK). It works perfectly, but i've found too much bandwidth unutilized on MPLS-tunnel with use of static route. So further tunning was to implement PBR on the HUB router to route into Inet-tunnel only replies from proxies to data subnet in SPOK. Seemed to be a piece of cake. IP SLA, track object , access list & route-map was implemented & installed on router's interfaces where proxies appear to be behind. sho route-map confirms traffic hits... but no route is policed. traffic takes path over MPLS-tunnel :/

It was confirmed by traffic capture applied to the Inet-tunnel interface. With PBR traffic from proxies gets to MPLS-tunnel instead of Inet-tunnel. So i returned to unfair but working static route. Investigations on possible restrictions/limitations didnt help. So i wonder did anybody have the similar behavior?

Below is pieces of config:

PBR (doesnt work)

access-list 101 remark PBR
access-list 101 permit ip host 192.168.2.100 172.21.44.0 0.0.0.255
access-list 101 permit ip host 192.168.2.105 172.21.44.0 0.0.0.255

route-map PBR_4_ULM permit 10
 match ip address 101
 set ip next-hop verify-availability 172.31.12.17 1 track 101

rtr-01#sho ip policy
Interface Route map
Po1.2088 PBR
Po1.2089 PBR

rtr-01#sho route-map PBR
route-map PBR, permit, sequence 10
Match clauses:
ip address (access-lists): 101
Set clauses:
ip next-hop verify-availability 172.31.12.17 1 track 101 [up]
Policy routing matches: 4416660 packets, 2085890679 bytes

Static route (working):

ip route 172.21.44.0 255.255.255.0 172.31.12.17 track 101

[Georg Pauwen]

Hello,

better post the full configuration, so we can see the SLA and the interface the route map is applied to as well...

[noc-germany1]

Hi Georg & thank u for input.

Actually it's unlikely possible because of huge config of the router. Also in previous post there was an error in output of sho route-map. Correct is:
route-map PBR, permit, sequence 10
Match clauses:
ip address (access-lists): 101
Set clauses:
ip next-hop verify-availability 172.31.12.17 1 track 101 [up]
Nexthop tracking current: 172.31.12.17
172.31.12.17, fib_nh:7F89473A1810,oce:7F8947349F08,status:1

Policy routing matches: 23278776 packets, 12908045008 bytes

Below some relevant pieces of config in addition:

track 101 ip sla 101
ip sla 101
icmp-echo 172.31.12.17
ip sla schedule 101 life forever start-time now

EIGRP route w/o static:

rtr-01#sho ip route 172.21.44.1
Routing entry for 172.21.44.0/22
Known via "eigrp 10", distance 90, metric 76800640, type internal
Redistributing via nhrp, eigrp 10
Last update from 172.31.8.17 on Tunnel20, 00:00:05 ago
Routing Descriptor Blocks:
* 172.31.8.17, from 172.31.8.17, 00:00:05 ago, via Tunnel20
Route metric is 76800640, traffic share count is 1
Total delay is 50001 microseconds, minimum bandwidth is 100 Kbit
Reliability 255/255, minimum MTU 1400 bytes
Loading 255/255, Hops 1

Interfaces:

interface Tunnel30
description DMVPN #1 via Inet
ip address 172.31.12.1 255.255.255.0
ip helper-address 10.0.43.36
no ip redirects
ip mtu 1400
ip wccp 61 redirect in
ip wccp 61 redirect out
ip wccp 62 redirect in
ip wccp 62 redirect out
ip nhrp map multicast dynamic
ip nhrp network-id 30
ip nhrp holdtime 60
ip nhrp redirect
ip tcp adjust-mss 1360
delay 5010
qos pre-classify
tunnel source Port-channel1.2088
tunnel mode gre multipoint
tunnel key 30
tunnel bandwidth transmit 50000
tunnel bandwidth receive 50000
tunnel protection ipsec profile dmvpn-inet
end

interface Port-channel1.2088
description Transfer zur FW
encapsulation dot1Q 2088
ip address 172.31.0.28 255.255.255.248
standby version 2
standby 2088 ip 172.31.0.30
standby 2088 preempt delay minimum 300 reload 900 sync 1800
standby 2088 authentication *****
ip policy route-map PBR
service-policy input policy-lan-ingress
end

!Below interface is toward HSRP Active in stdby 2088 (replies are forwarded to & come from there)

interface Port-channel1.2089
description r1-r2-interco
encapsulation dot1Q 2089
ip address 172.31.0.33 255.255.255.248
ip policy route-map PBR
end

[Georg Pauwen]

Hello,

sorry for the confusion, I cannot figure out your setup with the information provided. Where is 172.31.12.17, the next hop for the route map ? Is that the other end of Tunnel 30 ?

Can you provide a schematic drawing of your setup ?

[andy!doesnt!like!uucp]

Hello Georg,

topo is pretty simple (pls find in attach). 172.31.12.17 is Tu30 end on the north rtr, as well 172.31.8.17 is tu20 end on the nort rtr. rtr on the left recieves replies from server because it's default GW for FW.

[paul driver]

Hello

Curious, Have you tried specifying the interface instead of the next-hop or just the next-hop without the verify-reachability.

Can you also put on a debug for PBR and post the results

debug ip policy

res
Paul

[andy!doesnt!like!uucp]

Hi Paul
implementing just ïp next-hop 172.31.12.17"w/o verify-reachability didnt change behavior.

With regard to configuring interface instead of next-hop, i dont think it will work at all because of well known effect of ARPing will not work for destination address over the multipoint tunnel. 

[paul driver]

Hello

wha does th debug show 

Depending on what iOS your using. Cef may need to be disabled

Maybe try tht also

res

Paul

[andy!doesnt!like!uucp]

Hello Paul

debug force output to appear only when map is changed

Below i delete&add again map sequence 20 i added recently for the additional tests purpose

#sho debug
IOSXE Conditional Debug Configs:
Conditional Debug Global State: Stop
IOSXE Packet Tracing Configs:
Policy Routing:
Policy routing debugging is on
#no route-map PBR permit 20
#
Jul 19 09:18:05.943 UTC: PBR Nexthop: Unregister tracking for 172.31.8.17, handle: 7F89473A1360
Jul 19 09:18:05.943 UTC: PBR Nexthop Callback invoked: 7F893004E298, (172.31.8.17) tableid 0, status: 0, type: SET NEXTHOPmap: PBR, sequence: 20
Jul 19 09:18:05.943 UTC: PBR Control Plane Notification: PBR_CP_INVALID - shut off all
Jul 19 09:18:05.943 UTC: Policy NextHop Inquiry: PBR seq: 20, type: INVALID SW_OBJ_TYPE: 0, SW_HANDLE: 0
Jul 19 09:18:05.943 UTC: PBR CP Notification sent: Type:INVALID, UNKNOWNSW_OBJ_TYPE: 0, SW_HANDLE: 0
Jul 19 09:18:05.943 UTC: PBR Control Plane Notification: PBR_CP_INVALID - shut off all
Jul 19 09:18:05.943 UTC: Policy NextHop Inquiry: PBR seq: 20, type: INVALID SW_OBJ_TYPE: 0, SW_HANDLE: 0
Jul 19 09:18:05.943 UTC: PBR CP Notification sent: Type:INVALID, UNKNOWNSW_OBJ_TYPE: 0, SW_HANDLE: 0
Jul 19 09:18:05.943 UTC: PBR Nexthop: Unregister tracking for 172.31.8.17, handle: 0
Jul 19 09:18:05.944 UTC: PR-RP: Set Port-channel1.2088 policy_routemap=PBR; cached_map=PBR
Jul 19 09:18:05.944 UTC: PR-RP: Set Port-channel1.2089 policy_routemap=PBR; cached_map=PBR
#route-map PBR permit 20
# match ip address PBR_ADD
# set ip next-hop 172.31.8.17
#
Jul 19 09:24:00.346 UTC: PR-RP: Set Port-channel1.2088 policy_routemap=PBR; cached_map=PBR
Jul 19 09:24:00.346 UTC: PR-RP: Set Port-channel1.2089 policy_routemap=PBR; cached_map=PBR
Jul 19 09:24:00.569 UTC: PR-RP: Set Port-channel1.2088 policy_routemap=PBR; cached_map=PBR
Jul 19 09:24:00.569 UTC: PR-RP: Set Port-channel1.2089 policy_routemap=PBR; cached_map=PBR
Jul 19 09:24:00.584 UTC: PBR Nexthop Callback invoked: 7F89475E4950, (172.31.8.17) tableid 0, status: 2, type: SET NEXTHOPmap: PBR, sequence: 20
Jul 19 09:24:00.584 UTC: PBR Control Plane Notification: 172.31.8.17 PBR_CP_SET_NEXTHOP
Jul 19 09:24:00.584 UTC: Policy NextHop Inquiry: PBR seq: 20, type: SET NEXTHOP Nexthop: 172.31.8.17SW_OBJ_TYPE: 1D, SW_HANDLE: 7F89473475C8
Jul 19 09:24:00.584 UTC: PBR CP Notification sent: Type:SET NEXTHOP, 172.31.8.17SW_OBJ_TYPE: 1D, SW_HANDLE: 7F89473475C8
Jul 19 09:24:00.584 UTC: PBR Nexthop: Register tracking for 172.31.8.17, handle: 7F89473A1360
Jul 19 09:24:00.785 UTC: PR-RP: Set Port-channel1.2088 policy_routemap=PBR; cached_map=PBR
Jul 19 09:24:00.785 UTC: PR-RP: Set Port-channel1.2089 policy_routemap=PBR; cached_map=PBR

No familiar PBR debug appears. Which makes me think that policy is still not in effect. But why i can see hits against map? :)

IOS/Platform:

#sho ver
Cisco IOS XE Software, Version 03.13.02.S - Extended Support Release
Cisco IOS Software, ASR1000 Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 15.4(3)S2, RELEASE SOFTWARE (fc3)
System image file is "bootflash:/asr1001x-universalk9.03.13.02.S.154-3.S2-ext.SPA.bin"
License Level: adventerprise
License Type: Permanent
cisco ASR1001-X (1NG) processor (revision 1NG) with 3751704K/6147K bytes of memory.
Processor board ID FXS1836Q096
6 Gigabit Ethernet interfaces
2 Ten Gigabit Ethernet interfaces

[Georg Pauwen]

Hello,

I am wondering if you simply run into a PBR limitation. I found the below in the guidelines and limitations for policy-based routing:

Setting a tunnel interface or an IP address via a tunnel interface as a next hop in a policy-based routing policy is not supported.

Here is the link to the document that I got this from (it is for NX-OS, but I seem to recall that these are general PBR guidelines):

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/sw/5_x/nx-os/unicast/configuration/guide/l3_cli_nxos/l3_pbr.html#58316

[andy!doesnt!like!uucp]

Hi Georg,
thank u for the link. it would assemble my broken mind if platform were NX-OS.

Now i can state that PBR doesnt work as expected in my environment. take a look at below diagnosis - traceroute was done during pbr debugging on the policing rtr (note that i added second map for tests using 172.31.0.38 as src, & this map set NH 172.31.8.17 behind tun20 to override static route i'm using currently)

#traceroute 172.21.44.53

+++

  1 172.31.0.33 0 msec 1 msec 0 msec

  2 172.31.12.17 30 msec 29 msec 28 msec //traffic follows static route!!!

  3 172.21.47.137 29 msec 28 msec 29 msec

  4  *  *

07/19 13:25:56.396 [cpp-dp]: (verbose): QFP:0.0 Thread:044 TS:00017237982332111072 PBR: intf Port-channel1.2089, src 172.31.0.38, dst 172.21.44.53, TCAM hit, result_addr 28dec300

07/19 13:25:56.396 [cpp-dp]: (verbose): QFP:0.0 Thread:044 TS:00017237982332137858 PBR: intf Port-channel1.2089, src 172.31.0.38, dst 172.21.44.53, flags 42, tos 0, prec 0 nh_or_adj ac1f0811 (172.31.8.17), new_table_id 0, stats_addr 3447c630, color_list 0, num_color 0

07/19 13:25:56.396 [cpp-dp]: (verbose): QFP:0.0 Thread:044 TS:00017237982332191708 PBR: intf Port-channel1.2089, src 172.31.0.38, dst 172.21.44.53, ipv4 next-hop lookup 172.31.8.17, table_id 0

07/19 13:25:56.396 [cpp-dp]: (verbose): QFP:0.0 Thread:044 TS:00017237982332203102 PBR: intf Port-channel1.2089, src 172.31.0.38, dst 172.21.44.53, ipv4 next-hop 172.31.8.17 lookup okay, table_id 0

07/19 13:25:56.396 [cpp-dp]: (verbose): QFP:0.0 Thread:044 TS:00017237982332213776 PBR: intf Port-channel1.2089, src 172.31.0.38, dst 172.21.44.53, ipv4 set next-hop 172.31.8.17 routed, table_id 0, PBR exit

[Georg Pauwen]

Hello,

actually, I am pretty sure that this is a PBR limitation. I faintly remember that being an exam question in one of the old CCNP exams, so I googled and found the link below:

http://vceguide.com/which-three-guidelines-and-limitation-of-policy-based-routing-are-true/

[andy!doesnt!like!uucp]

Thank u man. I think u r right.

[Georg Pauwen]

Annoying indeed, one would think that NX (next generation) would have added that feature...

[andy!doesnt!like!uucp]

well... i think i cant see familiar PBR debug output because it's implemented in hardware in my case. I'm faced with no way to verify whether PBR in effect or not :)