Inter VRF PBR not working on ISR4331 (IOS XE)

[remi-reszka]

Hello networking experts,

I have this setup to PBR between the IVRF and FVRF matching on ACLs and I get strange behavior from the router. I have this exact same setup in the LAB on 1921 router and it works perfectly fine but not on this 4331 router running IOS XE latest and stable release (isr4300-universalk9.03.16.01a.S.155-3.S1a-ext.SPA.bin).

First of of all "show access-list" does not show any packets matching the rules however the "show route-map" does show packets matching the PBR policy. In the LAB on 1921 I can see packets matching the ACLs statements.

When I try to "debug ip policy" I don't get any output. In the LAB I get the usual output for this debug and routing according to the PBR rules.

When configuring the PBR on 4331  I get the following error and cannot find any description to it.

Dec 18 14:50:45.500: PBR Nexthop Callback invoked: 7F8726A597F0, (192.168.20.254) tableid 5, status: 1, type: SET NEXTHOP TRACKINGmap: LB_TRAFFIC_MAP, sequence: 20

Dec 18 14:50:45.500: PBR Control Plane Notification: PBR_CP_INVALID - shut off all

Dec 18 14:50:45.500: Policy NextHop Inquiry: LB_TRAFFIC_MAP seq: 20, type: INVALID SW_OBJ_TYPE: 0, SW_HANDLE: 0

Dec 18 14:50:45.500: PBR CP Notification sent: Type:INVALID, UNKNOWNSW_OBJ_TYPE: 0, SW_HANDLE: 0

Dec 18 14:50:45.500: PBR Nexthop: Register tracking for 192.168.20.254, tableid 5, handle: 7F87269E5B98

Here is my config (again in the LAB on 1921 it works perfectly fine). Am I missing something on configuration on IOS XE software? Basically I have FVRF on each WAN interface and IVRFs on subinterfaces. The default routing to the Internet works fine between the IVRFs and both FVRFs.

ip sla 1
 icmp-echo 187.188.X.X source-interface GigabitEthernet0/0/0
 vrf WAN1
 threshold 500
 timeout 500
 frequency 1
ip sla schedule 1 life forever start-time now
ip sla 2
 icmp-echo 10.226.0.1 source-interface GigabitEthernet0/0/1
 vrf WAN2
 threshold 500
 timeout 500
 frequency 1
ip sla schedule 2 life forever start-time now
!
ip access-list extended GROUP1_ACL
 permit ip 10.10.8.0 0.0.7.255 any
 permit ip 10.10.16.0 0.0.7.255 any
 permit ip 10.10.24.0 0.0.7.255 any
 permit ip 10.10.32.0 0.0.7.255 any
 permit ip 10.10.40.0 0.0.7.255 any
 permit ip 10.10.100.0 0.0.0.255 any
 permit ip 10.26.8.48 0.0.0.15 any
 deny   ip any any
ip access-list extended GROUP2_ACL
 permit ip 10.10.60.0 0.0.0.255 any
 permit ip 10.10.70.0 0.0.0.255 any
 permit ip 10.10.80.0 0.0.0.255 any
 permit ip 10.10.90.0 0.0.0.255 any
 permit ip 10.10.110.0 0.0.0.255 any
 deny   ip any any
!
route-map LB_TRAFFIC_MAP permit 10
 description Route Group 2 over WAN1 and failover to WAN2
 match ip address GROUP2_ACL
 set ip vrf WAN1 next-hop verify-availability 187.188.X.X 1 track 1
 set ip vrf WAN2 next-hop verify-availability 192.168.20.254 2 track 2
route-map LB_TRAFFIC_MAP permit 20
 description Route Group 1 over WAN2 and failover to WAN1
 match ip address GROUP1_ACL
 set ip vrf WAN2 next-hop verify-availability 192.168.20.254 1 track 2
 set ip vrf WAN1 next-hop verify-availability 187.188.X.X 2 track 1
!
!
!
interface GigabitEthernet0/0/2.301
 description TRANSIT
 encapsulation dot1Q 301
 vrf forwarding TRANSIT
 ip address 172.16.31.2 255.255.255.240
 ip nat inside
 ip tcp adjust-mss 1412
 ip policy route-map LB_TRAFFIC_MAP
 no cdp enable
 ip virtual-reassembly
end
!

Thanks in advance for your very valuable comments.

Remi

[Masoud Pourshabanian]

Hello,

Is your IP SLA working correctly. How you checked the output of show ip sla and show track to see its status?

Masoud

[remi-reszka]

Hi Masoud,

Thank you for you reply, all the ip sla and tracking is working fine:

wan-rtr1#sh ip sla stat 1
IPSLAs Latest Operation Statistics

IPSLA operation id: 1
        Latest RTT: 1 milliseconds
Latest operation start time: 10:52:11 UTC Fri Dec 18 2015
Latest operation return code: OK
Number of successes: 1727
Number of failures: 0
Operation time to live: Forever


wan-rtr1#sh ip sla stat 2
IPSLAs Latest Operation Statistics

IPSLA operation id: 2
        Latest RTT: 4 milliseconds
Latest operation start time: 10:52:17 UTC Fri Dec 18 2015
Latest operation return code: OK
Number of successes: 2256
Number of failures: 13
Operation time to live: Forever


wan-rtr1#sh track 1
Track 1
  IP SLA 1 reachability
  Reachability is Up
    10 changes, last change 00:28:56
  Delay up 1 sec, down 1 sec
  Latest operation return code: OK
  Latest RTT (millisecs) 1
  Tracked by:
    Route Map 0
wan-rtr1#sh track 2
Track 2
  IP SLA 2 reachability
  Reachability is Up
    18 changes, last change 00:21:00
  Delay up 1 sec, down 1 sec
  Latest operation return code: OK
  Latest RTT (millisecs) 5
  Tracked by:
    Route Map 0
wan-rtr1#

I noticed however one error I had, there was no response from the next-hop 192.168.20.254 pinging from the TRANSIT IVRF. I noticed that by showing route-map:

Nexthop tracking current: 187.188.X.X
187.188.X.X, fib_nh:7F8726A29638,oce:7F871F197008,status:1

192.168.20.254, fib_nh:7F8726A29688,oce:7F87269B5EC0,status:0

I fixed that and now I can see this:

Nexthop tracking current: 187.188.X.X
187.188.X.X, fib_nh:7F8726A29638,oce:7F871F197008,status:1

192.168.20.254, fib_nh:7F8726A29688,oce:7F87269B5EC0,status:1

I'm doing now the test with trace route from the internal segments to see the route it takes based on the ACL statements. And looks it routes just fine as it should do.

The thing is I don't get any "debug ip policy" output on the console of the 4331 router neither the packets are matched on the ACL statements while showing the ACL:

show access-list

Extended IP access list GROUP1_ACL
    10 permit ip 10.10.8.0 0.0.7.255 any
    20 permit ip 10.10.16.0 0.0.7.255 any
    30 permit ip 10.10.24.0 0.0.7.255 any
    40 permit ip 10.10.32.0 0.0.7.255 any
    50 permit ip 10.10.40.0 0.0.7.255 any
    60 permit ip 10.10.100.0 0.0.0.255 any
    70 permit ip 10.26.8.48 0.0.0.15 any
    80 deny ip any any
Extended IP access list GROUP2_ACL
    10 permit ip 10.10.60.0 0.0.0.255 any
    20 permit ip 10.10.70.0 0.0.0.255 any
    30 permit ip 10.10.80.0 0.0.0.255 any
    40 permit ip 10.10.90.0 0.0.0.255 any
    50 permit ip 10.10.110.0 0.0.0.255 any
    60 deny ip any any

On the LAB router 1921 I can see the packets matching the ACL and also output for the "debug ip policy"

Thanks in advance.

[Masoud Pourshabanian]

Your configuration looks good to me.

Please post the configuration of interface g0/0/0 and gig 0/0/1

Masoud

[remi-reszka]

Here you go:

wan-rtr1#sh run int gi0/0/0
Building configuration...

Current configuration : 256 bytes
!
interface GigabitEthernet0/0/0
 description WAN Edge ISP-1
 vrf forwarding WAN1
 ip address 187.188.X.X 255.255.255.240
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip nat outside
 negotiation auto
 no cdp enable
 ip virtual-reassembly
end

wan-rtr1#sh run int gi0/0/1
Building configuration...

Current configuration : 253 bytes
!
interface GigabitEthernet0/0/1
 description WAN Edge ISP-2
 vrf forwarding WAN2
 ip address 192.168.20.20 255.255.255.0
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip nat outside
 negotiation auto
 no cdp enable
 ip virtual-reassembly
end

wan-rtr1#

[Masoud Pourshabanian]

You have two outgoing interfaces with IP NAT outside configured. Did you use route-map for your NAT configuration ?

something like this

Ip nat inside source route-map

Try to narrow your problem. Test this configuration without IP SLA . There is only one set  for each route-map and also IP SLA has not been used. I do not have a compatible router to test. Use question mark if something is wrong with the commands.

route-map LB_TRAFFIC_MAP permit 10
 description Route Group 2 over WAN1 and failover to WAN2
 match ip address GROUP2_ACL
 set ip vrf WAN1 next-hop  187.188.X.X 1

route-map LB_TRAFFIC_MAP permit 20
 description Route Group 1 over WAN2 and failover to WAN1
 match ip address GROUP1_ACL
 set ip vrf WAN2 next-hop  192.168.20.254 1

Give your feedback after test.

[remi-reszka]

Many thanks for your comments. I have route maps for the PAT each set with its relevant interface. As I mentioned it looks like the PBR is working fine trying to trace route from each source that I have in the ACLs.

However I cannot verify it on the router as the debug for ip policy is not working :-(.

Hmmm... what can it be? A software bug?

[Masoud Pourshabanian]

It might be a bug. Try to use the configuration in my previous problem and give me your feedback. Different IOSs may have different interpretation of commands.

Masoud

[Masoud Pourshabanian]

I just noticed that you do not have route-map for return traffic. How do do route return traffic to  transit vrf?

You need to add two route maps for return traffic.

[remi-reszka]

The traffic returns to TRANSIT VRF based on the NAT or rather PAT translation table statements and also based on the firewall inspect statements, there is no need to apply the route-map for the returning traffic unless there is specific requirement.

Besides, both WAN VRFs need to allow traffic to each segment in case one of the WAN links goes down. My config enables both Internet access load balancing and redundancy based on PBR.

[Masoud Pourshabanian]

I am sure when you route VRF to global routing table, you need to configure route-map to redirect the return traffic to VRF even though you have configured NAT. Make sure of that.

And what is the reason you have assigned your WAN interfaces into VRF. DO have special need for that?

[remi-reszka]

Any clues? Many thanks.

[DmitryKirillov]

Hi, remi-reszka!

You have solved the problem?

[remi-reszka]

Hello DmitryKirillov,

Yes it has been resolved. In fact it has been working only I cannot debug ip policy as it is well done on ISR 2800/3800 or ISR G2 2900/3900 routers. No returning route-maps needed. I have a setup with FVRF and IVRF.

Cheers!

[Eccom-TAC]

hello，Can you tell me how to resolve this problem。Because I has meet the question