Policy Based Routing Issue on ISR 4k Router

[mustafa.chapal]

Hi,

I have a Cisco ISR 4431 router running on IOS XE version 16.09.07 with securityk9 and ipbase license. The hosts under interface

Port-channel1.20 

on which PBR is configured are able to reach the

 next hop

but packets destined further to any other external destination get dropped after the

 next hop

I have tried by changing the IOS XE version to 16.09.06 and 16.03.09 but the issue persists. I am using the exact same PBR configuraton on ASR 1001-X and Cisco 897VA routers and it is working fine on them. Attached is the configuration.

Any help will be greatly appreciated.

Thank you

version 16.9
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime msec localtime show-timezone
service timestamps log datetime msec localtime show-timezone
service password-encryption
service sequence-numbers
service unsupported-transceiver
platform qfp utilization monitor load 80
no platform punt-keepalive disable-kernel-core
platform hardware throughput level 1000000
!
hostname cr1
!
boot-start-marker
boot system bootflash:isr4400-universalk9.16.09.07.SPA.bin
boot-end-marker
!
!
vrf definition Mgmt-intf
 !
 address-family ipv4
 exit-address-family
 !
 address-family ipv6
 exit-address-family
!
security authentication failure rate 3 log
security passwords min-length 6
logging buffered 51200
logging console critical
!
aaa new-model
!
!
aaa authentication login default local
aaa authorization exec default local
!
!
!
!
!
!
aaa session-id common
clock timezone Chicago -6 0
no ip source-route
no ip gratuitous-arps
!
!
!
!
!
!
!
no ip bootp server
ip name-server 8.8.8.8 8.8.4.4 1.1.1.1 1.0.0.1 208.67.222.222 208.67.220.220
ip dhcp bootp ignore
ip dhcp excluded-address 10.5.1.1
ip dhcp excluded-address 10.5.1.1 10.5.1.150
ip dhcp excluded-address 10.5.1.200 10.5.1.254
ip dhcp excluded-address 10.5.2.1
!
ip dhcp pool ccp-pool1
 import all
 network 10.5.1.0 255.255.255.0
 default-router 10.5.1.1
dns-server 8.8.8.8 8.8.4.4 1.1.1.1 1.0.0.1 208.67.222.222 208.67.220.220
!
ip dhcp pool ccp-pool2
 import all
 network 10.5.2.0 255.255.255.0
 default-router 10.5.2.1
 dns-server 8.8.8.8 8.8.4.4 1.1.1.1 1.0.0.1 208.67.222.222 208.67.220.220
!
!
!
!
login on-success log
!
!
!
!
!
!
!
subscriber templating
!
!
!
!
!
multilink bundle-name authenticated
!
!
!
!
!        
!
license udi pid ISR4431/K9
license boot level appxk9
license boot level securityk9
no license smart enable
diagnostic bootup level minimal
!
spanning-tree extend system-id
!
!
!
!
redundancy
 mode none
!
!
!
!
!
!
no cdp run
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
interface Null0
 no ip unreachables
!
interface Port-channel1
 no ip address
 no negotiation auto
!
interface Port-channel1.10
 encapsulation dot1Q 10
 ip address 10.5.1.1 255.255.255.0
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip nat inside
 ip verify unicast reverse-path
!
interface Port-channel1.20
 encapsulation dot1Q 20
 ip address 10.5.2.1 255.255.255.0
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip nat inside
 ip verify unicast reverse-path
 ip policy route-map 20
!
interface Port-channel2
 no ip address
 no negotiation auto
!
interface Port-channel2.98
 encapsulation dot1Q 98
 ip address 98.1.1.2 255.255.255.252
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip nat outside
 ip verify unicast reverse-path
 ip access-group inside in
 ip tcp adjust-mss 1460
!
interface Port-channel2.99
 encapsulation dot1Q 99
 ip address 99.1.1.2 255.255.255.252
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip nat outside
 ip verify unicast reverse-path
 ip access-group inside in
 ip tcp adjust-mss 1400
!
interface GigabitEthernet0/0/0
 no ip address
 load-interval 30
 negotiation auto
 no cdp enable
 no lldp transmit
 no lldp receive
 channel-group 1
!        
interface GigabitEthernet0/0/1
 no ip address
 load-interval 30
 negotiation auto
 no cdp enable
 no lldp transmit
 no lldp receive
 channel-group 1
!
interface GigabitEthernet0/0/2
 no ip address
 load-interval 30
 negotiation auto
 no cdp enable
 no lldp transmit
 no lldp receive
 channel-group 2
!
interface GigabitEthernet0/0/3
 no ip address
 load-interval 30
 negotiation auto
 no cdp enable
 no lldp transmit
 no lldp receive
 channel-group 2
!
interface GigabitEthernet0
 vrf forwarding Mgmt-intf
 no ip address
 negotiation auto
!
interface Virtual-Template1
 ip unnumbered Port-channel1.10
 ip nat inside
!
ip forward-protocol nd
ip tcp synwait-time 10
no ip http server
ip http authentication local
no ip http secure-server
ip http timeout-policy idle 60 life 86400 requests 10000
ip tftp source-interface Port-channel1.10
ip nat inside source route-map 1 interface Port-channel2.98 overload
ip nat inside source route-map 2 interface Port-channel2.99 overload
ip route 0.0.0.0 0.0.0.0 Port-channel2.98 98.1.1.1
ip route 0.0.0.0 0.0.0.0 Port-channel2.99 99.1.1.1 10
!
ip ssh time-out 60
ip ssh authentication-retries 2
ip ssh port 2222 rotary 1
ip ssh version 2
!
!
ip access-list extended abc
 permit ip 10.0.0.0 0.255.255.255 any
ip access-list extended external
 permit ip any any
ip access-list extended inside
 deny   ip 0.0.0.0 0.255.255.255 any
 deny   ip 10.0.0.0 0.255.255.255 any
 deny   ip 100.64.0.0 0.63.255.255 any
 deny   ip 127.0.0.0 0.255.255.255 any
 deny   ip 169.254.0.0 0.0.255.255 any
 deny   ip 172.16.0.0 0.15.255.255 any
 deny   ip 192.0.0.0 0.0.0.255 any
 deny   ip 192.88.99.0 0.0.0.255 any
 deny   ip 192.0.2.0 0.0.0.255 any
 deny   ip 192.168.0.0 0.0.255.255 any
 deny   ip 198.18.0.0 0.1.255.255 any
 deny   ip 198.51.100.0 0.0.0.255 any
 deny   ip 203.0.113.0 0.0.0.255 any
 deny   ip 224.0.0.0 15.255.255.255 any
 deny   ip 233.252.0.0 0.0.0.255 any
 deny   ip 240.0.0.0 15.255.255.255 any
 deny   ip host 255.255.255.255 any
 deny   ip host 0.0.0.0 any
 deny   udp any any eq snmp
 deny   udp any any eq snmptrap
 deny   tcp any any fragments
 deny   udp any any fragments
 deny   ip any any fragments
 deny   icmp any any fragments
 permit icmp any any ttl-exceeded
 permit icmp any any port-unreachable
 permit icmp any any echo-reply
 permit icmp any any echo
 permit icmp any any packet-too-big
 deny   icmp any any
 permit ip any any
ip access-list extended internal
 permit ip 10.0.0.0 0.255.255.255 10.0.0.0 0.255.255.255
 permit udp any any eq bootps
ip access-list extended vty
 deny   tcp any any eq 22
 permit ip any any

access-list 1 remark CCP_ACL Category=2
access-list 1 permit 10.0.0.0 0.255.255.255
!
!
!
route-map 1 permit 10
 match ip address 1
 match interface Port-channel2.98
!
route-map 2 permit 10
 match ip address 1
 match interface Port-channel2.99
!
route-map 20 permit 10
 match ip address internal
 set ip next-hop self
!
route-map 20 permit 20
 match ip address external
 set ip next-hop 99.1.1.1
!        
!
!
!
control-plane
!
!
line con 0
 privilege level 15
 transport input none
 transport output ssh
 stopbits 1
line aux 0
 exec-timeout 0 1
 no exec
 transport output none
 stopbits 1
line vty 0 4
 access-class vty in
 privilege level 15
 rotary 1
 transport input ssh
 transport output none
!
end

[Elliot Dierksen]

I am not a big fan of PBR, but I suspect in the ISR4K you may have to apply the policy on the parent interface. Make sure your

permit

entry only lists the IP's that live on Po1.20.

[mustafa.chapal]

I attached the route map to the parent interface along with the specific ip mentioned in the list and in this case it is neither dropping the packet nor routing through the

next hop

mentioned meaning it is not following the policy. Apart from this I even tried three other configs but still not working as expected.

Config 1

interface Port-channel1
 no ip address
 no negotiation auto
 ip policy route-map 20
!
interface Port-channel1.20
 encapsulation dot1Q 20
 ip address 10.5.2.1 255.255.255.0
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip nat inside
 ip verify unicast reverse-path

ip access-list extended abc
 permit ip host 10.5.2.70 any

ip access-list extended internal
 permit ip 10.0.0.0 0.255.255.255 10.0.0.0 0.255.255.255
 permit udp any any eq bootps
access-list 1 remark CCP_ACL Category=2
access-list 1 permit 10.0.0.0 0.255.255.255
!
!
!
!
route-map 20 permit 10
 match ip address internal
 set ip next-hop self
!
route-map 20 permit 20
 match ip address abc
 set ip next-hop 99.1.1.1


Config 2

interface Port-channel1
 no ip address
 no negotiation auto
!
interface Port-channel1.20
 encapsulation dot1Q 20
 ip address 10.5.2.1 255.255.255.0
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip nat inside
 ip verify unicast reverse-path
 ip policy route-map 20

ip access-list extended abc
 permit ip host 10.5.2.70 any

ip access-list extended internal
 permit ip 10.0.0.0 0.255.255.255 10.0.0.0 0.255.255.255
 permit udp any any eq bootps
access-list 1 remark CCP_ACL Category=2
access-list 1 permit 10.0.0.0 0.255.255.255
!
!
!
!
route-map 20 permit 10
 match ip address internal
 set ip next-hop self
!
route-map 20 permit 20
 match ip address abc
 set ip next-hop 99.1.1.1


Config 3

interface Port-channel1
 no ip address
 no negotiation auto
 ip policy route-map 20
!
interface Port-channel1.20
 encapsulation dot1Q 20
 ip address 10.5.2.1 255.255.255.0
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip nat inside
 ip verify unicast reverse-path

ip access-list extended internal
 permit ip 10.0.0.0 0.255.255.255 10.0.0.0 0.255.255.255
 permit udp any any eq bootps
access-list 1 remark CCP_ACL Category=2
access-list 1 permit 10.0.0.0 0.255.255.255
access-list 70 permit 10.5.2.70
!
!
!
!
route-map 20 permit 10
 match ip address internal
 set ip next-hop self
!
route-map 20 permit 20
 match ip address 70
 set ip next-hop 99.1.1.1


Config 4

interface Port-channel1
 no ip address
 no negotiation auto
!
interface Port-channel1.20
 encapsulation dot1Q 20
 ip address 10.5.2.1 255.255.255.0
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip nat inside
 ip verify unicast reverse-path
 ip policy route-map 20

ip access-list extended internal
 permit ip 10.0.0.0 0.255.255.255 10.0.0.0 0.255.255.255
 permit udp any any eq bootps
access-list 1 remark CCP_ACL Category=2
access-list 1 permit 10.0.0.0 0.255.255.255
access-list 70 permit 10.5.2.70
!
!
!
!
route-map 20 permit 10
 match ip address internal
 set ip next-hop self
!
route-map 20 permit 20
 match ip address 70
 set ip next-hop 99.1.1.1

[MHM Cisco World]

set ip next-hop self <- this only use for BGP why you use it here ???

[MHM Cisco World]

set default ip next-hop <- if you want the router to check the routing before check PBR you must use default keyword.

[Richard Burts]

I agree with MHM in being surprised about set

ip next hop

self. But I do not believe that this is part of the main problem described in the original post. There are things in this config that I do not understand. But I have looked especially at the parts related to PBR and believe that other than perhaps the

 next-hop self

that it is correctly configured. And the statement in the original config The hosts under interface

Port-channel1.20

on which PBR is configured are able to reach the

next hop 

seems to confirm that PBR is working. The problem is that packets destined further to any other external destination get dropped after the

next hop

 I believe that the problem is something on 99.1.1.1. What is this device? Do you have any visibility into it?

[MHM Cisco World]

ip access-list extended internal
 permit ip 10.0.0.0 0.255.255.255 10.0.0.0 0.255.255.255

Also can you elaborate more about this acl?

How source and destiantion same? If they same then packet will bridge not routing and it never hit the port channel.

[Richard Burts]

Georg I am not familiar with issues about 

ip verify unicast reverse-path 

impacting PBR. I do know that some platforms in PBR do not support the verify-availability, but that is not an issue in this configuration. As a test I agree with you about remove this on the subinterface and see if the behavior changes.

MHM this "permit ip 10.0.0.0 0.255.255.255 10.0.0.0 0.255.255.255" is just a way of saying that any subnet of network 10.0.0.0 can talk to any subnetwork of 10.0.0.0 (inside source to inside destination) and not be Policy Routed. That traffic would still be routed and not bridged. A more usual solution for this would be to modify the acl external and place this as the first line of the acl

deny ip 10.0.0.0 0.255.255.255 10.0.0.0 0.255.255.255

and then have the permit ip any any

[Georg Pauwen · Accepted solution]

Hello,

I might be off here, but I remotely remember

ip verify unicast reverse-path

causing problems with PBR, at least on that platform. Try and remove that command from the interface...

[mustafa.chapal]

Thank you everyone for the help.

@Georg Pauwen @Richard Burts Removing 

ip verify unicast reverse-path

from the outside facing 

interface Port-channel2.99 

resolved the issue. Also this might be of help to someone using a similar platform that PBR using a

next hop

as the other end in which a tunnel interface is being used then

set ip next-hop

recursive works.

[Richard Burts]

Thanks for the update. Interesting that removing verify unicast reverse-path did resolve the issue.

[MHM Cisco World]

I don't think only this change he make many change, 
please @mustafa.chapal  can you share the final config.

[mustafa.chapal]

@MHM Cisco World final config

version 16.9
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime msec localtime show-timezone
service timestamps log datetime msec localtime show-timezone
service password-encryption
service sequence-numbers
service unsupported-transceiver
platform qfp utilization monitor load 80
no platform punt-keepalive disable-kernel-core
platform hardware throughput level 1000000
!
hostname cr1
!
boot-start-marker
boot system bootflash:isr4400-universalk9.16.09.07.SPA.bin
boot-end-marker
!
!
vrf definition Mgmt-intf
 !
 address-family ipv4
 exit-address-family
 !
 address-family ipv6
 exit-address-family
!
security authentication failure rate 3 log
security passwords min-length 6
logging buffered 51200
logging console critical
!
aaa new-model
!
!
aaa authentication login default local
aaa authorization exec default local
!
!
!
!
!
!
aaa session-id common
clock timezone Chicago -6 0
no ip source-route
no ip gratuitous-arps
!
!
!
!
!
!
!
no ip bootp server
ip name-server 8.8.8.8 8.8.4.4 1.1.1.1 1.0.0.1 208.67.222.222 208.67.220.220
ip dhcp bootp ignore
ip dhcp excluded-address 10.5.1.1
ip dhcp excluded-address 10.5.1.1 10.5.1.150
ip dhcp excluded-address 10.5.1.200 10.5.1.254
ip dhcp excluded-address 10.5.2.1
!
ip dhcp pool ccp-pool1
 import all
 network 10.5.1.0 255.255.255.0
 default-router 10.5.1.1
dns-server 8.8.8.8 8.8.4.4 1.1.1.1 1.0.0.1 208.67.222.222 208.67.220.220
!
ip dhcp pool ccp-pool2
 import all
 network 10.5.2.0 255.255.255.0
 default-router 10.5.2.1
 dns-server 8.8.8.8 8.8.4.4 1.1.1.1 1.0.0.1 208.67.222.222 208.67.220.220
!
!
!
!
login on-success log
!
!
!
!
!
!
!
subscriber templating
!
!
!
!
!
multilink bundle-name authenticated
!
!
!
!
!        
!
license udi pid ISR4431/K9
license boot level appxk9
license boot level securityk9
no license smart enable
diagnostic bootup level minimal
!
spanning-tree extend system-id
!
!
!
!
redundancy
 mode none
!
!
!
!
!
!
no cdp run
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
interface Null0
 no ip unreachables
!
interface Port-channel1
 no ip address
 no negotiation auto
!
interface Port-channel1.10
 encapsulation dot1Q 10
 ip address 10.5.1.1 255.255.255.0
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip nat inside
 ip verify unicast reverse-path
!
interface Port-channel1.20
 encapsulation dot1Q 20
 ip address 10.5.2.1 255.255.255.0
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip nat inside
 ip verify unicast reverse-path
 ip policy route-map 20
!
interface Port-channel2
 no ip address
 no negotiation auto
!
interface Port-channel2.98
 encapsulation dot1Q 98
 ip address 98.1.1.2 255.255.255.252
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip nat outside
 ip access-group inside in
 ip tcp adjust-mss 1460
!
interface Port-channel2.99
 encapsulation dot1Q 99
 ip address 99.1.1.2 255.255.255.252
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip nat outside
 ip access-group inside in
 ip tcp adjust-mss 1400
!
interface GigabitEthernet0/0/0
 no ip address
 load-interval 30
 negotiation auto
 no cdp enable
 no lldp transmit
 no lldp receive
 channel-group 1
!        
interface GigabitEthernet0/0/1
 no ip address
 load-interval 30
 negotiation auto
 no cdp enable
 no lldp transmit
 no lldp receive
 channel-group 1
!
interface GigabitEthernet0/0/2
 no ip address
 load-interval 30
 negotiation auto
 no cdp enable
 no lldp transmit
 no lldp receive
 channel-group 2
!
interface GigabitEthernet0/0/3
 no ip address
 load-interval 30
 negotiation auto
 no cdp enable
 no lldp transmit
 no lldp receive
 channel-group 2
!
interface GigabitEthernet0
 vrf forwarding Mgmt-intf
 no ip address
 negotiation auto
!
interface Virtual-Template1
 ip unnumbered Port-channel1.10
 ip nat inside
!
ip forward-protocol nd
ip tcp synwait-time 10
no ip http server
ip http authentication local
no ip http secure-server
ip http timeout-policy idle 60 life 86400 requests 10000
ip tftp source-interface Port-channel1.10
ip nat inside source route-map 1 interface Port-channel2.98 overload
ip nat inside source route-map 2 interface Port-channel2.99 overload
ip route 0.0.0.0 0.0.0.0 Port-channel2.98 98.1.1.1
ip route 0.0.0.0 0.0.0.0 Port-channel2.99 99.1.1.1 10
!
ip ssh time-out 60
ip ssh authentication-retries 2
ip ssh port 2222 rotary 1
ip ssh version 2
!
!
ip access-list extended external
 deny ip 10.0.0.0 0.255.255.255 10.0.0.0 0.255.255.255
 permit ip any any
ip access-list extended inside
 deny   ip 0.0.0.0 0.255.255.255 any
 deny   ip 10.0.0.0 0.255.255.255 any
 deny   ip 100.64.0.0 0.63.255.255 any
 deny   ip 127.0.0.0 0.255.255.255 any
 deny   ip 169.254.0.0 0.0.255.255 any
 deny   ip 172.16.0.0 0.15.255.255 any
 deny   ip 192.0.0.0 0.0.0.255 any
 deny   ip 192.88.99.0 0.0.0.255 any
 deny   ip 192.0.2.0 0.0.0.255 any
 deny   ip 192.168.0.0 0.0.255.255 any
 deny   ip 198.18.0.0 0.1.255.255 any
 deny   ip 198.51.100.0 0.0.0.255 any
 deny   ip 203.0.113.0 0.0.0.255 any
 deny   ip 224.0.0.0 15.255.255.255 any
 deny   ip 233.252.0.0 0.0.0.255 any
 deny   ip 240.0.0.0 15.255.255.255 any
 deny   ip host 255.255.255.255 any
 deny   ip host 0.0.0.0 any
 deny   udp any any eq snmp
 deny   udp any any eq snmptrap
 deny   tcp any any fragments
 deny   udp any any fragments
 deny   ip any any fragments
 deny   icmp any any fragments
 permit icmp any any ttl-exceeded
 permit icmp any any port-unreachable
 permit icmp any any echo-reply
 permit icmp any any echo
 permit icmp any any packet-too-big
 deny   icmp any any
 permit ip any any
ip access-list extended vty
 deny   tcp any any eq 22
 permit ip any any

access-list 1 remark CCP_ACL Category=2
access-list 1 permit 10.0.0.0 0.255.255.255
!
!
!
route-map 1 permit 10
 match ip address 1
 match interface Port-channel2.98
!
route-map 2 permit 10
 match ip address 1
 match interface Port-channel2.99
!
route-map 20 permit 10
 match ip address external
 set ip next-hop 99.1.1.1
 set interface Null0
!        
!
!
!
control-plane
!
!
line con 0
 privilege level 15
 transport input none
 transport output ssh
 stopbits 1
line aux 0
 exec-timeout 0 1
 no exec
 transport output none
 stopbits 1
line vty 0 4
 access-class vty in
 privilege level 15
 rotary 1
 transport input ssh
 transport output none
!
end

[paul driver]

Hello T
Try the following:

interface Port-channel1.20
no ip verify unicast reverse-path

no route-map 20 permit 10
route-map 20 permit 20
no match ip address external