PBR not working on incoming interface, but working on outgoing subinterface

saba · ‎02-05-2018

lab topology

I have a PBR configured on R6 which lets a packet from SW1 lo0 heading to the even ones of R1 lo1 go through R4 and a packet from SW1 lo0 heading to the odd ones through R5.

The thing is that the route-map doesn't work when it's configured on e1/0, and it works when it's configured on e1/0.46. Which doesn't make sense to me since e1/0.46 is not the interface into which the target packet is coming.

Can anyone help me understand what's going on here?

The configuration is as follows:

int e1/0
no shut
du full
ip add 1.1.67.6 255.255.255.240
ip policy route-map PBR
!
int e1/0.46
en dot1q 46
ip add 1.1.46.6 255.255.255.240
ip ospf message-digest-key 55 md5 cisco55
!
route-map PBR permit 100
match ip add 100
set ip next-hop 1.1.46.4
!
route-map PBR permit 200
match ip add 110
set ip next-hop 1.1.56.5
!
access-list 100 permit ip 7.7.7.0 0.0.0.255 172.16.0.0 0.0.6.255
!
access-list 110 permit ip 7.7.7.0 0.0.0.255 172.16.1.0 0.0.6.255
!
!vlan 46 network: 1.1.46.0/28
!vlan 56 network: 1.1.56.0/28
!SW1 Lo0: 7.7.7.7/24

Peter Paluch · ‎02-05-2018

Hello saba,

This is interesting. Please allow me a couple of questions:

What is the IOS version, and the actual platform you're running this topology on?
How do you test whether the PBR works as expected?

Thank you!

Best regards,
Peter

Richard Burts · ‎02-05-2018

The ACL used for PBR is checking for 7.7.7.0. The post suggests that this is a loopback on SW1. Can you verify this?

Interface E1/0 appears to be a trunk. Can you provide details about how that trunk is configured. We are especially interested in how traffic from the loopback is sent over the trunk.

HTH

Rick

HTH

Rick

saba · ‎02-05-2018

I am sorry that I didn't include some info in the original post.

1. What is the IOS version, and the actual platform you're running this topology on?

This lab is done on Dynamips. Not sure the version.. If this is relevant I will find it out.
2. How do you test whether the PBR works as expected?

I did traceroute with "debug ip policy" on.

3. The ACL used for PBR is checking for 7.7.7.0. The post suggests that this is a loopback on SW1. Can you verify this?

Yes. 7.7.7.0 is lo0 on SW1.

4. Interface E1/0 appears to be a trunk. Can you provide details about how that trunk is configured. We are especially interested in how traffic from the loopback is sent over the trunk.

Yes. e1/0 is trunk, and I changed the native vlan to vlan 67 so that I can use the parent interface with vlan 67.

SW1 config is as follows:

!

interface Loopback0

ip address 7.7.7.7 255.255.255.0

ip ospf network point-to-point

!

interface Port-channel1

switchport trunk allowed vlan 1,46,56,78,1002-1005

switchport mode trunk

!

interface FastEthernet1/5

switchport access vlan 56

!

interface FastEthernet1/6

switchport trunk native vlan 67

switchport trunk allowed vlan 1,46,67,1002-1005

switchport mode trunk

!

interface FastEthernet1/13

switchport trunk allowed vlan 1,46,56,78,1002-1005

switchport mode trunk

channel-group 1 mode on

!

interface FastEthernet1/14

switchport trunk allowed vlan 1,46,56,78,1002-1005

switchport mode trunk

channel-group 1 mode on

!

interface Vlan67

ip address 1.1.67.7 255.255.255.240

!

interface Vlan78

ip address 1.1.78.7 255.255.255.240

!

router ospf 1357

router-id 7.7.7.7

log-adjacency-changes

area 0 authentication message-digest

area 67 virtual-link 6.6.6.6 message-digest-key 55 md5 cisco55

passive-interface default

no passive-interface Vlan67

no passive-interface Vlan78

network 7.7.7.7 0.0.0.0 area 67

network 1.1.67.0 0.0.0.15 area 67

network 1.1.78.0 0.0.0.15 area 78

!

The following is the output of traceroute:

SW1#tr 172.16.1.1 source 7.7.7.7

Type escape sequence to abort.

Tracing the route to 172.16.1.1

1 1.1.67.6 284 msec 220 msec 180 msec

2 1.1.56.5 624 msec 284 msec 12 msec

3 1.1.100.3 16 msec 16 msec 12 msec

4 1.1.23.1 16 msec 16 msec 16 msec

5 1.1.12.1 20 msec * 24 msec

SW1#tr 172.16.2.1 source 7.7.7.7

Type escape sequence to abort.

Tracing the route to 172.16.2.1

1 1.1.67.6 152 msec 8 msec 8 msec

2 1.1.56.5 12 msec 12 msec 8 msec

3 1.1.100.3 16 msec 12 msec 12 msec

4 1.1.23.1 16 msec 16 msec 16 msec

5 1.1.12.1 20 msec * 24 msec

And the following is the output of debug ip policy when I did the said traceroute:

R6#

*Mar 1 00:02:37.683: IP: s=1.1.67.6 (local), d=7.7.7.7, len 56, policy rejected

-- normal forwarding

*Mar 1 00:02:37.807: IP: s=1.1.67.6 (local), d=7.7.7.7, len 56, policy rejected

-- normal forwarding

*Mar 1 00:02:37.859: IP: s=1.1.67.6 (local), d=7.7.7.7, len 56, policy rejected

-- normal forwarding

R6#

*Mar 1 00:02:53.007: IP: s=1.1.67.6 (local), d=7.7.7.7, len 56, policy rejected -- normal forwarding

*Mar 1 00:02:53.063: IP: s=1.1.67.6 (local), d=7.7.7.7, len 56, policy rejected -- normal forwarding

*Mar 1 00:02:53.119: IP: s=1.1.67.6 (local), d=7.7.7.7, len 56, policy rejected -- normal forwarding

R6#

*Mar 1 00:03:03.287: IP: s=1.1.67.6 (local), d=7.7.7.7, len 56, policy rejected -- normal forwarding

*Mar 1 00:03:03.335: IP: s=1.1.67.6 (local), d=7.7.7.7, len 56, policy rejected -- normal forwarding

*Mar 1 00:03:03.383: IP: s=1.1.67.6 (local), d=7.7.7.7, len 56, policy rejected -- normal forwarding

R6#

When I changed the setting as follows:

R6(config)#int e1/0

R6(config-if)#no ip policy route-map ROUTING-PLCY

R6(config-if)#int e1/0.46

R6(config-subif)#ip policy route-map ROUTING-PLCY

The output for debug changese as follows:

R6#

*Mar 1 00:08:43.999: IP: s=1.1.67.6 (local), d=7.7.7.7, len 56, policy rejected -- normal forwarding

*Mar 1 00:08:44.051: IP: s=1.1.67.6 (local), d=7.7.7.7, len 56, policy rejected -- normal forwarding

*Mar 1 00:08:44.123: IP: s=1.1.67.6 (local), d=7.7.7.7, len 56, policy rejected -- normal forwarding

*Mar 1 00:08:44.191: IP: s=7.7.7.7 (Ethernet1/0.46), d=172.16.2.1, len 28, FIB policy match

*Mar 1 00:08:44.195: IP: s=7.7.7.7 (Ethernet1/0.46), d=172.16.2.1, g=1.1.46.4, len 28, FIB policy routed

*Mar 1 00:08:44.227: IP: s=1.1.46.4 (Ethernet1/0.46), d=7.7.7.7, len 56, FIB policy rejected(no match) - normal forwarding

*Mar 1 00:08:44.291: IP: s=7.7.7.7 (Ethernet1/0.46), d=172.16.2.1, len 28, FIB policy match

*Mar 1 00:08:44.295: IP: s=7.7.7.7 (Ethernet1/0.46), d=172.16.2.1, g=1.1.46.4, len 28, FIB policy routed

*Mar 1 00:08:44.367: IP: s=1.1.46.4 (Ethernet1/0.46), d=7.7.7.7, len 56, FIB policy rejected(no match) - normal forwarding

*Mar 1 00:08:44.435: IP: s=7.7.7.7

R6#(Ethernet1/0.46), d=172.16.2.1, len 28, FIB policy match

*Mar 1 00:08:44.439: IP: s=7.7.7.7 (Ethernet1/0.46), d=172.16.2.1, g=1.1.46.4, len 28, FIB policy routed

*Mar 1 00:08:44.491: IP: s=1.1.46.4 (Ethernet1/0.46), d=7.7.7.7, len 56, FIB policy rejected(no match) - normal forwarding

*Mar 1 00:08:44.655: IP: s=7.7.7.7 (Ethernet1/0.46), d=172.16.2.1, len 28, FIB policy match

*Mar 1 00:08:44.659: IP: s=7.7.7.7 (Ethernet1/0.46), d=172.16.2.1, g=1.1.46.4, len 28, FIB policy routed

*Mar 1 00:08:44.731: IP: s=1.1.100.3 (Ethernet1/0.46), d=7.7.7.7, len 56, FIB policy rejected(no match) - normal forwarding

*Mar 1 00:08:44.783: IP: s=7.7.7.7 (Ethernet1/0.46), d=172.16.2.1, len 28, FIB policy match

*Mar 1 00:08:44.783: IP: s=7.7.7.7 (Ethernet1/0.46), d=172.16.2.1, g=1.1.46.4, len 28, FIB policy routed

*Mar 1 00:08:45.143: IP: s=1.1.100.3 (Ethernet1/0.46), d=7.7.7.7, len 56, FIB policy rejected(no match) - normal forwarding

*Mar 1 00:08:45.219: IP: s=7.7.7.7 (Ethernet1/0.46), d

R6#=172.16.2.1, len 28, FIB policy match

*Mar 1 00:08:45.223: IP: s=7.7.7.7 (Ethernet1/0.46), d=172.16.2.1, g=1.1.46.4, len 28, FIB policy routed

*Mar 1 00:08:45.271: IP: s=1.1.100.3 (Ethernet1/0.46), d=7.7.7.7, len 56, FIB policy rejected(no match) - normal forwarding

*Mar 1 00:08:45.331: IP: s=7.7.7.7 (Ethernet1/0.46), d=172.16.2.1, len 28, FIB policy match

*Mar 1 00:08:45.331: IP: s=7.7.7.7 (Ethernet1/0.46), d=172.16.2.1, g=1.1.46.4, len 28, FIB policy routed

*Mar 1 00:08:45.435: IP: s=7.7.7.7 (Ethernet1/0.46), d=172.16.2.1, len 28, FIB policy match

*Mar 1 00:08:45.435: IP: s=7.7.7.7 (Ethernet1/0.46), d=172.16.2.1, g=1.1.46.4, len 28, FIB policy routed

*Mar 1 00:08:45.535: IP: s=7.7.7.7 (Ethernet1/0.46), d=172.16.2.1, len 28, FIB policy match

*Mar 1 00:08:45.535: IP: s=7.7.7.7 (Ethernet1/0.46), d=172.16.2.1, g=1.1.46.4, len 28, FIB policy routed

*Mar 1 00:08:45.627: IP: s=7.7.7.7 (Ethernet1/0.46), d=172.16.2.1, len 28, FIB policy match

*Mar 1 00:08:45.631: IP: s=7.

R6#7.7.7 (Ethernet1/0.46), d=172.16.2.1, g=1.1.46.4, len 28, FIB policy routed

*Mar 1 00:08:45.755: IP: s=7.7.7.7 (Ethernet1/0.46), d=172.16.2.1, len 28, FIB policy match

*Mar 1 00:08:45.755: IP: s=7.7.7.7 (Ethernet1/0.46), d=172.16.2.1, g=1.1.46.4, len 28, FIB policy routed

R6#

*Mar 1 00:08:48.847: IP: s=7.7.7.7 (Ethernet1/0.46), d=172.16.2.1, len 28, FIB policy match

*Mar 1 00:08:48.851: IP: s=7.7.7.7 (Ethernet1/0.46), d=172.16.2.1, g=1.1.46.4, len 28, FIB policy routed

Giuseppe Larosa · ‎02-06-2018

Hello Saba,

thanks to additional details you have provided we can say the following:

PBR works only inbound that is it should consider only traffic coming into the interface.

The only "strange" aspect of your debug output are lines sourced with eth1/6 IP address and directed to IP 7.7.7.7. But the source IP address is marked as local.

So what you see is probably a result of Cisco implementation: the LAN interface own IP address appears inbound on the router and this is why we see the misleading output.

I report from your post:

"

*Mar 1 00:08:43.999: IP: s=1.1.67.6 (local), d=7.7.7.7, len 56, policy rejected -- normal forwarding

*Mar 1 00:08:44.051: IP: s=1.1.67.6 (local), d=7.7.7.7, len 56, policy rejected -- normal forwarding

*Mar 1 00:08:44.123: IP: s=1.1.67.6 (local), d=7.7.7.7, len 56, policy rejected -- normal forwarding

*Mar 1 00:08:44.191: IP: s=7.7.7.7 (Ethernet1/0.46), d=172.16.2.1, len 28, FIB policy match

*Mar 1 00:08:44.195: IP: s=7.7.7.7 (Ethernet1/0.46), d=172.16.2.1, g=1.1.46.4, len 28, FIB policy routed

"

The first three lines just appear because the traffic is locally generated and the source is local, it is that of interface towards the multilayer switch where you have used native vlan 67.

The last two lines show that your PBR works as expected incoming traffic that matches the route-map match condition is policy routed.

I agree that the first lines are misleading, but they appear for the reason that you are sending traffic sourced from the interface where PBR is applied to the SW1 loopback address.

Also they do not match the ACLs used in the route-map, so they are normally routed sent to the SW1 using destination based routing.

Again, what you see is a side effect of the Cisco implementation of ICMP (ping) on LAN interfaces.

But PBR is working correctly as far as we can see.

Hope to help

Giuseppe

saba · ‎02-06-2018

First, thanks for the explanation on the first three lines of debug report.

But what I still don't understand is that why the ip policy does not work on the incoming interface of vlan 67? Like you said, PBR works inbound. Doesn't that mean it works only when it's placed on the incoming interface for the packet it is designed to route? For example, if I want to do PBR for a packet that is coming from R4 and heading toward R5, I would need to do PBR on e1/0.46. If the desired packet is coming from R5, PBR on e1/1. And so on. So in my example, shouldn't PBR be on e1/0 since the packet to be routed is coming from SW1 through vlan 67 interface.

SW1 as a L3 switch has two routing interfaces, vlan 67 and vlan 78. While e1/0 and e1/0.46 on R6 are connected to SW1 through fa1/6 as a trunk, shouldn't L3 switch only be able to use e1/0 to route a packet from SW1 to R6? How can SW1 use e1/0.46 on R6 which is encapsulated to dedicated to vlan 46 when SW1 doesn't have an interface for vlan 46?

Giuseppe Larosa · ‎02-06-2018

Hello Saba,

OK now I understand that the PBR looks like working on the eth1/0.46 subinterface that connects router R6 to R4 at OSI Layer3.

pleae note that Your lab is not so simple you have an OSPF virtual link between SW1 and I suppose R6 in area 67.

I can only guess that area 67 has no direct connection to backbone area 0 or you would not use a virtual-link.

To see if this has an effect on your PBR tests I would suggest to change the network topology so that area 67 is directly connected to OSPF area 0, so that you can remove the virtual-link configuration on SW1 and likely on R6.

Hope to help

Giuseppe

saba · ‎02-07-2018

Hi, Giuseppe.

Yes, as you have guessed, R4-R5-R6 is the backbone area, R6-SW1 the transit area(67), and SW1-SW2 some other area. I now realize that some part of this lab might not be as simple as a real life one because of all the repeated use of the same switches everywhere and whatnot.

I did try what you have suggested and removed the virtual link. After that, the policy worked for the first traceroute attempt and then quit working. I'm going to try this with the Packet-tracer to see if this problem is universal or what. But this is my mid-report.

The following is my first attempt of traceroute:

SW1#tr 172.16.3.1 source lo0

Type escape sequence to abort.
Tracing the route to 172.16.3.1

1 1.1.67.6 152 msec 416 msec *
2 1.1.56.5 1524 msec 104 msec 116 msec
3 1.1.100.3 132 msec 124 msec 132 msec
4 1.1.23.1 136 msec 140 msec 128 msec
5 1.1.12.1 144 msec * 168 msec

The following is the output for debug report:

R6#debug ip policy
Policy routing debugging is on
R6#
*Mar 1 00:02:54.883: IP: s=7.7.7.7 (Ethernet1/0), d=172.16.3.1, len 28, FIB policy match
*Mar 1 00:02:54.887: IP: s=7.7.7.7 (Ethernet1/0), d=172.16.3.1, g=1.1.56.5, len 28, FIB policy routed
*Mar 1 00:02:54.999: IP: s=7.7.7.7 (Ethernet1/0), d=172.16.3.1, len 28, FIB policy match
*Mar 1 00:02:54.999: IP: s=7.7.7.7 (Ethernet1/0), d=172.16.3.1, g=1.1.56.5, len 28, FIB policy routed
*Mar 1 00:02:55.131: IP: s=7.7.7.7 (Ethernet1/0), d=172.16.3.1, len 28, FIB policy match
*Mar 1 00:02:55.131: IP: s=7.7.7.7 (Ethernet1/0), d=172.16.3.1, g=1.1.56.5, len 28, FIB policy routed
R6#sh route-map ROUTING-PLCY
route-map ROUTING-PLCY, permit, sequence 100
Match clauses:
    ip address (access-lists): SW1-R1-EVEN
Set clauses:
    ip next-hop 1.1.46.4
Policy routing matches: 0 packets, 0 bytes
route-map ROUTING-PLCY, permit, sequence 200
Match clauses:
    ip address (access-lists): SW1-R1-ODD
Set clauses:
    ip next-hop 1.1.56.5
Policy routing matches: 3 packets, 180 bytes

saba · ‎02-17-2018

From what I have seen and tried so far, it doesn't seem to have any viable solution. Some people say that I shouldn't have used the main interface when its sub-interface is configured and that it's why unexpected things happen.

chrihussey · ‎02-05-2018

It appears that SW1 must be a layer 3 switch as the 7.7.7.7 needs to be routed by the switch. I only see OSPF configured on the E1/0.46 interface. So this may be why you are seeing these results. Take a look at the routing table on SW1 and this may clear things up for you.

Hope this helps

saba · ‎02-05-2018

Yes, SW1 works as both L2 & L3 switch. And it is configured that way. R4, R5, R6 are ospf area 0 ABRs. R6 and SW1 are ABRs for area 67. Perhaps I should draw this in too.