static route overrides PBR on cisco6509

[nokialabcd]

Hi All

Just met a strange problem with PBR vs static route.

#Topology

#Purpose

A static route exists on 6509 for servers in 172.16.0.0/16:

ip route 172.76.0.0 255.255.0.0 10.69.151.69

All the traffic towards servers will be sent to router R1 by 6509.

Now I want to configure a PBR to force the traffic from host A towards servers to go through router R2. And there're some other devices behind R2 for following process.

#Configuration

The configuration is as below:

...

access-list 134 permit ip host 10.56.136.4 172.76.0.0 0.0.255.255

...

route-map test permit 10

match ip address 134

set ip next-hop 10.56.137.7

...

interface Vlan2

ip address 10.56.136.1 255.255.255.128

ip policy route-map test

...

...

ip route 172.76.0.0 255.255.0.0 10.69.151.69

...

#Problem:

According to common understanding, PBR has higher priority than static route in routing table.

It's also described in many doc:

PBR is used to bypass the routing table. It allows network administrators to configure different routing rules outside the boundaries of the original IP routing table.  One of the main uses of PBR is to route packets that are based on the source IP address instead of the destination IP address

PBR static routes have priority over static routes in the routing table. If the packet does not match a PBR static route, the packet is forwarded according to the priority of the static routes in the routing table.

But the strange thing is, I found the PBR setting as above didn't take effect. Instead, the request from hostA to 172.76.0.0/16 is still handled by static route, e.g., sent to R1.

Then I deleted the static route "ip route 172.76.0.0 255.255.0.0 10.69.151.69", the PBR works immediately! If I add that static route back, PBR fails again.

Why 6509 act as this? Why static route take higher priority than PBR here?

[Rolf Fischer]

Hi Leo,

I'm using a very similar setup with a Sup720-3B 12.2(33)SXI2a and it works perfectly.

Could you tell us what SE/IOS you're using?

What about the counters of show access-list 134 and show route-map test, are they increasing?

Regards

Rolf

[nokialabcd]

Hi Rolf,

The IOS version: 12.2(18)SXF14

And counters don't increase if the static route entry existing. It increases only after I delete that route entry. 

6509#show access-lists 134

Extended IP access list 134

    10 permit ip host 10.56.136.4 172.76.0.0 0.0.255.255 (141 matches)

6509#show route-map test

route-map test, permit, sequence 10

  Match clauses:

    ip address (access-lists): 134

  Set clauses:

    ip next-hop 10.56.137.7

  Policy routing matches: 270 packets, 23142 bytes

It's quite a simple PBR setting. No idea why it doesn't work. A bug in IOS?

[nokialabcd]

hi

Could anyone give some suggestion? thanks a lot.

[Rolf Fischer]

Hi Leo,

we would expect that behavior when using the "set ip default next-hop" clause - which you don't use.

Have you tried to delete the complete PBR config and configure it again from scratch afterwards?

Unfortunately I can't say if it's an IOS bug or not, I found some documented PBR bugs for that version but nothing related to your particular problem. Would an IOS update be out of question?

Regards

Rolf

[nokialabcd]

Hi Rolf

Yes, I tried delete PBR rules and created again, but still failed.

IOS update could be a good try, but now this 6509 is taking heave traffic and i cannot do it.

[Bharat Negi]

Hello Leo

Can you please share the configuration and routing table of 6509?

Also have you tried "set interface" under route map?

regards

Bharat

[nokialabcd]

Hi Bharat,

the related configuration:

#Configuration

The configuration is as below:

...

access-list 134 permit ip host 10.56.136.4 172.76.0.0 0.0.255.255

...

route-map test permit 10

match ip address 134

set ip next-hop 10.56.137.7

...

interface Vlan2

ip address 10.56.136.1 255.255.255.128

ip policy route-map test

...

...

ip route 172.76.0.0 255.255.0.0 10.69.151.69

...

the routing table is too long, i paste only this line which matters:

S       172.76.0.0 [254/0] via 10.69.151.69

And I used "set ip next-hop" only, are you saying to use "set interface" instead?

[Bharat Negi]

Hello Leo

Do you have any other switch between host and 6509?

If you can test by removing interface VLAN from 6509 and configure a the physical port with VLAN2 IP address and apply poicy-map on that interface and then check if route-map is working.

Regards

Bharat

[Bharat Negi]

Hello Leo

Yes, you can try using set interface under route-map. 

As per your previous post, when you remove static route then only route-map comes into action and starts noticing logs on access-list and route-map.  It seems that packets on vlan2 are processed without looking the route-map.  Once static route is removed only then the route-map is looked upon.  Packets on VLAN2 seems to be routed/switched before it hits the policy on the interface.

Defining an interface as a next-hop on a static route brings down the AD value to 0 from 1.  You can also try increasing the AD value of the static route.

[nokialabcd]

Hi,

I did some more test and here're intesting update:

1. use "set ip next-hop" in route-map.

a. when static route exists:

acl log doesn't increase.

route-map doesn't log increases.

packet doesn't reach peer.

b. when static route doesn't exist:

acl log doesn't increase.

route-map doesn't log increases.

packet reaches peer.

2. use "set interface" in route-map.

no matter static route exists or not:

acl log increases.

route-map log increases.

packet doesn't reach peer.

1.a is the same as I found before. No log changes, no packet reaches peer.

The interesting thing is in 1.b where acl and route-map logs don't increase but packet is correctly routed to peer according to route-map.

The result of 2 is also different. When "set interface" is used instaed of "set ip next-hop", I can see both acl and route-map logs increasing. Route-map works as expected. But the packet doesn't reach peer, it could be incorrect interface used in my test. But at least, the route-map works and logs increases.

Back to 1.b, packet reach peer but no log increse for both acl and route-map. I don't understand this behaviour. One thing may be related to this, I have CEF switching enabled on this 6509.

According to Cisco:

Beginning in Cisco IOS Release 12.0, PBR is supported in the Cisco Express Forwarding (CEF) switching path. CEF-switched PBR has better performance than fast-switched PBR and, therefore, is the optimal way to perform PBR on a router.

No special configuration is required to enable CEF-switched PBR. It is on by default as soon as you enable CEF and PBR on the router.

Did I missed something with CEF config, which cause 6509 doesn't check acl but use some cache to route packet (jus t my guess)?

So I did one more test:

Now i put 2 static routes:

ip route 172.76.0.0 255.255.0.0 10.69.151.69 (old one)

ip route 173.200.0.0 255.255.0.0 10.56.232.10

The second route is a fake one. 10.56.232.10 is a working router, but it doesn't have route for 173.200.0.0/16 in its routing table.

In 6509, I can see both items in routing table:

6509_core#sh ip route | i 172.76

S       172.76.0.0 [1/0] via 10.69.151.69

6509_core#sh ip route | i 173.200

S    173.200.0.0/16 [1/0] via 10.56.232.10

Then I tested with "set ip next-hop 10.56.137.7":

Route-map doesnt work for 172.76/16, and it's still sent to 10.69.151.69 according to static route.

Route-map works for 173.200/16, and it's sent to  10.56.137.7. But, acl/route-map logs didn't change.

Now I would say I'm totally confused. 2 static routes, one has higher priority than route-map, the other has lower.

[Bharat Negi]

Hello Leo

Have you tried removing interface vlan and shifting that network to the physical interface connected to host and applying route map to it?  The packets which are entering 6509 are hitting physical interface first then going to interface VLAN and in between the routing decesion is being taken.  Putting policy on the interface may force packet to follow route-map.

Also do you have more switches between host and 6509? If yes, what type and can you share your diagram from host to 6509 including other switches.

Regards

Bharat

[nokialabcd]

Hi Bharat,

Now I cannot remove that vlan inteface (int vlan2) because some other traffic is still pending on it. But I'll find a spare port and do the test as you suggested. Will update the result once finished.

There's no any other switch between the host and 6509. The host 10.56.136.4 is directly attached on port G1/2 on 6509.

Thanks.

[nokialabcd]

Hi Bharat,

Just did the test, connected my laptop(192.168.1.2) to g1/3 and set as:

interface GigabitEthernet1/3

ip address 192.168.1.1 255.255.255.0

ip policy route-map test

6509_core#show access-lists 134

Extended IP access list 134

    10 permit ip host 10.56.136.4 173.200.0.0 0.0.255.255 (48 matches)

    20 permit ip host 10.56.136.4 172.76.0.0 0.0.255.255 (209 matches)

    30 permit ip host 192.168.1.2 173.200.0.0 0.0.255.255

    40 permit ip host 192.168.1.2 172.76.0.0 0.0.255.255

6509_core#show route-map test

route-map test, permit, sequence 10

  Match clauses:

    ip address (access-lists): 134

  Set clauses:

    ip next-hop 10.56.137.7

  Policy routing matches: 66186 packets, 63734148 bytes

The result is same.

Reqeust to 173.200/16 can be rotued to 137.7 by PBR, without acl/route-map counter increased.

Reqeust to 172.76/16 is still handled by static route.

No matter the L3 interface is a physical int or vlan int, the behaviour is same.

Maybe upgrade IOS is the last try...

[Bharat Negi · Accepted solution]

Hi Leo

Any progress on this?

regards

Bharat