4500X PBR problem on SVIs

[jnesbitt1]

I have been trying to get PBR to work on a 4500x running entservices for the last few days in my lab before deploying into production. However, I am having problems getting PBR to work on the SVIs; local PBR is working as desired on the core. I did however get PBR to function properly on a 3750x running ipservcies. Below is the general setup:

I just noticed the subnet on the firewall for V405 and V405 on the core are marked as different subnets. I verified this in the config and both are /29. I just mismarked the diagram. (9-19-2018)

Vlan 400: MGT Plane for all switches

Vlan 405: L3 Interconnect between core and firewall

Vlan 150: User subnet 10.10.150.0 /24

Vlan 160: User subnet 10.10.160.0 /24

Desired traffic flow:

All traffic coming from any user IP address directed to a switch IP on vlan 400 needs to go through the firewall first. In addition, all traffic originated on vlan 400 such as syslog, ping and ntp need to go through the firewall; this part functions properly as the local PBR route-map on the core directs traffic from the core's vlan 400 to the firewall. Additionally the access layer switches direct their traffic to their default gateway of 192.168.30.1 (Firewall sub-interface).

Here is the PBR configuration that worked as intended on a 3750x but not a 4500x:

-----------------------------------------------

access-list 130 permit ip any 192.168.30.0 0.0.0.255

access-list 135 permit ip 192.168.30.0 0.0.0.255 any

route-map pcnm permit 300
match ip address 130
set ip next-hop 192.168.35.1
exit

route-map local permit 300

match ip address 135

set ip next-hop 192.168.30.1

exit

interface vlan 150

ip policy route-map pcnm

exit

interface vlan 160

ip policy route-map pcnm

exit

ip local policy route-map local

-----------------------------------------------

When issuing a show route-map I see matches on both policies. To help troubleshoot I issued a 'debug ip policy' on the core and received the following results:

Test1:

Ping from PBR-Core to PC#1: Traffic is routed through the firewall as expected

*Aug 30 15:52:41.361 MST: IP: s=192.168.30.2 (local), d=10.10.150.5, len 100, policy match route map local, item 300, permit
*Aug 30 15:52:41.361 MST: IP: s=192.168.30.2 (local), d=10.10.150.5 (Vlan400), len 100, policy routed local to Vlan400 192.168.30.1
*Aug 30 15:52:41.364 MST: IP: s=192.168.30.2 (local), d=10.10.150.5, len 100, policy match route map local, item 300, permit

Test2: 

Ping from PC#1 to PBR-Core: Traffic is routed laterally and does not go through firewall; however PBR-Core sends ICMP reply back through firewall which is expected. What is not seen is a policy match for PC#1 (10.10.150.5) to PBR-Core (192.168.30.2). This traffic is defined in access list 130 but for some reason the inbound traffic on interface 150 does not check against the route-map.

*Aug 30 15:55:23.659 MST: IP: s=192.168.30.2 (local), d=10.10.150.5, len 60, policy match route map local, item 300, permit
*Aug 30 15:55:23.659 MST: IP: s=192.168.30.2 (local), d=10.10.150.5 (Vlan400), len 60, policy routed local to Vlan400 192.168.30.1

Test3:

Ping from PC#1 to Switch#1: No log appears on the PBR-Core showing inbound traffic against the route-map. However, the traffic is observed on the firewall and traffic to and from Switch#1 to PC#1 behave as desired.

What am I missing? My money is on one simple command that's needed on the 4500x. Thanks in advance.

[paul driver]

Hello

try the following:

access-list 115 permit ip 10.10.150.0 255.255.255.255 192.168.30.0 255.255.255.0

route-map pcnm permit 300
match ip address 115
set ip next-hop 192.168.35.1

[jnesbitt1]

Hi Paul,

Thanks for the quick reply. I'll give that a try on Monday when I get back to the office. One question I do have with taking the match statement out is will that direct all traffic regardless of destination to route through the 35.1 address? One thing I forgot to put in the initial architecture is end users in different subnets can still talk laterally across the same zone. For example PC1 to PC2 without having to go through the firewall.

I'll let you know the results of taking the match statement out.

-Jon

[paul driver]

Hello

Yes i have amended it now to accommodate that -  Just try and let us know the result.

One question why do you want traffic sourced from vlan 400 to PBR to a different next hop?

[jnesbitt1]

Are you talking about the local policy for traffic originating on the core? If so, the reason is because traffic that is being sourced from the core's IP address of 192.168.30.2 gets routed to 192.168.35.1. I believe this is because the gateway of last resort of 0.0.0.0 0.0.0.0 192.168.35.1 is overriding the ip default gateway of 192.168.30.1 that is configured on the core. 

[paul driver]

Hello

So why not just change it so you have a default route to 192.168.35.1 (vlan 405) of the FW?  From you OP the connection between the fw and the core looks like it supports both vlans 400 -405 ?

[jnesbitt1]

I will test this first thing Monday morning and let you know. The reason I want the V400 going to a different next hop is to keep the MGT plane and data plane separate. This also allows me to utilize the firewall to control what traffic and type of traffic can access the switches. Without local PBR the V400 would hit the 35.1 address on the firewall and thus be put in the wrong security zone making it so it would not get checked against the correct firewall rules. I still use ACLs on the switches to control what IPs can access the switches but introducing the firewall to the mix helps create a defense in depth.

-Jon

[jnesbitt1]

Okay, I tested the config you suggested. Unfortunately it did not function as desired. Test#1 and #2 had the same results and test #3 ended up breaking whereas traffic from PC#1 to SW#1 routed laterally across with the return ICMP from SW#1 going through the firewall.

[jnesbitt1]

Bumping. Still having the same issue. Anyone have any ideas?

[jnesbitt1]

Bumping again as a solution has not been found. Any help on this is much appreciated.

[Jon Marshall]

Paul’s acl suggestion is correct but the masks are wrong. 

It should be -

access-list 115 permit ip 10.10.150 0.0.0.255 192.168.30.0 0.0.0.255

Jon

[jnesbitt1]

Hi Jon, thanks for the reply. I'll head over to my lab and give this a shot. I should have results in about an hour.

[jnesbitt1]

Hello again. Okay, I made the changes to access list 115 and the results did not change. The traffic behaves the exact same way as in my initial testing. Could this be a bug in the 4500x IOS since the 3750x worked just fine? Also just to confirm I did a show route-map command and the packet counter for Policy routing matches did not increase.

[Jon Marshall]

I have just looked at your diagram again and it makes no sense. 

You have the 192.168.1.0/24 subnet on both sides of a L3 interconnect in vlan 405 which means if you do send the traffic from a PC to the firewall it has a locally connected interface in the 192.168.1.0/24 subnet so it will never route it back to your switch because you don't route within the same IP subnet. 

Or is that link between the switch and firewall a trunk link ? 

Jon 

[jnesbitt1]

Hi Jon,

The L3 link between the core and FW is a trunk link. What I want to happen is use the firewall to determine which traffic and what type of traffic can go to and from the MGT plane of the switches. In this case the MGT plane is being sourced from VLAN 400 on the switches. I have attached a couple pictures as well. Please keep in mind this is in a lab so the configs are very basic

Thanks again for taking the time to look at this. Please let me know if there are any configs you'd like to see.

-Jon