Reading the document below

benholtsclaw · ‎06-04-2015

We have a 4500x VSS at the core of our network. Up until this point, we have had one default route out to the Internet for the many vlans/subnets that terminate on the core switch. I've introduced a second gateway and am trying to selectively point vlans to it. I've never used PBR before, but thought that was the best option for what I'm trying to accomplish.

I have a client 10.32.0.2 in vlan32. The default route is set to the original gateway, an ASA at 10.10.253.253. The new gateway, a Barracuda NG Firewall, is at 10.10.253.251. Both are on vlan253. I have the following config.

interface Vlan32
 ip address 10.32.0.1 255.255.255.0
 ip helper-address 10.10.8.200
 ip policy route-map Vlan32RouteToBarracuda
!
ip route 0.0.0.0 0.0.0.0 10.10.253.253
!
!
ip access-list extended vlan32-acl
 permit ip 10.32.0.0 0.0.0.255 any
!
!         
route-map Vlan32RouteToBarracuda permit 10
 match ip address vlan32-acl
 set ip default next-hop 10.10.253.251
!

The route table is nothing more than the VLANs/SVI. Here is the first bit of it...

Gateway of last resort is 10.10.253.253 to network 0.0.0.0

S*    0.0.0.0/0 [1/0] via 10.10.253.253
      10.0.0.0/8 is variably subnetted, 146 subnets, 6 masks
C        10.1.0.0/24 is directly connected, Vlan1
L        10.1.0.1/32 is directly connected, Vlan1
C        10.4.0.0/24 is directly connected, Vlan4
L        10.4.0.1/32 is directly connected, Vlan4
C        10.6.0.0/24 is directly connected, Vlan6
L        10.6.0.1/32 is directly connected, Vlan6
C        10.8.0.0/24 is directly connected, Vlan8
L        10.8.0.1/32 is directly connected, Vlan8
C        10.9.0.0/24 is directly connected, Vlan9
L        10.9.0.1/32 is directly connected, Vlan9
C        10.10.0.0/22 is directly connected, Vlan10
L        10.10.0.1/32 is directly connected, Vlan10

When I try to hit an Internet address from the client in vlan32, I get this.


Jun  4 18:20:53.520: IP: s=10.32.0.2 (Vlan32), d=10.32.0.255, len 78, policy match
Jun  4 18:20:53.520: IP: route map Vlan32RouteToBarracuda, item 10, permit
Jun  4 18:20:53.520: IP: s=10.32.0.2 (Vlan32), d=10.32.0.255, len 78, policy rejected -- normal forwarding
Jun  4 18:20:53.520: IP: s=10.32.0.2 (Vlan32), d=10.32.0.255, len 78, policy match
Jun  4 18:20:53.520: IP: route map Vlan32RouteToBarracuda, item 10, permit
Jun  4 18:20:53.520: IP: s=10.32.0.2 (Vlan32), d=10.32.0.255, len 78, policy rejected -- normal forwarding
Jun  4 18:20:53.520: IP: s=10.32.0.2 (Vlan32), d=10.32.0.255, len 78, policy match
Jun  4 18:20:53.520: IP: route map Vlan32RouteToBarracuda, item 10, permit
Jun  4 18:20:53.520: IP: s=10.32.0.2 (Vlan32), d=10.32.0.255, len 78, policy rejected -- normal forwarding

At that point, for whatever reason, traffic doesn't seem to go to either gateway. As best I can tell, the packets don't seem to be hitting the ASA or the Barracuda. What am I missing here? Why does the debug say policy match, then policy rejected? Thanks for your help.

Rolf Fischer · ‎06-05-2015

Hi,

Why does the debug say policy match, then policy rejected?

The 4500-X supports hardware-based PBR, so possibly the debug ip policy output shows only the locally destined traffic. The destination here is the subnet's broadcast IP (10.32.0.255), so (policy) routing doesn't take place for those packets.

I assume the Barracuda's internal IP is reachable? Do you have an ARP table entry for 10.10.253.251? If so, is there any chance to verify if the client's external traffic arrives at the Barracuda or not?

Directing traffic by PBR to a parallel firewall during a migration is a commonly used approach, but often the other tricky part is the return traffic. You need to avoid that external traffic leaves your network through the Barracuda but the return traffic goes through the ASA and vice versa.

HTH

Rolf

benholtsclaw · ‎06-05-2015

I assume the Barracuda's internal IP is reachable? Do you have an ARP table entry for 10.10.253.251? If so, is there any chance to verify if the client's external traffic arrives at the Barracuda or not?

Yes, the client on vlan32 can ping 10.10.253.251 (the barracuda) without issue. The NG Firewall has a pretty good, real-time interface for seeing packets hit rules. I never get any indication that traffic from the vlan32 client hits the device.

On the other hand, if I add a static route on 4500-x to point some Internet network out toward the Barracuda, then everything works as expected.

Rolf Fischer · ‎06-05-2015

I know it makes little sense, but could you create a standard ACL (permit 10.32.0.0 0.0.0.255) and see if the behavior changes when you apply it to the policy-map instead?

Btw: What IOS and feature set are you running (CSCuq03562 - PBR: Not working in XE 3.5.0E under IPBASE license)?

benholtsclaw · ‎06-08-2015

Actually, when I first started trying to get PBR to work, I was using a standard ACL. I had switched to an extended, hoping that might make a difference, before I ever posted here. Started out with

access-list 1 permit 10.32.0.0 0.0.0.255

As far as feature set, I'm running

License Information for 'WS-C4500X-32'
License Level: entservices Type: Permanent
Next reboot license Level: entservices

In short, I never found the answer to the problem. Over the weekend, I switched the default gateway for the core to point to the new Barracuda. At this point, I won't be doing further troubleshooting because I don't have another next-hop to even test with.

Everyone, thanks for all the input.

devils_advocate · ‎06-05-2015

Reading the document below lists the following:

The set ip default next-hop command verifies the existence of the destination IP address in the routing table, and…

if the destination IP address exists, the command does not policy route the packet, but forwards the packet based on the routing table.
if the destination IP address does not exist, the command policy routes the packet by sending it to the specified next hop.

The set ip next-hop command verifies the existence of the next hop specified, and…

if the next hop exists in the routing table, then the command policy routes the packet to the next hop.
if the next hop does not exist in the routing table, the command uses the normal routing table to forward the packet.

Source -

http://www.cisco.com/c/en/us/support/docs/ip/ip-routed-protocols/47121-pbr-cmds-ce.html

If you change this:

route-map Vlan32RouteToBarracuda permit 10
match ip address vlan32-acl
set ip default next-hop 10.10.253.251

To this:

route-map Vlan32RouteToBarracuda permit 10
match ip address vlan32-acl
set ip next-hop 10.10.253.251

Does it work?

The 10.10.253.0/24 subnet will exist in the routing table as you have a directly connected interface so if the information above is correct then it will not be policy routing it, rather using the routing table to lookup the next hop.

Thanks

benholtsclaw · ‎06-05-2015

Actually, I had already tried the set ip next-hop. It doesn't work either. When I implemented that, I seemed to not have connectivity to any other subnet. In other words, with that set, I can't even ping my internal DNS at 10.10.8.13.

devils_advocate · ‎06-05-2015

Hmmm, then I am a little stumped as the configuration above suggests it should work.

Could you post a .txt file with the whole config of the 4500x?
You can omit hostnames, password etc.

Jon Marshall · ‎06-05-2015

Which suggests PBR is working ie. you couldn't access any other subnet because all traffic even internal was being sent to the firewall and it probably isn't setup to allow traffic back out of the same interface.

Your debug doesn't tie up with you trying to access an internet IP because as Rolf says the destination IP is a broadcast IP.

What does a traceroute show (although potentially not much if both firewalls don't respond to it).

Jon

Jon Marshall · ‎06-05-2015

Just for your info because I only found this out last week :-), that documentation is a bit misleading.

What it doesn't mention is that with the "set ip default next-hop ..." command the default route is not used ie. if there is a more specific route than the default route in the IP routing table it will use it but if the only matching route is the default route then it won't use it and instead will use the next hop specified in your PBR configuration.

Must admit came as a bit of a surprise to me until I tested it :-)

Jon

devils_advocate · ‎06-05-2015

Ahhh that does make sense, it's a shame the documentation is not very clear on that :)

It might explain why the broadcasts in the original post are not policy routed though?
They are designed for 10.32.0.255 which there is a directly connected route for.

Jon Marshall · ‎06-05-2015

The broadcast wouldn't be routed either way as far as I know, certainly not using the IP routing table but also with PBR I would have thought.

Although to be honest considering how surprised I was about the PBR and default route bit I'm not sure I can say anything definite about PBR at the moment :-)

The debugs don't tie up with accessing an internet IP though.

I did do a quick check to see if the "set ip default next-hop .." command is supported and it is so like you I can't see why it isn't working at the moment.

Jon

Policy-Based Routing on 4500x