2 Gateways of last resort:-Route based on source subnet to unknown destinations (PBR): Cisco 3750x switch

[VRAI]

Hi,

I have some new VLANs on my 3750x IOS:15.0(2)SE C3750E-UNIVERSALK9-M (SDM-Prefere routing enabled) switch that I need to direct to a different IP address/gateway for the purposes of internet access than the rest of the existing VLANs. Both sets of VLANs (Old and New) are still required to talk to each other. 

Looking through some similar scenarios on this discussion forum I came across a couple of possible solutions. One of them being PBR which seemed simple enough but for some reason when I tried it I could only get the traffic to hit the policy (the hit count goes up on the route-map/ACL) but the policy still routes it out of the pre-existing Gateway #1

Initially, I created the PBR and added a new default gateway (Gateway#2) with a higher metric so it does not initiate Equal cost load balancing because that would defeat the purpose of what I am trying to do. When I put the higher metric on gateway#2  noticed that the traffic from the new VLAN source was still taking the old gateway (Gateway#1). 

my aims:

1. Route traffic from new VLANs destined for unknown destination based on source IP subnet to a different gateway
2. Continue to have pre-existing VLANs use the old gateway 
3. Allow existing and NEW VLANs to still communicate with each other

What I want to know is:

1.  Is my assumption correct that you can have 2 gateways of last resort for unknown traffic where one of the gateways is used only for source-based traffic matching a policy and is destined for internet?
2. Does the "set ip next-hop" command force all PBR matched traffic to use that next-hop IP for ALL traffic no matter if it is internet bound /unknown or not? (not what I want) or will it send it if there is no matching destination?
3. Can both old and new VLANS still talk to each other

Example config:

interface vlan500

Description NEW_VLAN 

IP address 192.168.50.254 255.255.255.0

IP policy route-map PBR

!

access-list 101 permit ip 192.168.50.0 0.0.0.255 any

!

route-map PBR permit 10

match ip address 101

set ip next-hop 192.168.50.253

!

ip route 0.0.0.0 0.0.0.0 192.168.1.253

ip route 0.0.0.0 0.0.0.0 192.168.50.253 10

!

I want to avoid having to do VRF if at all possible but depending on your answers I may not have a choice. In which case I will come back with a follow up on how I will do that.

Thanks!

[Richard Burts]

This is surprising. Your config does look correct. Can you post the configuration of the interface where the new/second gateway is? Can you ping the next hop of the new/second gateway?

[VRAI]

I have been checking to see if the next-hop gateways are in the arp table for that new gateway.

interface GigabitEthernet1/0/24
description New gateway of last resort for new vlans
switchport access vlan 1231
switchport mode access
speed 1000
duplex full

I can't find the entries in the arp table or ping it but that might be because of some security feature on the FW for that port.

I suspect this may be the issue because for PBR to work I think the next hop gateway has to be in the arp table as its an access port

when I tried a different gateway(which was reachable) in the next hop command then it looked like it was being routes as per policy at least

I caught this in the debug logs:

Sep 7 20:04:20.502: % Warning: Next hop address is our address

not sure if this is informational or an issue

[Richard Burts]

Thank you for the additional information which is helpful. In what you posted you gave us this

interface GigabitEthernet1/0/24
description New gateway of last resort for new vlans
switchport access vlan 1231

But in a previous post you gave us this about vlan 1231

interface Vlan1231
ip address 10.1.1.254 255.255.255.0
ip policy route-map PBR_RMAP

If the new gateway is in vlan 1231 then it needs to have an IP address in 10.1.1.0. But you are making the new gateway address 192.168.50.250. That address is not part of vlan 1231. So this is the reason why your PBR was not working. 

Where is 192.168.50.0 subnet? Or if the new gateway really is on Gi1/0/24 then what address in 10.1.1.0 is the gateway. Once you get this resolved I am hopeful that your PBR will begin to work.

[VRAI]

I think I got confused when sanitising the output and mixed some up... :/

I will amend and try again....

[VRAI]

At this point, I am convinced that the Policy map has to be applied on the Nexus switch SVI's (not the 3750 svis) since the nexus SVIs are the ones that are acting as the default gateways for each VLAN both old and new.

The only issue is that Nexus 9ks don't support the "Deny" statement on PBR according to cisco and I would have to create two ACLs as described here:

https://community.cisco.com/t5/switching/nexus9000-deny-acls-in-pbr/td-p/2752365

I wonder if the Nexus would support the "IP default next-hop" version of what we are trying to accomplish that was mentioned originally but the 3750 platform does not support that which is why we went with the "IP next-hop" method instead?

[VRAI]

Ok so I have applied the changes and got mixed results.

1)The new VLANS route to the new gateway of last resort which is great, just what I wanted

2) After the change none of the RFC1918 addresses living over the WAN could reach the old Vlans connected to the switch but could reach the new VLANs fine....

I wonder if the fact that the Servers have the Nexus switch SVIs set as their default gateways mean the PBR should go on them rather than the 3750 switch which whos SVI is the default gateway for the Nexus switch

[VRAI]

Ok so I have applied the changes and got mixed results.

1)The new VLANS route to the new gateway of last resort which is great, just what I wanted

2) After the change none of the RFC1918 addresses living over the WAN could reach the old Vlans connected to the switch but could reach the new VLANs fine....

[VRAI]

I am going to try and use the "set IP default next-hop" approach since that will allow me to use/match an ACL with permit statements rather than the "Deny" with the "set IP next-hop" method. Just not sure if there is an implicit deny at the end of the statement or not because of the fact the nexus can't deal with deny/PBR...

should I have a permit ip any any at the end of the ACL?

[VRAI]

Great just found out that the nexus 9ks don't support "IP default next-hop"....

this is a complete nightmare!

[Richard Burts]

Sorry that this has gotten complicated. There have been many posts and several alternatives suggested. I would like to try to address several points and suggest what I think should work:

- the PBR route map needs to be applied on the L3 interface that is doing the routing for the new vlan/subnet. I believe that this would be the Nexus rather than the 3750. I am not clear how many new vlans there are. It is possible to do a unique route map (with a unique access list) for each of the new vlans. But that gets complicated. So I suggest using a single route map (with a single access list) that would be applied on multiple vlan interfaces.

- if the Nexus does not support the set ip default next-hop then just use set ip next-hop.

- If you are using set ip next-hop then the acl should start with 3 statements for each subnet of the new vlans which deny that subnet to 10.0.0.0, to 172.16.0.0, and to 192.168.0.0. After the deny statements the acl should permit ip any any.

- the route map should look something like this

route-map PBR permit 10

match ip PBR-ACL

set ip next-hop <new-gateway>