2 Gateways of last resort:-Route based on source subnet to unknown destinations (PBR): Cisco 3750x switch

[VRAI]

Hi,

I have some new VLANs on my 3750x IOS:15.0(2)SE C3750E-UNIVERSALK9-M (SDM-Prefere routing enabled) switch that I need to direct to a different IP address/gateway for the purposes of internet access than the rest of the existing VLANs. Both sets of VLANs (Old and New) are still required to talk to each other. 

Looking through some similar scenarios on this discussion forum I came across a couple of possible solutions. One of them being PBR which seemed simple enough but for some reason when I tried it I could only get the traffic to hit the policy (the hit count goes up on the route-map/ACL) but the policy still routes it out of the pre-existing Gateway #1

Initially, I created the PBR and added a new default gateway (Gateway#2) with a higher metric so it does not initiate Equal cost load balancing because that would defeat the purpose of what I am trying to do. When I put the higher metric on gateway#2  noticed that the traffic from the new VLAN source was still taking the old gateway (Gateway#1). 

my aims:

1. Route traffic from new VLANs destined for unknown destination based on source IP subnet to a different gateway
2. Continue to have pre-existing VLANs use the old gateway 
3. Allow existing and NEW VLANs to still communicate with each other

What I want to know is:

1.  Is my assumption correct that you can have 2 gateways of last resort for unknown traffic where one of the gateways is used only for source-based traffic matching a policy and is destined for internet?
2. Does the "set ip next-hop" command force all PBR matched traffic to use that next-hop IP for ALL traffic no matter if it is internet bound /unknown or not? (not what I want) or will it send it if there is no matching destination?
3. Can both old and new VLANS still talk to each other

Example config:

interface vlan500

Description NEW_VLAN 

IP address 192.168.50.254 255.255.255.0

IP policy route-map PBR

!

access-list 101 permit ip 192.168.50.0 0.0.0.255 any

!

route-map PBR permit 10

match ip address 101

set ip next-hop 192.168.50.253

!

ip route 0.0.0.0 0.0.0.0 192.168.1.253

ip route 0.0.0.0 0.0.0.0 192.168.50.253 10

!

I want to avoid having to do VRF if at all possible but depending on your answers I may not have a choice. In which case I will come back with a follow up on how I will do that.

Thanks!

[Georg Pauwen]

Hello,

the problem is that only one static default route appears in the routing table. Unless you 'trick' the router into believing there are actually different gateways. Here is how you do that:

First, create static routes for non-existing destinations. Make sure the IP addresses you use are not actually in use anywhere in your network. In this example, I have used 10.1.1.1 and 10.1.2.1. 

ip route 10.1.1.1 255.255.255.255 192.168.1.253
ip route 10.1.2.1 255.255.255.255 192.168.50.253

Now you can enter static default routes, and they both will show up in the routing table:

ip route 0.0.0.0 0.0.0.0 10.1.1.1
ip route 0.0.0.0 0.0.0.0 10.1.2.1

You need to change your route map to:

route-map PBR permit 10
match ip address 101
set ip next-hop 10.1.2.1

Give that a try and see if you get it to work.

[VRAI]

Thank you, I will try this.

Do I really need to change the original static route (0.0.0.0 0.0.0.0 192.168.1.250) that is already in place for my original subnets/vlans? Unless I am mistaken that will disrupt the existing setup.

Can I not just leave the existing one in place and do what you suggest for the new Vlan subnets only?

[Giuseppe Larosa]

Hello @VRAI ,

be aware that in writing the extended ACL 101 you can deny all the traffic to internal destination firsts in order to have them routed by destination and only at the end of ACL 101 you will have a permit statement for traffic to the iinternet.

access-list 101 deny ip 192.168.50.0 0.0.0.255 192.168.0.0 0.0.255.255

access-list 101 permit ip 192.168.50.0 0.0.0.255 any

only traffic matching the second line will be processed by PBR.

The specified next-hop should be reachable at L2 ( an ARP entry for it must exist on a transit/backbone VLAN)

a dedicated Vlan fo the next-hop is to be preferred or a routed p2p link so that if the next-hop fails there is a fallback to standard destination based routing.

This is because an SVI autostate the L3 interface is up until at least one L2 interface either access or trunk is up/up and in STP forwarding state.

This can make difficult to detect next-hop failure if the next-hop is in a shared multi port Vlan.

Edit1:

Also you don't need a second default route to the default gateway 2 because set ip next-hop does not check the IP routing table just the reachability of the next-hop is checked.

If you would use set default ip next-hop the IP routing table would be checked first but set ip next-hop does not check the IP routing table.

Edit 2:

the device gateway2 needs routes for the new Vlans IP subnets in order to be able to send back return traffic to your multilayer switch.

In addition it will need NAT configuration for their IP subnets.

Hope to help

Giuseppe

[VRAI]

Thank you I will investigate this.

Is what Georg said in his reply in regards to configuring a route for a non-existent network still valid to "trick" the router into seeing both old and new gateways as up?

[VRAI]

Hi Giuseppe,

I meant to ask if the changes will only mean that the Servers in the new VLAN will use the new gateway of last resort #2 when they are going to unknown destination/internet rather than sending all traffic to that gateway regardless as a default gateway?

Also note that servers in the DMZ subnets have the Internet CPE/FW as their default gateway where their VLANs terminate (.250).

Topology:

                             [Internet Gateway-FW#1]                                                          [ (new)Internet Gateway-FW#2]          

                              {192.1681.250}                                                                                 {192.168.50.250}                     

                                          |                                                     |                                                      |

       [--------------------------------------C3750x Switch--------------------------------------------]

            | {OLD-VLAN SVI ip }                {NEW - VLAN SVI ip}                                       {NEW - DMZ VLAN#}          |

            |  {192.168.1.254}                         {10.1.1.254}                                                         {No IP}                     | 

            |________________________________________________________________________________|

                                                                             ^                               ^ 

                                                                             |                                  |

                                                                             |                                  |

                                                                             | Layer 2 Port-channel |

                                                                             |                                  |

       [-----------------------------------------Nexus leaf switch--------------------------------------------]

         [OLD-VLAN SVI ip  Gateway]         [NEW - VLAN SVI ip Gateway]                       [NEW DMZ- VLAN#]   

                 {192.168.1.253}                                 {10.1.1.253}                                             {No IP}

                            |                                                       |                                                                |       

                       Server                                                Server                                               DMZ Server -inside

             IP: 192.168.1.1/24                                      IP: 10.1.1.1/24                                  IP: 192.168.50.1/24                 

             DGW: 192.168.1.253                                  DGW: 10.1.1.253                               DGW: 192.168.50.250 

I want to make sure that:

• Both Servers on Old and New vlans are able to talk on the L3 switch as normal (They are using their default gateways which are set as the VLAN SVI IP addresses on the L3 switch)
• When new servers on new vlans require internet/unknown destination they use the additional/new Gateway of last resort #2
• Old servers/vlans continue to use old gateway of last resort#1 for unknown destinations/internet

Thanks!

[Georg Pauwen]

Hello,

that should be possible. The additional config would look like this then:

ip route 10.1.2.1 255.255.255.255 192.168.50.253

ip route 0.0.0.0 0.0.0.0 10.1.2.1

You probably need a second route map sequence in order to send the rest of the traffic out the original default route (0.0.0.0 0.0.0.0 192.168.1.250):

route-map PBR permit 10
match ip address 101
set ip next-hop 10.1.2.1

!

route-map PBR permit 20

set ip next-hop 192.168.1.250

[VRAI]

I am beginning to wonder if it would be easier to specify match criteria based on both source and destination addresses (Not sure if the destination can be 0.0.0.0 0.0.0.0 in a policy-map?)

[Georg Pauwen]

It is becoming hard to follow what you eventually want to accomplish. Which traffic do you want to go where ? Can you briefly re-summarize ?

[Richard Burts]

I believe that the original poster was pretty clear in describing what they want to happen. 

1) Existing vlans will continue to use the existing default route to get to the original gateway of last resort.

2) New vlans and existing vlans can talk to each other.

3) New vlans will use a new gateway of last resort.

That should be fairly easy to achieve using PBR.

It is not clear whether the original poster would want this new gateway of last resort to operate as a failover gateway for the original vlans in the event of a problem with the original gateway. So clarification on this point would be appreciated. Pending that clarification I am going to assume that it is not desired for the new gateway to be a failover for the original vlans. And in that case you do not want to add a second default route with a higher AD. Just one default route will achieve what is desired.

The purpose of PBR is to provide an alternative approach to forwarding certain traffic in the network. Normal routing uses the destination address and chooses the best way to forward traffic using only the destination address. With PBR we can make forwarding decisions based on other attributes such as the source address.

To implement PBR the first step is to configure an access list which will identify which traffic which should use the alternate forwarding logic. The next step is to configure a route map which will have a match statement to associate the acl with PBR and will have a set statement which will identify the different next hop. The final step is to assign the route map to the interface on which the traffic will arrive at the router/switch.

I am going to start by describing the set statement, because there are some alternatives in how to configure the set statement which will impact how you configure the access list. One approach (and the more common one) is 

set ip next-hop <ip address of new gateway>

The other approach is

set ip default next hop <ip address of new gateway>

If you want to use the more common approach of set ip next-hop then the access list should deny traffic for which you want normal forwarding to operate and then should permit other traffic. In this case any traffic from new vlan to old vlan should be denied so that it will use normal forwarding. (Sometimes people are worried about denying traffic in the access list because they fear that the traffic would be dropped if it was denied in the access list. But there is no reason to worry about the deny of traffic because it is only denying that the traffic from new vlan to old vlan gets special routing treatment.) Then the access list should permit traffic from new vlan IP addresses to any.

If you want to use the other approach of set ip default next-hop then the access list can be more simple. In this case the access list needs only to permit traffic from new vlan IP addresses to any destination. This form of the route map will do normal forwarding for any destination which matches an existing route in the routing table and will forward any traffic which would have used the existing default route and send it out the new gateway of last resort.

Note that if there are multiple new vlans, then a single route map should be ok and that route map would be applied to each of the new vlan interfaces.

If you want more information about the two approaches in the set statement this link might be helpful

https://www.cisco.com/c/en/us/support/docs/ip/ip-routed-protocols/47121-pbr-cmds-ce.html?dtid=osscdc000283

[VRAI]

Thank you, in the beginning, you have summarised what I am after perfectly.  I will read your post to digest it and try to comprehend and get back to you. 

Thank you both for your time so far!

[VRAI]

Hi to answer your initial question:

1) "It is not clear whether the original poster would want this new gateway of last resort to operate as a failover gateway for the original vlans in the event of a problem with the original gateway. So clarification on this point would be appreciated. Pending that clarification I am going to assume that it is not desired for the new gateway to be a failover for the original vlans. And in that case you do not want to add a second default route with a higher AD. Just one default route will achieve what is desired."

• Your assumption is correct, failover gateway is not a necessity.

2)  "(Sometimes people are worried about denying traffic in the access list because they fear that the traffic would be dropped if it         was denied in the access list. But there is no reason to worry about the deny of traffic because it is only denying that the traffic from new vlan to old vlan gets special routing treatment.) Then the access list should permit traffic from new vlan IP addresses to any."

• You are exactly right this was my worry and thank you for clearing that up. This is an important lesson with a gap in my knowledge.

3) "IP next-hop" vs "IP default next-hop.

So is my understanding/summary correct:

"IP next-hop" method

 Access-list should:

• Deny traffic (from OLD VLANS) for which you want normal forwarding to operate
• Permit other traffic(New VLANS). (In this case, any traffic from new VLAN to old VLAN will be "skipped" by Policy so that it will use normal forwarding based on pre-existing Dynamic and static routing already in place). 
• Permit traffic from new VLAN IP addresses to any ("IP any any").
• Apply route map to VLAN SVI IP interfaces on 3750 switch 

"IP default next-hop" method

• Access list needs only to permit traffic from new VLAN IP addresses to any destination. This form of the route map will do normal forwarding for any destination which matches an existing route in the routing table and will forward any traffic which would have used the existing default route and send it out the new gateway of last resort.

Comment: This is the bit that trips me up. Will this forward traffic from New VLANS to the destination unknown (0.0.0.0/0.0.0.0) out of the new gateway of last resort? While Old VLANs will continue to use old\existing gateway of last resort and inter VLAN communication will remain working as per routing table?

[Richard Burts]

Thank you for the clarifications. It is good to know that we do not need to provide the new default gateway as a failover for the old vlans. This simplifies things quite a bit and means that existing routing logic should not be changed and that PBR is an appropriate solution.

I am glad that my comment that with PBR the access list denying traffic only denies special routing but does not drop the traffic was helpful. This is a point that is frequently not understood correctly.

There are a few things that I would comment about in your understanding of the ip next-hop method. You said "Deny traffic (from OLD VLANS) for which you want normal forwarding to operate" It is traffic from new vlans to old vlans that should be denied. Remember that this access list will be applied on the interface for the new vlan(s). So new vlan is the source and old vlan is the destination. 

You also said "Permit traffic from new VLAN IP addresses to any ("IP any any")." 

I would not do any any but would specify the new vlan addresses as the source to any destination.

Your comment about ip default next-hop says that this is the part that trips you up. But your statement of your understanding is right. In this implementation the access list would permit traffic with the new vlan addressing as source to any destination. The PBR logic looks at each packet that is to be forwarded from the new vlan interface. If there is a route in the current IP routing table that matches this destination then the route from the routing table is used (normal routing). Any packet that would have used the default route in the routing table (original default gateway) is forwarded using the new default gateway.

[VRAI]

Thank you for the clarification on the "IP next-hop" method.

Regarding the "IP default next-hop":

your comment: "In this implementation, the access list would permit traffic with the new vlan addressing as source to any destination. The PBR logic looks at each packet that is to be forwarded from the new vlan interface. If there is a route in the current IP routing table that matches this destination then the route from the routing table is used (normal routing). Any packet that would have used the default route in the routing table (original default gateway) is forwarded using the new default gateway."

This last statement is only for the matched/new VLANs in the ACL right not the old VLANs?

I am trying to map this out so I am sure how the traffic paths would map out to:

                  Packet                                    |    Interface       |  |      ACL      |              |           PBR          |

[Src: New VLAN | Dst: unknown/int]---->{New VLAN SVI} |  |= Src-Match|------>(IP default-next hop)----->[New default gateway] 

[Src: New VLAN | Dst: New VLAN]---->{New VLAN SVI}| |= Scr-Match |----->(use global routing table)----->[New VLAN SVI] 

[Src: Old VLAN | Dst: New VLAN]---->{New VLAN SVI} | = No Match    |----->(use global routing table)----->[New VLAN SVI] 

[Src: New VLAN | Dst: Old VLAN]---->{New VLAN SVI} | = DENY Match found |--->(use global routing table)----->[Old VLAN SVI] 

[Src: Old VLAN | Dst: unknown/internet]---->{OLD VLAN SVI} | (Not affected by any policy uses normal routing)---->[Old Default gateway]

[Src: Old VLAN | Dst: Old VLAN]---->{OLD VLAN SVI} | (Not affected by any policy uses normal routing)

I know some of this is wrong but keep getting thrown. Also, how would the next-hop method look in comparison?

[VRAI]

Also, If as discussed, I deny traffic from new VLAN to all Old VLANS is it ok to use summary-address like deny from old VLANs to 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16  because I don't want to have to put in all the subnets for each specific VLAN on the switch?