PBR Route decision based on source-address

[MadisonPropertyManagement]

Hello Community,

Long time follower, first time poster. I really got stumped on this one, and could use some real help to get things running the way I'd like. So I recently got a second ISP for our organization. We're using more and more cloud services, so the connection to the internet becomes as critical as our internal network. With the second ISP, I'd like all the end users traffic to flow in and out of there (unless the link fails), while all our services flow through the other ISP (webservices, email, SQL, VPN). Eventually I'd like to setup failover for both, to each other. But for now, I'd really just like to try to focus on getting this separation setup. It's all in a test enviroment, so you'll see things like, a single IP address represents all our users (as far as the ACLs go).

So here's what I was trying to do. Make it so that ISP1 is our default gateway (since all our services will be run out that, it'd be easier to let all those go out the default, and then just configure one rule for all the users to go the other way, rather than let the users have the default gateway, and configure all the services one at a time to go out ISP1). So ISP1 is the gateway, and the second ISP has a default route with a higher metric setup. Then, using a combination of SLA, Tracking, Route-Maps, and NAT, I'd be able to send the users out to ISP2. Here's what I got:

-ISP1 = S0/0/0

-ISP2 = FA0/1

-Inside=FA0/0

route-map NAT_UsersISP2 permit 10
 match ip address 7
 match interface FastEthernet0/1

route-map NAT_UsersISP1 permit 10
 match ip address 7
 match interface Serial0/0/1

ip nat inside source route-map NAT_UsersISP1 interface Serial0/0/1 overload
ip nat inside source route-map NAT_UsersISP2 interface FastEthernet0/1 overload

ip route 0.0.0.0 0.0.0.0 ISP1-Gateway

ip route 0.0.0.0 0.0.0.0 ISP2-Gateway 10

This takes care of the users NAT'ing, I would then repeat something similar for each of the services. Access-list 7 is the users, access-list 8 is the services

access-list 7 permit host 10.3.250.3
access-list 8 permit host 10.3.250.4
access-list 8 permit host 10.3.250.5
access-list 8 permit host 10.3.250.7

ip sla 1
 icmp-echo ISP2-Gateway source-ip ISP2-Public-Address
 frequency 10
 exit
ip sla sche 1 start now life forever
track 1 ip sla 1
exit

ip sla 2
 icmp-echo ISP1-Gateway source-ip ISP1-Public-Address
 frequency 10
 exit
ip sla sched 2 start now life forever
track 2 ip sla 2
exit

route-map Everything_Map permit 10
 match ip address 8
 set ip next veri ISP1-Gateway 1 track 2
 set ip next veri ISP2-Gateway 2 track 1

route-map Everything_Map permit 20
 match ip address 7
 set ip next veri ISP2-Gateway 1 track 1
 set ip next veri ISP1-Gateway 2 track 2

int f0/0
 ip policy route-map Everything_Map

So what my thought was that because routing occurs before Nat'ing, then as the packets come from our users into interface f0/0, then they would be assigned a new gateway/next hop to go to, then they would be NAT'd to the appropriate public IP address. Am I at least on the right track? Because right now everything is still going out ISP1, which I figure is because the default gateway is set as such, meaning, it's hitting the routing table before hitting my route map. But I thought that's the reason to put the route map on the ingress interface, is so it's processed as the packet arrives, and before it hits the routing table. Please, any help would be great. Thanks.

[LJ Gabrillo]

Hi Mads,

route-map syntax typically have a 'set' command:

So, if you want it to go to a desired interface:

route-map NAT_UsersISP2 permit 10
 match ip address 7
 set interface FastEthernet0/1

using all match commands typically says, "oh you are traffic from ACL 7 so you are classified as NAT_UsersISP2 and oh! traffic from f0/1 so i'll classify you as NAT_UsersISP2 as well"

Try setting the "set" command to tell the router what to do with the traffic :)

[Richard Burts]

While the approach suggested in this response seems intuitive it is badly mistaken. When you want to use a route map to control NAT you want to have two match statements (one matching an ACL and the second matching the interface) and you do not want any set statements.

I believe that the reason that user traffic is going out ISP1 is that there are two set statements in the route map instance 20. Since there is a default route configured and since there is verify availability configured with tracking if the first set is not going to be executed there is no need for the second. Just let the default route do its work.

HTH

Rick

[LJ Gabrillo]

Hi Richard, thanks for the corection just verified this as well and yeah Richard is right :) let the default route do their thing and let PBR do its xD just have 2 route maps each pointing to its desired ISP

[MadisonPropertyManagement]

So what you're suggesting is that I remove the second set statement from the Everything_Map ? Just to be clear that I'm trying to route based on the source address. Which is why I have statement 10 (which routes all the users from the match statement out one ISP), and statement 20 (which matches the services, and routes them out the other way first) If there is failure on either side, I want each set of traffic to fail over to the otherside. So if ISP1 goes down, I want to send both the users and services out ISP2.
I know that the NAT'ing works well, and I've tested it, but I don't see the route maps being hit. Strangely enough I see a packet come in and get routed out like it normally should, based on the routing table. 

I applied the route map on the incoming interface so that this would not happen, so that it would have to evaluate the route map first, and then route afterwards. At least that's what I thought routers did: Evaluate Inbound Route Maps, then Route based on routing table, then NAT. So I went through this logically in my head, and with that in mind it should have hit Route map Everything_Map, then be changed before going to the routing table and onward to NAT.

Am I mistaken, or missing something in that process? Or perhaps have applied the route map in the wrong place, because right now everything gets routed out the routing tables default gateway, and not being changed to what I would expect to see because of the route maps.

[Richard Burts]

It seems that my suggestion was not clear, so let me try again. You have configured this

route-map Everything_Map permit 20
 match ip address 7
 set ip next veri ISP2-Gateway 1 track 1
 set ip next veri ISP1-Gateway 2 track 2

my suggestion is to remove the last line. My reason for suggesting this is that I believe that the last line is superfluous. On the first set statement you are specifying verify availability and track so  the router will check and if this next hop is not available it will not use it and go on. The second set statement has the same effect as the default route, which is what would be used if the second set statement were not there. Since the second set statement does not change any behavior then I think that you do not need it and it can be removed.

I am not sure why everything would be going through ISP 1. I wonder what the results are of your verify availability and track. Can you post the output of show track?

HTH

Rick

[MadisonPropertyManagement]

First off, Richard, I've read up on your credentials and you have quite a successful background, so I am very glad that you decided to work with me on this. I'm confident in your abilities to help me get to the bottom of this.

So, to continue on though, I did understand your first post, and you are right that by having the second set command is just redundant, and unnecessary since the default gateway will catch that. I've since removed it from my config and yet the same results. My problem is that I look at this and it looks very sound.

If you're coming from 10.3.250.3, then your first route/next hop should be out to ISP2-Gateway. If ISP2-Gateway is not available then use the routing table to make your decision.

But what I found is that all the traffic is going out to ISP1-Gateway. This is the real problem.

Luckily I have my own router stack at home including 4 x 2821, 1x 3845, 4 x 3550, 2 x 2950, 2 x 3750, 2 x 1841, 2 x 2621xm, 1 x ASA 5510, 1 x ASA 5505, and one ESXi box with many nics to simulate being in many locations on the network. Oh yeah, plus all kinds of HWICs, Network analyzers, gigastacks,  etc... So I've built this all at home, and can run many of the debug commands without having to parse through to look for the lines I want to see.

I've attached a crude drawing of what I have setup from GNS3. Note that "R4" or the "Border Router" is the one which I am trying to run all these route maps on. (see attached)

Also here are some things which you may find useful:

debug ip icmp
*Jul 23 00:18:29.031: ICMP: echo reply sent, src 4.2.2.1, dst 50.93.210.3
*Jul 23 00:18:30.035: ICMP: echo reply sent, src 4.2.2.1, dst 50.93.210.3
*Jul 23 00:18:31.047: ICMP: echo reply sent, src 4.2.2.1, dst 50.93.210.3
*Jul 23 00:18:32.059: ICMP: echo reply sent, src 4.2.2.1, dst 50.93.210.3

show track
Track 1
  IP SLA 1 state
  State is Down
    1 change, last change 01:13:36
  Latest operation return code: Timeout
  Tracked by:
    ROUTE-MAP 0
Track 2
  IP SLA 2 state
  State is Up
    2 changes, last change 01:13:01
  Latest operation return code: OK
  Latest RTT (millisecs) 1
  Tracked by:
    ROUTE-MAP 0

So the ICMP debug command was issued on "R2" or "ISP2". Then the ping was initiated from the host at 10.3.250.3, and it went out towards ISP1 (out the serial link) instead of ISP2 (which is what the route map should be telling it to do). I only know this because the IP address which the ICMP message is sourced from is the NAT'd address from the border router to ISP1.

But attached is also the show track command you asked for. 

So how come things are still going out ISP1 even though the route map is in place? Is the route map being applied properly? On the inbound interface right? because I applied it on the interface coming from the ASA. Am I wrong in thinking that it gets applied in this order: route map first, then routing table, then NAT. I'm pulling my hair out trying to figure out what's wrong. Because it all seems right to me.

Let me know, I'll keep you posted as well.

[Richard Burts]

I believe that we are making some progress and that some of the information in your recent post will be quite helpful.

First let me say that I believe that your logic is correct - the ip policy which references the route map is applied to the interface where the traffic enters the router.

Here is what I believe is most significant in the recent information that you sent

Track 1
  IP SLA 1 state
  State is Down

and here is where that is used

route-map Everything_Map permit 20
 match ip address 7
 set ip next veri ISP2-Gateway 1 track 1

so if track 1 state is down then the verify availability in this statement will not allow the ISP 2 gateway to be used. And the result is that all traffic will be sent using ISP 1. So now that we know why IP 2 is not being used we need to figure out what is the issue with track 1.

I would suggest that you start with a ping to the ISP 2 gateway address with source address of the interface connecting to ISP 2 (which is what your track is trying to do). I would also suggest that you do a traceroute to the ISP 2 gateway address and check how the router is attempting to get there.

HTH

Rick