topic Setting Gateway for Loopbacks in Routing and SD-WAN

Setting Gateway for Loopbacks

CCIE Aspirant — Mon, 16 Aug 2021 11:03:22 GMT

Hi,

i have a router which is connected to two isp. ISP1 and ISP2. we are running bgp with both of them. for inbound traffic we are receiving the traffic almost as expected but for outbound traffic all the traffic is following ISP1 always. and we do not wanted like that and need to send traffic to ISP1 and some to ISP2 (both ISP are sending default route through BGP). after doing many changes we did not get desired results so i went with other solution.

! Step 1

! Define SLA to track the default route and set default route according to availability.

! SLA for ISP1 as default Gateway

ip sla 15

icmp-echo 172.16.180.161

! ISP1 Side IP

frequency 5

ip sla schedule 15 life forever start-time now

!!!!!! Track for SLA 15 to check the status of next hop ISP1

track 15 ip sla 15 reachability

delay down 120 up 1

! Default route for all traffic is ISP1 with default metrics. and Track the availability of ISP1.

ip route 0.0.0.0 0.0.0.0 172.16.180.161 track 15

! Default route for all traffic to ISP2 with metric 15 set lower then 20 (EBGP)

! ISP2 side 172.17.4.137

ip route 0.0.0.0 0.0.0.0 172.17.4.137 15

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

! For Load balancing create route-map and call the ACL for preferred path.

! ACL to match the subnet we need to prefer to ISP1.

access-list 80 permit 192.12.176.0 0.0.0.255

access-list 81 permit 192.12.177.0 0.0.0.255

! ACL to match the subnet we need to prefer to ISP2.

access-list 82 permit 192.12.178.0 0.0.0.255

access-list 83 permit 192.12.179.0 0.0.0.255

!!!!! IP SLA for main ISP1 and backup ISP2

ip sla 16

icmp-echo 172.16.180.161

frequency 5

ip sla schedule 16 life forever start-time now

!!!!!! Track for IP SLA 16 for making the next hop for preferred subnet to ISP1

track 16 ip sla 16 reachability

delay down 120 up 1

!!!!! IP SLA for main ISP1 and backup ISP2

ip sla 17

icmp-echo 172.17.4.137

frequency 5

ip sla schedule 17 life forever start-time now

!!!!!! Track for IP SLA 16 for making the next hop for preferred subnet to ISP1

track 17 ip sla 17 reachability

delay down 120 up 1

!!!! Route map to set ISP1 as main and ISP2 as backup for subnet 80 and 81

route-map MAIN_ISP1_BACK_ISP2 permit 10

match ip address 80 81

set ip next-hop verify-availability 172.16.180.161 16 track 16

!!!! Route map to set ISP2 as main and ISP1 as backup for subnet 82 and 83

route-map MAIN_ISP2_BACK_ISP1 permit 10

match ip address 82 83

set ip next-hop verify-availability 172.17.4.137 17 track 17

int Loopback 100

ip address 192.12.176.1 255.255.255.255

ip policy route-map MAIN_ISP1_BACK_ISP2

int Loopback 101

ip address 192.12.177.1 255.255.255.255

ip policy route-map MAIN_ISP1_BACK_ISP2

Int Loopback 200

ip address 192.12.178.1 255.255.255.255

ip policy route-map MAIN_ISP2_BACK_ISP1

int Loopback 201

ip address 192.12.179.1 255.255.255.255

ip policy route-map MAIN_ISP2_BACK_ISP1

After doing this still the traceroute source as any of these loopback is following ISP1 which we do not want.

so kindly let me know where i am making mistake

Thanks

Re: Setting Gateway for Loopbacks

pman — Mon, 16 Aug 2021 20:33:27 GMT

Hi,

if your bgp peers send you 0.0.0.0/0 why you configure static route and tracking after this default route ?

After all, if for example peer is down from ISP1 or ISP2 then deafult-route (0.0.0.0/0 via bgp) will be deleted from your routing table and of course next-hop address (ISP peer address) will also be deleted from routing table.

If you want to send traffic through a particular ISP you can use weight for example instead of static route.

in this case traffic that need to exit from you router will uses the secondary ISP (which still publishes default-route 0.0.0.0/0) .

Another thing,

Is it mandatory for you to send traffic from specific addresses through a specific ISP?
Is there a reason you not use load-balancing for 0.0.0.0/0 via ISP1 and ISP2?

Re: Setting Gateway for Loopbacks

Richard Burts — Mon, 16 Aug 2021 19:04:17 GMT

My first comment is that the behavior described in the original post that routing was sending outbound traffic through a single ISP even though there are 2 ISP and each ISP is advertising a default route is the expected behavior. The default when running BGP is to pick a single best path and to use that with the other default route as a backup in case the primary route is withdrawn. You can change you BGP configuration to allow equal cost forwarding on 2 paths and that might solve your problem.

My second comment is that using Policy Based Routing is another way to accomplish load sharing using both ISP. But there are several things about your implementation that are questionable.

- Your PBR uses a separate access list to identify each subnet you want to forward. That is overly complex. It would be better if you used a single access list (with 2 permit statements) to identify traffic for ISP 1 and a single access list (with 2 permit statements) to identify traffic for ISP 2.

- The route map for PBR should be associated with the interface where that traffic arrives on the router. So for ISP 1 it should be the interfaces where 192.12.176.0 and 192.12.177.0 arrives. And for ISP 2 it should be the interfaces where 192.12.178.0 and 192.12.179.0 arrive. But the interfaces you use are loopback interfaces. What traffic arrives on a loopback interface?

- Normal PBR processes traffic that arrives at the router but does not process traffic that is generated by the router. If you want PBR to process your traceroute request you need to configured local policy.

Re: Setting Gateway for Loopbacks

CCIE Aspirant — Mon, 16 Aug 2021 21:18:24 GMT

Hi Pman,

Thanks for the reply,

if your bgp peers send you 0.0.0.0/0 why you configure static route and tracking after this default route ?

actually i had tried to go with setting bgp attribute to load balance or to prefer paths for my outbound traffic. the problem with that was all my traffic was going to isp1 although i tried to set the recommended attributes by cisco website or by trainers. at my gns3 lab it was working as expected but on real devices it was not working as expected. so my inbound traffic was working as expected but outbound had problem so i decided to go by this scenario as we need to work on some project and we need the setup. the thing i suspected might go wrong was the default route send by both isp and by bgp best path selection it always selected the next hop availability instead of going to check other attributes.

note that we have our own public ASN and both isp are in different ASN

may be i am wrong somewhere but i thought this strategy will solve my problems.

is it mandatory for you to send traffic from specific addresses through a specific ISP?
Is there a reason you not use load-balancing for 0.0.0.0/0 via ISP1 and ISP2?

for both of above questions i would say that we need to utilize both isp links in a way to prefer two subnets on isp1 and two on other and they should not do load balance instead they should do failover.

Re: Setting Gateway for Loopbacks

CCIE Aspirant — Mon, 16 Aug 2021 21:30:09 GMT

Hi @Richard Burts

Thanks for your wonderful reply.

i agree at the point the bgp behavior now is what it should behave. but the thing is i do not want load balance i need a sort of failover **bleep** load balance. i mean i have 4 public subnets and i need two of them should be advertised through isp1 and two should be isp2 as main and in case of failure of one isp the other isp should take the responsibility of all subnets.

Like for first two subnets isp1 is main and isp2 is backup and for last two subnets isp2 is main and isp1 is backup

so is that possible with bgp?

the loopbacks that i have defined are the ipsec source @( local router) and destination @( remote site) ip addresses and they should be publicly pingable.

so i want my some tunnels load should come from isp1 and some from isp2 but what currently going on is that traffic for tunnels preferred by isp1 and that preferred by isp2 are coming from isp1 and isp2 respectively but the response of these traffic is going out through isp1 only. so i need to resolve that problem by any means.

last how we can set local policy?

thanks for the support

Re: Setting Gateway for Loopbacks

MHM Cisco World — Mon, 16 Aug 2021 23:06:02 GMT

if both ISP advertise the same Routes,
config INBOUND route-map change the weight for some route to be prefer from ISP1 and other route to be prefer from ISP2.
for default route it must be either from ISP1 and ISP2.

Re: Setting Gateway for Loopbacks

Richard Burts — Tue, 17 Aug 2021 07:04:35 GMT

Thank you for the additional information. Your clarifications are helpful. In considering BGP and traffic for the Internet we need to consider separately what we do for inbound traffic (Internet to your networks) and outbound traffic (your networks going to the Internet).

- For inbound traffic there is a relatively simple solution in BGP. You would advertise a summary route for the 4 subnets to both ISP and you advertise the more specific routes for the first 2 subnets to ISP 1 and advertise more specific routes for the other 2 subnets to ISP 2.

- For outbound traffic there is not a good solution in BGP. If your intention is that the first 2 subnets go out through ISP 1 and use ISP 2 as a failover and that the last 2 subnets go out through ISP 2 and use ISP 1 as a failover then the better solution would be PBR.

- As I indicated in my previous response the PBR route map needs to be applied on the router interface where the traffic from those subnets enters the router. Configuring loopback interfaces with a /32 IP address from the subnet does not satisfy that requirement.

For information about local PBR this link provides information that I think will be helpful

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst4500/12-2/25ew/configuration/guide/conf/pbroute.pdf

Your recent response introduces a new aspect of what we need to address. You mention tunnels and ipsec. If there are site to site vpn tunnels then we need information about these and about how you want them to work to be able to give you good advice. Sending user traffic to the Internet is one thing and sending user traffic out through ipsec vpn tunnels is a very different thing.

Re: Setting Gateway for Loopbacks

CCIE Aspirant — Tue, 17 Aug 2021 08:04:14 GMT

Hi,

Thanks for the reply,

Actually the loopbacks are at our routers and we need it to go outside through isp1 and isp2. but in a way first two subnets should be reachable (inbound and outbound) as main via isp1 and last two subnets should be reachable ( inbound and outbound) as main via isp2 and in case of one isp link failure all of loopbacks defined at our router should follow ( inbound and outbound) the other isp. is that possible?

Best regards

Re: Setting Gateway for Loopbacks

CCIE Aspirant — Tue, 17 Aug 2021 08:28:23 GMT

Hi @Richard Burts ,

Thanks for your response. I appreciate your support.

So you mean to say with BGP, we can not achieve what I am trying to achieve right? and i have to use PBR. But with PBR as you said it is meant for traffic coming to our router interface and in this case the traffic is not from outside. The traffic is from the Loopbacks. As the loopbacks are sources of IPSEC tunnels to our remote sites. so i need to know the mechanism to let my outbound traffic for first two subnets follow ISP1 and last two subnets follow ISP2 and they should not look at the default gateway.

Regarding our IPSEC scenario, i will explain them.

we are using "Cisco IR809G-LTE-GA-K9" at our branch sites with cellular sim inside them. we have created point to point tunnel from these router to our router at head office. at branch router the loopback we defined at our HO router is destination for tunnel and source of tunnel is cellular.

when we establish the tunnel, then we send all our LAN traffic at sites to our Head office. so we need a way that some of our remote sites tunnel should follow inbound and outbound ISP1 and some should follow Inbound and outbound ISP2. So if tunnel is established from Site1 (ISP1 Loopback destination) let say , then I need all the LAN traffic of that specific tunnel should follow in and out ISP1 and for example Site2( ISP2 loopback destination) , then I need all the LAN traffic of that specific tunnel should follow in and out ISP2. But what is happening now that all the respected tunnels inbound traffic is coming through it respected ISP but the response traffic is only going out of ISP1 due to default gateway.

so what i am thinking that if i make our loopbacks to force them ( outbound) in a way to override the default gateway, then it might solve the problem.

Let me share the utilization of both ISP graph, which clears that inbound is coming through respected isp but outbound is always going through isp1 which makes the routing asymmetric.

Thanks and appreciate your support.

Re: Setting Gateway for Loopbacks

pman — Tue, 17 Aug 2021 14:29:32 GMT

about outgoing traffic:

Can you attach the following output: show ip bgp 0.0.0.0/0?

Re: Setting Gateway for Loopbacks

CCIE Aspirant — Tue, 17 Aug 2021 11:46:59 GMT

Hi @pman

The output is

========================================================================

Sup_32UR#sh ip bgp 0.0.0.0/0
BGP routing table entry for 0.0.0.0/0, version 1656
Paths: (2 available, best #1, table default, RIB-failure(17) - next-hop mismatch)
Not advertised to any peer
Refresh Epoch 1
25233, (received & used)
172.16.180.161 from 172.16.180.161 (x.93.193.x)
Origin IGP, localpref 100, valid, external, best
rx pathid: 0, tx pathid: 0x0
Refresh Epoch 2
29690 39386, (received-only)
172.17.4.137 from 172.17.4.137 (x.70.41.x)
Origin IGP, localpref 100, valid, external
rx pathid: 0, tx pathid: 0

================================================================

No i have not tried this " maximum-path 2" command

what it will do if i use it?

Appreciate your support

Best regards

Re: Setting Gateway for Loopbacks

CCIE Aspirant — Tue, 17 Aug 2021 12:09:58 GMT

Hi @Richard Burts ,

i tried to apply local policy as you suggested for the traffic that is locally generated, but i can see this command exist

"ip local-policy route-map"

(config-if)#ip local-policy route-map ISP_FES
^
% Invalid input detected at '^' marker.

=====================================================

HS-Edge#sh version
Cisco IOS XE Software, Version 03.13.01.S - Extended Support Release
Cisco IOS Software, ASR1000 Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 15.4(3)S1, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.

===============================================================================

Re: Setting Gateway for Loopbacks

pman — Tue, 17 Aug 2021 14:43:38 GMT

"table default, RIB-failure(17) - next-hop mismatch)"

I guess it happens because you point with static route that have better AD to the next-hop.

from Cisco:

" A rib-failure occurs when Border Gateway Protocol (BGP) tries to install the bestpath prefix into the RIB, but the RIB rejects the BGP route because a route with better administrative distance already exists in the routing table. "

another thing,

about the outbound traffic:
If you want to use both ISP1 and ISP2 (load balance from different AS numbers)

then you can use:
maximum-paths 2
bgp bestpath as-path multipath-relax

and again I apologize if I misunderstood,

Re: Setting Gateway for Loopbacks

CCIE Aspirant — Tue, 17 Aug 2021 12:39:37 GMT

HI,

Actually i am using static route as well you can see in my original post and it is causing RIB failure. so do u think that is the reason of Asymmetric routing.

Thanks for the support

Best regards

Re: Setting Gateway for Loopbacks

Richard Burts — Tue, 17 Aug 2021 16:43:13 GMT

If you configure maximum-paths 2 under BGP then it would allow BGP to use both of the advertised default routes. In the beginning of this discussion I thought that this command might achieve your objective of using both ISP. But as you have further explained your objectives it is not just to use both ISP but is that traffic from certain subnets should use ISP 1 as primary while other subnets should use ISP 2 as primary. maximum-paths would not help with that.

As I said previously to achieve your objectives you need to consider inbound traffic and outbound traffic separately and there would be separate solutions for each objective.

For inbound traffic I explained in a previous reply how to achieve in BGP that traffic coming from the Internet for certain subnets should use ISP 1 as primary while other traffic from the Internet to other subnets would use ISP 2 as primary. If there are parts of that explanation then please ask about them.

For outbound traffic there is not any solution using BGP. BGP routing uses destination addresses to make forwarding decisions. Your objective is to use source address to make forwarding decisions. And that is what PBR does. I am surprised that configuring local policy did not work. Would you try ip local policy instead of ip local-policy and let us know if that works?

Re: Setting Gateway for Loopbacks

CCIE Aspirant — Tue, 17 Aug 2021 21:27:30 GMT

HI @Richard Burts ,

So i will try tomorrow as per your suggestion, to advertise first two most specific subnets plus complete /22 subnet to isp1 and other two specific subnet plus /22 summary route to isp2. it should solve my inbound traffic issue.

For outbound if i try maximum path will it solve both things or not? Kindly advice

And for PBR i replied above that " ip local policy route-map " command is not working at my router. i have shared the version as well kindly check.

===============================================================
Cisco IOS XE Software, Version 03.13.01.S - Extended Support Release
Cisco IOS Software, ASR1000 Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 15.4(3)S1, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.

Re: Setting Gateway for Loopbacks

Richard Burts — Tue, 17 Aug 2021 22:11:01 GMT

If you make the changes that I suggest about advertising into BGP it should achieve your objective for inbound traffic from Internet to you. It should have 2 subnets come to you over ISP 1 while the other 2 subnets come to you over ISP 2. And if either ISP is not working then all traffic should use the ISP that is still working.

For outbound if you try maximum paths it will not achieve your objective. maximum paths will allow BGP to use both default routes. But nothing there will tell the first 2 subnets to use ISP 1 or tell the other 2 subnets to use ISP 2. To achieve that objective you need PBR.

This is what you posted in an earlier response

"ip local-policy route-map"

(config-if)#ip local-policy route-map ISP_FES
^
% Invalid input detected at '^' marker.

Clearly there is a dash between local and policy. local-policy is not a valid command. local policy should be a valid command.