Complex multihomed BGP and policy based routing and NAT

[bdedek]

Hi,

I have a fairly complex request.  I would like to setup a high availability Internet connection and control my routing paths without accepting the whole Internet routing table.

Here is a scenario.  I have two ISPs A and B.  Each provides me with a class C address space.  I would like to setup my ASA firewall and Internet edge routers to be able to use both providers, where ISP A is used for critical business with static NATs, and at the same time use ISP B for general usage and surfing.  The configuration needs to be adaptable so that if either ISP goes down, the other takes all the traffic.  So, the ASA has both static NATs with ISP A address space and dynamic NATs with ISP Bs address space.  The two Internet routers are both running eBGP to their respective ISP and iBGP between them.  Both receive their ISP routes and their neighbors, and a default gateway from ISP B.  They are also using GLBP to form a single virtual IP for the default route from the ASA firewall, also an ISP A address.  I have attempted some policy based routing with tracking routes to determine availability.  It seems to be working but I am having some issues with latency and asymmetrical routing.

I have a configuration but I'm not sure if its the most efficient or if there is a better way.  I have my outside interface on the ASA assigned with an address from ISP A.  I guess the ASA is smart enough to do proxy ARP for any NATs that are used from ISP Bs addressing, as I only have the one outside interface configured.  I have setup both ISP routers Ethernet interfaces with primary addresses from ISP A and assigned secondary IPs from ISP B.  I am attempting to do some policy based routing base on source address so I can specify the egress path, for example ISP B NATs go out ISP B and static ISP A NATs go out ISP A.  I also have one or two devices with ISP B addresses, outside the firewall on the segment formed by the Ethernet connections between the two routers and the firewall.  The problem is that since the firewall only has an IP from ISP A, when packets are destined for these outside boxes from behind the firewall, the packets go to the routers ( the firewall default gateway) and the routers forward them off to one of the ISPs based on PBR, rather than ARPing for the machine on the local segment.

I wanted to through this out there so maybe someone will say, oh there is a much better way to do this.  This seems really confusing, but maybe someone has seen this done before and has some recommendations.  Thanks for any help.

[Jon Marshall · Accepted solution]

The problem is that since the firewall only has an IP from ISP A, when packets are destined for these outside boxes from behind the firewall, the packets go to the routers ( the firewall default gateway) and the routers forward them off to one of the ISPs based on PBR, rather than ARPing for the machine on the local segment.

Why not simply use deny's for these destinations in your route-map acl so that it is not policy routed ?

There are however other issues with your setup i would like to expand on.

1) GLBP - gives you nothing because GLBP load-balances based on the different source mac-addresses and the ASA is the only source mac-address if the ASA is in routed mode. Mind you it doesn't hurt.

2) More importantly you want to use one ISP for all traffic  if the other ISP fails. With outbound internet connectivity this is straightforward but for your static NAT statements for internal servers how will you notify clients on the internet of the new address ie. how will the DNS records be updated from ISP A addresses to ISP B addresses.

It is unlikely that each ISP will advertise the other ones class C address space so you are going to have issues. This is why in this scenario provider independant addressing is so useful ie. both ISP's advertise the same address range so you do not need to update your DNS entries for the static NATs.

Another alternative is to look at something along the lines of a GSS (Global Site Selector) which can dynamically update DNS based on availabilty of the servers but this is not trivial to design/implement.

3) Not sure why you are using secondary addressing on the ISP router interfaces.

As you say, this is not a trivial thing to do. As i say if your addressing was provider independant this sort of scenario becomes a lot easier to implement. Have a look at this doc which may give you some ideas as well -

Enterprise multhoming

Jon

[bdedek]

Thanks for the reply.

I actually did some testing with the deny's and that worked well, in my testing.  Thanks for the information.

The GLBP could really be replaced with HSRP, as you are right, since the only device using the gateway is the firewall.

I am fortunate enough that both my ISP's are allowing/advertising the others class C, so eBGP is taking care of all the issues of routing when one ISP goes down.  The only device doing NAT is the firewall, so I'm not sure what you were talking about as far as the DNS issues.

I have secondary addresses on the Internet routers, because that's the only way I could think of that it would work.  I have to have the LAN Ethernet interfaces on a common subnet for iBGP and also for the GLBP.  But I still needed the address from the secondary ISP so that when packets arrived from the WAN they would know that they had reached their destination and the router would ARP for the destination machine.  Is my thinking flawed here, or did I completely miss the boat.

Thanks.

[Jon Marshall]

I am fortunate enough that both my ISP's are allowing/advertising the others class C, so eBGP is taking care of all the issues of routing when one ISP goes down.  The only device doing NAT is the firewall, so I'm not sure what you were talking about as far as the DNS issues.

Yes, that is fortunate with the advertising. What i meant about the NAT is that clients on the internet will connect to a DNS name eg. www.yourcompany.com which will resolve to an address from ISP A for example. Now if ISP A goes down and you use ISP B to represent www.yourcompany.com the DNS on the internet will still resolve that URL to ISP A address. However if ISP B is advertising ISP A addresses then there is no need to worry, just keep using ISP A's addresses for your static NATs.

I have secondary addresses on the Internet routers, because that's the only way I could think of that it would work.  I have to have the LAN Ethernet interfaces on a common subnet for iBGP and also for the GLBP.  But I still needed the address from the secondary ISP so that when packets arrived from the WAN they would know that they had reached their destination and the router would ARP for the destination machine.  Is my thinking flawed here, or did I completely miss the boat.

If you have devices in the LAN between the firewall and the routers that use ISP B addressing  then yes you may need secondary addressing but that is the only reason i can think you would need it. IBGP will peer with ISP A addresses and the firewall would answer requests from the router for ISP B addressing if you had static NAT statments using ISP B addressing.

If the devices between the ASA and routers used ISP A addressing and the ASA outside interface was using ISP A addressing then i can't see the need for secondary ISP B addressing on the LAN interfaces of the routers.

Is this what you mean ?

Jon

[Nilesh Joshi]

Hi,

Is IPSec  Stateful failover works on your bgp configuration , bcoz on my site i configured IPSEC stateful failover but it not works

http://www.gotolightspeed.com/en/US/prod/collateral/iosswrel/ps6537/ps6586/ps6635/white_paper_c11_472859.html

I done configuration as per given url also we r using bgp public as  no with full routing table but still ipsec stateful failover not works with TWO ISP bgp link

regards

Niljos

[ancarr]

I am looking to do something very similar in our network.  Please see attached visio for an example of something that might work in your case as well..  I am also wondering if this would be a viable solution for multihoming bgp with in-continguous address spaces from multiple providers.

[bdedek]

That is an interesting approach.  I have used virtual interfaces for the DMZ's but I didn't think to use it on the outside interface.  As you probably see from my first post, I am just addressing the ASA's with the primary ISP numbering, and using NAT's from the backup ISP.  This seems to work since the ASA will proxy ARP for the second ISP's addresses, but I didn't know if there was a more 'best practice' approach.  I would be interested in seeing if your design has any advantages over the solution I chose.

How does the ISP A router form a connection with ISP B router for iBGP, do you have the ASA allowing traffic between the same security levels, and its routing traffic from VLAN 43 to 54?

What does your routing table look like on the ASA?  Do you have multiple default routes?

Thanks for the input and the new design.

Billy

[ancarr]

Billy,

As I noted in the previous message, this is what I was thinking of putting into service for us.  But, now I am having second thoughts.  Maybe running eigrp or ospf on the outside would be a better idea.  I am also working on trying to eliminate the two disparate address spaces and get our own.  The ISPs require at least a /24 in order for them to advertise it.  I don't have the config yet, but I am thinking that if we go this route, we could redistribute the BGP routes from the routers into (ospf or eigrp) but filtered, using route maps.  We are planning on getting minimal routes from the ISPs, not the full routing tables.  That would probably put the onus on the ASA to make the routing decision on the best route for outbound traffic.  Any ideas...

We currently have two ISPs not using BGP.  We have two different address spaces and as a result we have two ip addresses for each service that we offer externally, for ISP failure purposes.

Thanks,

[bdedek]

Your situation sounds very similar to ours.  We have two ISPs and each provided us a /24 network, the minimum to advertise via BGP.  Both providers agreed to advertise both networks.  So should a failure occur, the other will still advertise a path inbound.  I have two routers, one for each provider but you can do the same thing with only one.  I accept the ISP connected or customers routes via a BGP filter and have one provider provide me with a default gateway, as well.  You can have both provide one and set preferences, but I just do it with a floating static if the provided one disappears.  I don't get my ASA involved with these routes, I just have a default gateway set on the ASA to point to my outside routers going to my ISPs.  Since I have two, I am running GLBP, but could do HSRP in case one becomes unavailable.  This provides me one IP to use as my gateway from the ASA.

As for selecting the best path, I have preferred to keep a 'primary' network and 'secondary'.  The primary from the most reliable ISP is the address space I put on the ASA NATs for all the critical inbound services, FTP, WEB, VPN, etc.  The second is used mostly for surfing and outbound traffic, dynamic NATs, and is the ISP that provides me the default gateway out.  I have some policy routing setup so that if the traffic is return traffic sourced from my static NATs they go back out the primary link, to try to keep the critical traffic on the primary link.  I recently made some changes with the policies to ensure that my links were up , verify availability, before policy routing.  It seems to be working pretty well.  As far as inbound link preference, you can pad your BGP advertisements for the respective providers so that each ISPs address space is preferred on their network, but it still fails over if one of the ISPs goes down.

If your thinking of doing BGP, you will need to register with ARIN and get yourself an AS number.  I think it was $500 the first time and $100 annually.  I have always heard BGP can be tricky and hard to configure, but its really not, in this scenario, and once you get it going, you set it and forget it.

Good luck.

Billy

[ancarr]

This sounds pretty close to what we want to accomplish here.  Would you be willing to email me the configs of your

routers and ASA, with the appropriate information removed of course?  I have been looking for example configurations with two address spaces and how it will work on the ASA.

Thanks,