ISR4331 Redundant ISP question

[Matt26]

Hi All,

Hope all is well, Have a question in regards to setting up DIA connections from multiple ISPs. Currently we have a BGP peer to ISP-A using a /27 owned by them and we use their public IPs inside our network on 2x ISR 4331 routers with 2 diverse paths, now my problem is if we bring in ISP-B with a different /27 public IP range how do we setup redundancy and potentially use ISP-B as the main connection as it's better speeds.

From what i gathered there's BGP multihoming but, it seems like for us to do that we need to own our own block of public IPs as well as register a public ASN, is this correct? Or what from what I gathered is we need to get the ISPs agree to let us advertise the IP from our routers? 

If we can't do BGP multihoming what are some other ways we can setup this redundancy? As mentioned we have 2x ISR 4331 routers sitting at the edge of the network and behind that is a Firepower FW (which is also doing the NAT).

Thank you all, this is new to me so sorry if my terminology is incorrect.

Thank you!

Matt

[Richard Burts]

Matt

There are things that we do not know about your network and that makes it difficult to provide good answers to your questions. One important thing to understand is the traffic flows for Internet traffic. I assume that most of your Internet traffic is initiated from devices inside your network accessing resources in the Internet. Are there any devices or services inside your network where traffic to them is initiated from the Internet?

It would also help if you could explain more about how you are using the /27 block of addresses from the first ISP. One obvious thing would be to provide addresses for dynamic network address translation for traffic to the Internet initiated from inside. Are there any static nat configured?

You mention " ISPs agree to let us advertise the IP from our routers? ". I am not real clear what you intend with this, but if you are using /27 address blocks you would not be able to advertise ISP A block to ISP B because ISP will generally not accept advertisement of a block smaller than /24. 

[Matt26]

Thanks Rick,

totally understand I guess I was looking for a thousand foot view on multihoming to see if it's even possible. 

For your questions yes it mostly is devices in our network going out. For outside traffic in, i guess a good example would be we have a Citrix netscaler in our DMZ that has a private IP with NAT to the public IP from ISP-A. Also there are site-to-site VPNs with some vendors that terminates at our firewalls outside interface.

For Static NAT, yes we do have static NAT, from a quick glance i see our NetScaler is static and I also see one of our Fax services is static NAT as well, those are just 2 examples.

I see that make sense that we would need a /24. if that is the case, i am not sure this would be possible.

Hope this helps, never had to do anything like this before so this is all new to me, appreciate the response. 

[Richard Burts]

Matt

From a high level yes multi homing is possible. I think it is helpful to consider separately what that means for traffic originated in your network going to the Internet and traffic originated in the Internet going to your network.

For traffic originated in your network it is helpful to run BGP with both providers. In the more simple case each ISP would advertise a default route to you. This allows you to route traffic to both providers and provides a mechanism to verify that both ISP are available. You can then decide whether you want to send traffic to both ISP (load sharing) or to use one ISP as primary and the other ISP as backup. If one ISP stops advertising its default route then you know that all Internet traffic should be sent to the other ISP. All outbound traffic from your network would be NATed at the router so it does not matter what IP addressing you are using inside your network. A somewhat more complicated approach is for one or both ISP to advertise partial updates (typically routes from the ISP network and its connected customer - or within a couple of AS from the ISP) so that you can use the more efficient route for those destinations. And some networks have the ISP advertise the complete Internet routing table so that they can optimize routing for EVERY Internet destination. I do not believe that this would be what you want to do.

For traffic originated from the Internet to your network it is more complicated. I have worked with customers who have their own Public IP address range (at least /24). They just advertise their address range to both ISP. And the Internet has access to their services which need to be accessible from the Internet. And it provides failover if one ISP becomes not functional. But that is not your case. I have worked with customer who have provider address blocks (at least /24) from one provider. They negotiate with the ISPs to be able to advertise that address block to the secondary provider. But that is not your case. Your case is more challenging. For a server/service that needs to be accessible from the Internet perhaps it is possible to configure static NAT for the server/service on both routers connecting to ISP and some process that detects failure of the primary ISP and does a DNS update to change to the address used for the second ISP? Or perhaps it would be possible to more that server/service to some third party to host it and to be able to forward to you using both ISP? Site to site vpn presents a real challenge. If the current vpns terminate on the firewall outside interface, am I correct in assuming that the firewall outside interface has an IP address from the block provided by your current ISP?

[Matt26]

Wow thank you for the very detailed reply Rick.

---

All outbound traffic from your network would be NATed at the router so it does not matter what IP addressing you are using inside your network. 

---

So does this mean I have to move the NATs off the firewall and move them up to the routers at the edge? was hoping to do the dual multi-homed so router-1 connects to ISP-A and router-2 connects to ISP-B. So does that mean each router would have a NAT to their respected ISP IPs?

---

am I correct in assuming that the firewall outside interface has an IP address from the block provided by your current ISP?

---

Yes you are correct, so this outside interface has a public IP from our current ISP.

[Richard Burts]

Matt

In reading through the discussion I realize that there are multiple things about your network that I do not know and that I made a couple of assumptions that relate to my statement about address translation being on the router. So let me back up and try a slightly different approach to my suggestion.

You told us that you have 2 routers providing access to the Internet using ISP A and that you will be bringing in a second ISP. I assumed that you would reconfigure one router so that each router connected to a single ISP. But maybe that is not the case. When you bring in ISP B will that connection be to one router or (like apparently ISP A does) will it connect to both routers?

I assumed that the decision about which ISP to forward a particular IP packet to would be made by the routers. But perhaps that decision could be made by the firewall. Which brings up the question of how you will make the forwarding decision. I had mentioned possibilities of actively using both ISP in a load sharing approach or the possibility of having a primary and a backup relationship for the ISPs. In reading the discussion again I think you are saying that you would prefer a relationship where ISP B is primary and ISP A is backup. Would the firewall be able to build that into its forwarding decision? What is the routing logic between the firewall and the routers? Does the firewall participate in BGP? Or does it use a dynamic interior routing protocol with the routers? Or does it use static routes? Or do the routers use HSRP and the firewall just as a default route with the virtual address as the next hop?

Perhaps I should have emphasized that you will need 2 different sets of NAT configuration. You do not want to forward something to ISP A with an ISP B address or forward something to ISP B with an ISP A address. It is less important whether that is done on the firewall or on the router. To me it seemed more simple on the router, but doing it on the firewall might be possible.

[Matt26]

No problem Rick, 

You told us that you have 2 routers providing access to the Internet using ISP A and that you will be bringing in a second ISP. I assumed that you would reconfigure one router so that each router connected to a single ISP. But maybe that is not the case. When you bring in ISP B will that connection be to one router or (like apparently ISP A does) will it connect to both routers?

---

Correct so currently we have 2 routers going to ISP-A on 2 diverse circuits but we're bringing in ISP-B due to cost and speeds, and the goal is to get rid of one of the ISP-A circuits, so one router to A and the other router reconfigured for B.

Or do the routers use HSRP and the firewall just as a default route with the virtual address as the next hop?

---

Exactly! the FW has a default route to a virtual address which is actually an IP from the ISPs public block, the inside interfaces of the routers are also using the public IPs, (not sure if that's best practice this is just what I inherited). 

Thank you again for your help, I came into this environment and it seems a little more complex for my skill level, but I guess that's the best way to learn it.

Thank you,

Matt

[Richard Burts]

Matt

Thanks for the clarification. To make sure that I have it right: you will have router1 connecting to ISP A and router2 connecting to ISP B. I assume that you will run EBGP between the router and its respective ISP and run IBGP between the routers. You prefer to operate in an active/standby relationship with traffic preferring ISP B and using ISP A when ISP B is not available.

There are several ways that you might accomplish that. There is one solution that is more simple and more reliable and is the one that I will suggest. For the BGP between your router and its ISP have the ISP advertise a default route to you. On router2 assign a Local Preference that makes its routes more favorable. The IBGP between your routers will help the routers know whether traffic should go to ISP B (preferred) or go to ISP A (backup). There might be a question of whether you should advertise anything to the ISPs. Since each ISP knows the address block that it assigned to you they should be able to route that traffic to you without your advertising anything to them. So I do not see a need for you to advertise anything in BGP.

We need to consider questions about nat, how and where it is done. Since you will be using an address block from ISP A and an address block from ISP B you will need separate nat configuration to handle each address block. From my perspective since the decision about whether to send traffic through ISP A or ISP B will be made on the routers it would be simple to have the nat configured on the router. If you prefer to have the address translation done on the firewall it might be possible to do that. For this to work the firewall will need to know whether traffic is to be sent via router1 or via router2. So you will need to have a dynamic routing protocol between the routers and the firewall so that the firewall can know which router has the active BGP route. 

The preceding discussion addresses traffic originated in your network going to the Internet and its return traffic. The other aspect that needs to be addressed is traffic originated from the Internet and going to servers in your network. The key to getting this to work is about how the Internet would know to use an IP address that goes through ISP A or an IP address that goes through ISPB. So this is more about DNS (and how to trigger a change in DNS) that it is about routing.

You mention some site to site VPN are part of your traffic and that they terminate on the firewall. Your solution is probably for each peer to have 2 vpn tunnels configured (one with an address in the ISP A block and one with an address in the ISP B block. That would be fairly straight forward if the vpn terminated on the router. I am not sure how you could do that if the vpn terminates on the firewall.

[Matt26]

Thanks for the info Rick this is great,

BGP aside, are there any other methods of setting up dual ISP connections using 2 routers? I have a feeling i can't just take the 2nd router put the new IP on it and call it a day, will probably cause some asynchronous routing issues? 

Thank you,

Matt

[Richard Burts]

Matt

I read the entire discussion again and would like to start by addressing something that you said in your original post:"From what i gathered there's BGP multihoming but, it seems like for us to do that we need to own our own block of public IPs as well as register a public ASN, is this correct? " This is not correct. You can do BGP multi homing without needing your own block of public IP and you do not need to register a public ASN. Those things may be desirable for some organizations. But neither is required. It is quite possible to use BGP using a non public ASN and without your own public IP address block.

In your recent post you ask "BGP aside, are there any other methods of setting up dual ISP connections using 2 routers?" I would start my response by saying that there are other alternative ways to accomplish dual ISP connection and that we need to consider separately the solution of how to send traffic originated inside your network to Internet and how traffic originated from the Internet would be sent to you. I suggested BGP for traffic originated from your network because it is relatively straightforward and because it enables you to establish a preferred path to the Internet and automates failover to a backup path. But there are alternatives. I will suggest a couple of possibilities:

1) you mention that your firewall forwards Internet traffic to a virtual address. How is that virtual address controlled (which router has the active virtual address)? Perhaps it is possible that each router has a default route configured with its ISP as the next hop and some logic that detects when the preferred ISP is not working and switches the virtual IP to the other router.

2) each router configured a static default route to its connected ISP and configures a floating static default route (static route with AD higher value) with its next hop as the alternate router and implements IP SLA to track the connected ISP. If the connected ISP fails then the normal static default route is withdrawn from the routing table and the backup route takes over.

[Matt26]

Thanks Rick,

That's good to know about the IP addresses and the ASN, I am going to talk to our current ISP to see what we can do to accomplish this as they would probably know best and do this often.

1) Correct, our FWs outside interface has an IP address that is part of that ISPs /27 which then points to a HSRP virtual IP from that same /27, and the HSRP group is between our 2 routers that are pointing to our ISPs internet service on that 2 diverse path.

Now that we're brining in another provider I did think about taking the one of those routers to ISP-A and just give it the new IP they're providing us and just leave the HSRP in tact and change the active router to the new ISP router (if that makes sense). but will this work well? as mentioned we do have site-to-site VPNs which point to the FW outside interface using the IP from ISP-A, if we leave that will traffic still be sent/received from the current internet pipe and not the new one?

Thanks again for your help, I would love to test out these scenarios and get my hands dirty but unfortunately we're a 24/7 environment with no test lab so testing is kind of difficult, i thought about recreating our design in packet tracer, so can give that a try.

Thanks again,

Matt

[Richard Burts]

Matt

There are multiple ways to accomplish having connection to 2 ISP. You need to evaluate alternatives and decide which you think fits your organization the best and that you are comfortable with. 

I would also say that you need to think about how to implement routing for traffic originated in your network and going to the Internet and also about how to implement routing (and DNS etc) for traffic originating from the Internet and going to your network. This gets particularly complicated when you do not have your own address space that you can advertise.

I think that most of us have an almost automatic feeling that connection to 2 ISP is better than 1 because it is redundant. But you need to think carefully about that assumption. I suggest that as you think about redundancy that you think about what you are trying to protect against as you implement redundancy. What kind of failures do you worry about? Probably the most common failures are failure of a transit link or failure of an edge router, or failure of an ISP router. A single ISP with diverse paths and diverse routers protects against these. The main thing that 2 ISP protect against is a failure of an ISP? So how many times has your current ISP just stopped working and a second ISP would have been needed?