Pre-configure/test eBGP with Dual ISP

[johnlloyd_13]

hi,

just a sanity check here. we currently have a dual ISP link/router to a single ISP and the setup is via HSRP (with static default route to ISP hop). plan is to migrate to BGP with IPv4/v6 dual stack.

we would like to do some POC and pre-configure the new /30 IPv4/v6 WAN IP and eBGP (v4/v6 AF) on the secondary link/R2 while still have internet via the primary link/R1.

if we configure BGP on R2, would this have any effect on the live internet traffic on R1?

also, if we say the ISP is sending a BGP default route, do i still need to configure a static default route to ISP hop or there's no need for that?

[Richard Burts]

This is an interesting question. There are some things that we do not know about your environment that might impact our answer. You mention that you have 2 routers. You tell us that it is a single ISP but it is not clear whether both routers connect to the same ISP router or whether there are 2 ISP routers. And you do not tell us whether the new /30 connection will repurpose existing interfaces or will be on different router interfaces. And you do not tell us how the devices in your network decide whether to send their outbound traffic to R1 or to R2.

From a high level perspective I would want to say that as long as your production traffic is using static routes to get to the ISP and the ISP is using static routes to forward traffic to your network, that you should be able to set up a test environment for BGP without impacting your production traffic. But here are some challenges that I see when we look into the details:

- Using HSRP seems to indicate that your 2 routers are connected to ISP through a switch. How do you introduce the connection for testing into this? If the test will use a new physical connection and new interfaces it is easy. But if the intent is to use the existing physical connection how do you get a second subnet? I thought briefly about using secondary addressing to set a second subnet onto the vlan on the switch. But the problem is that the BGP packets would be sourced from the primary address and not the secondary address.

- If you are going to use your second router to test you probably will need to disable HSRP on it. So there is some challenge to the stability of the network because you have removed the automatic failover if there is some network problem on the primary router.

- And if you are using your second router to test you probably will need some way to make sure that your production traffic is using only the primary router to access the ISP.

- Assuming you find a solution for establishing a second connection to the ISP, it should be easy to set up BGP and to test establishing neighbor relationship. You can test advertising some subnet to them (preferably one that is not active on your network). And they can test advertising a default route to you - it should not have any impact on production traffic because the AD of the BGP default route is less preferable than the static default route that your router is using. But if the ISP advertises any more specific routes then it could impact your network since their more specific route would be preferred over the static default route.

HTH

Rick

[johnlloyd_13]

hi rick!

happy new year! thanks for your time and having interest in my inquiry.

i put some brief diagrams for clarity.

we have 2x ISP routers (colo in our rack) and our R1 and R2 WAN is connected via a switch L2 VLAN (VLAN 10). see brief diagram (HSRP-CURRENT). the plan is to move out of the L2 VLAN (VLAN 10 for HSRP) and directly connect our R1 and R2 to ISP1 and ISP2 respectively using the same WAN interface and but will assign a new /30 IPv4 and IPv6 subnets.

the 78.1.6.0/27 HSRP subnet will be re-purpose for our public LAN (as secondary IP) and we'll still retain HSRP for our LAN. refer to diagram BGP-POC.

we'll use a separate port both on R2 and ISP2 for the BGP POC. for this POC, do i need to temporarily shutdown R2 LAN G0/0/0 (HSRP)? i also had the other question whether if i need to configure static default route with ISP sending us a BGP default route. do you see other "challenge" in the said POC or is this just a straightforward BGP test on a separate secondary WAN link/R2.

[Richard Burts]

Happy New Year. Thank you for the drawings and the information. They do help clarify some things. You were clear in the original description that your edge routers run HSRP so that the ISP forwards traffic to the virtual address shared by your routers. The drawing seems to suggest that currently the ISP routers also run HSRP and that you are forwarding traffic to the virtual address shared by the ISP routers. Is that correct? And am I correct in assuming that your company lan also runs HSRP to direct traffic to the appropriate edge router?

Seeing that you will provision a new connection for the POC testing makes the testing easier and reduces any potential impact from testing since the existing HSRP and static default routes will continue to work and will have more preferable AD than the BGP default route. You will need to be careful in testing if the ISP router testing BGP will advertise any routes more specific than the default route. 

Perhaps we need to understand better what are your intentions in the POC test. It is pretty clear that the first part of the test will be setting up the new connection (IP addressing, testing connectivity, etc), configuring BGP on both routers, verifying that they successfully establish a neighbor relationship. It will be easy for them to advertise a default route and for you to verify that your test router has learned that BGP default route. The test router could continue to forward production traffic since the static default route that it has is more attractive than the BGP advertised default route. I am not sure how much beyond that you want to go in this test. Do you want them to advertise some destinations in the Internet and have some of your traffic use the test connection? Do you want to advertise some subnets of your network to the ISP so that they will forward some traffic to you over the test connection? These would take a little more effort and introduce the possibility that the BGP could have some impact on your production network, but would make it a more robust test for POC.

HTH

Rick

[johnlloyd_13]

hi rick,

yes, we're running HSRP on both ISP/WAN and to our internal LAN. there's bi-directional static default route to reach the ISP next hop and to reach our HSRP VIP (for the WAN).

BGP advertisement (for IPv4) seems to be tricky and don't want to impact our production network. i would probably ask the ISP for a test IPv4 network (for BGP) and create a loopback for it on R2. IPv6 BGP advertisement won't be an issue since we don't have IPv6 yet in our network environment. would this POC test plan work?

[paul driver]

Hello

I hope you guys don't mind me joining this interesting discussion?

You do have various options to accomplish this but an elegant way would be to incorporate a dynamic igp routeing between your bgp rtrs and the nexus core, the following is pretty high level but its just an brief explanation on a possible topology setup regarding your query.

Your bgp rtrs will each have a ibgp peering between each other and ebgp peeing to their directly connect ISP
These rtrs will also have an IGP (OSPF) peering with the nexus cores which in turn will have a ospf  peering to each bgp rtr

Both bgp rtrs will each receive a default route from their isp rtr which you could then redistribute into the ospf domain with a ospf type 1 route for primary and type 2 for the secondary towards the nexus cores.

This would provide the resiliency for the advertised default routes in case the primary isp fails, As such that the primary bgp rtr will  lose it received default route from its ISP and in turn remove its own advertised default route to the nexus cores so then the secondary default route being advertised would then take preference

You could append bgp local preference between the bgp rtrs so the primary bgp rtr is chosen for egress traffic to it respective ISP.

For ISP ingress traffic you would advertised all local nexus routes to each ISP and then incorporate AS-Path prepending to the less preferred ISP(s) for all or specific local routes

Lastly you also need to make sure you do not become a transit path between either isp, which you could do by simply appending an as-path filer for local routes only to be advertised towards either ISP.

[johnlloyd_13]

hi paul,

feel free to jump in. the more input the merrier :)

i don't think configuring an IGP (i.e. OSPF) would be necessary for the POC in this case. i just need to verify basic IPv4/v6 connectivity and BGP advertisement test on the secondary link with ISP.

i do plan to configure IGP (OSPF) for iBGP between my R1 and R2 on the final change window.

[Richard Burts]

I agree that having multiple participants in the discussion and multiple points of view is a good thing and welcome Paul to the discussion.

I believe that the POC as you have clarified it should work fine and should allow you to do some limited testing of BGP without any impact to your production network. I had in mind that once we had a plan for the POC that we should have some discussion of alternatives for when you want to convert your network to use BGP. Paul beat me to it and has suggested one approach which would continue to use the model of having a primary path which carries all of the production traffic and a backup path which carries production traffic only when there are problems with the primary path. There are some other models of implementation including the possibility of doing some load sharing and using both paths at the same time. Have you thought about this and do you have any preference for which model to use?

One of the questions to consider as you prepare to implement BGP is what the ISP will advertise to you. For the POC to have no impact on production it would be best if the ISP advertises only a default route. If you want to do some very limited testing of traffic back and forth you could either use Policy Based Routing to send traffic from your test address to the BGP peer or the ISP could advertise a network for testing that would not be used by any production traffic. As you get ready to implement BGP for the production network you should think about what the ISP will advertise. If you will continue the model of a primary carrying all of the traffic and a backup then all you need is for the ISP to advertise a default. If you want the model that facilitates some load sharing then the ISP probably should advertise some limited routes in addition to the default route. You could apply local preference on the second router for some of those routes and then traffic for them would use the second router while the first router continues to carry most of the traffic. And in the case of a problem with one of the paths then all traffic would use the surviving path.

HTH

Rick