BGP - configuring route of last resort for local Internet breakout

[BHconsultants88]

Hi everyone.

I have a BGP query that I've inherited which I hope someone can help me with.

My company has a WAN MPLS circuit across Europe (provided by Verizon). There are 6 remote offices with the UK Data Centre in Hampshire, UK. We want to regionalise Internet access on a per country basis. The main driver for this change is to improve performance and language specific websites.

Here's the summary:

• in UK DC, Internet breaks out via Cisco Nexus (which handles all the BGP routes/advertisements).
• In Europe, Internet breaks out via local Palo Alto firewall and Prisma service
• All of these offices are inter-connected via this Verizon WAN. Each office has Verizon WAN router

I would like all the RFC subnets to remain routing via Verizon, but Internet traffic to break out locally. In order to break out locally, could someone help me with the routing changes I'd need to make? This is what I'm thinking:

• Advertise the following RFC subnets on the Cisco Nexus in UK DC:
• 10.0.0.0/8
• 172.16.0.0 /12
• 192.168.0.0 /12
• Advertise these subnets, tagging the Verizon AS number
• Routes would be added on each local Verizon WAN router
  • something like: 10.0.0.0 /8 via 10.50.0.100 (MPLS interface of local router)
• I would need to advertise the route of last resort for Internet traffic. How would this be done?

Currently on the Nexus, I have advertisements such as below configured:

• 10.50.137.0/24, ubest/mbest: 1/0
      *via 18.0.100.122, [20/0], 3w4d, bgp-85445, external, tag 85443
• 10.50.137.0 /24 = France office
• 18.0.100.122 = Verizon MPLS interface
• 85443 - Verizon AS number

To advertise the RFC subnets I've mentioned above, is the BGP configuration I need to apply on the UK DC Nexus the same as above - just replacing the initial /24 subnet with the required RFC ones?

For the route of last resort, does the following make sense?

0.0.0.0/0, ubest/mbest: 1/0   *via [appropriate interface on ISR], [20/0], 3w4d, bgp-85445, external, tag 85443

Once these advertisements are added, I am thinking Verizon will need to add routes on their devices. Sorry for the long winded nature of this question. I hope the above makes sense and I would be very grateful for an expert opinion!

Many thanks

[Jon Marshall]

I am not really following this at all. 

How does everything work at the moment ie. does all internet traffic go via the UK and how is this handled, is there are default route advertised across MPLS to all the remote offices ? 

If so that is what you need to be concentrating on but your post talks about advertising RFC subnets from the UK and it's not clear what that has to do with internet access. 

Also when you say Colt do you mean Verizon ? 

Jon

[BHconsultants88]

Hello Jon, thanks for taking the time to reply and sorry for the confusion.

Let's say all Internet traffic currently comes back to UKDC via the Verizon WAN and breaks out in UK. A default route exists on each core router in the European offices sending this traffic to UK.

I want to split this traffic so only WAN traffic (RFC subnets) comes back to UK. The rest (Internet) breaks out locally.

I'll try and get a high level diagram together and post to this discussion. I hope the above clarifies things a bit more.

[Jon Marshall]

It does help thanks. 

So the default route in the remote offices, is that learned via MPLS from the main UK site ? 

Do you want the remote offices to break out locally but if their local internet goes down they use the UK ? 

What do the remote sites look like in terms of routing etc. ie. is the default gateway for the clients the MPLS router, the local firewall or something else. 

If you don't want the UK as a backup then you simply need to change the default route at each remote site to be the local firewall but if you do want a backup then you would need something else as well, maybe IP SLA for example. 

If you could answer the above we can go from there. 

Jon

[BHconsultants88]

That's brilliant, thanks so much. That makes perfect sense. I've attached an overview diagram of what currently happens. 

I'll answer your questions in the order you raised them:

So the default route in the remote offices, is that learned via MPLS from the main UK site ? 

Yes, all of these routes are learned from the UK DC

Do you want the remote offices to break out locally but if their local internet goes down they use the UK ? 

Yes. So for instance routing preference for Munich is Munich first, then UKDC if Munich goes down.

What do the remote sites look like in terms of routing etc. ie. is the default gateway for the clients the MPLS router, the local firewall or something else.

Remote sites have a core switch acting as the gateway. Here's the static route configuration from the WAN router in Paris

France WAN Verizon router - Static Routes
set routing-instances LAN1-VRF routing-options static route 10.50.37.0/24 10.0.50.1 vni-0/4.0 preference 1

Paris LAN = 10.50.37.0 /24

Paris core switch = 10.0.50.1

The core switch has a default route of 0.0.0.0 /0 via 10.0.50.254 (Verizon router)

I was under the impression routes for the RFC subnets would need to be injected from UKDC. Do you think this isn't necessary?

[Jon Marshall]

I get an error when trying to open the attachement. 

Before we cover the default route the RFC subnets, aren't the local subnets already being advertised from each site into the MPLS network anyway, so why do you need to inject them at the UK site ? 

I think there may be something I am not following about those subnets, inject them where exactly. 

As far as the default route goes is the Verizon router at each site advertising it to the core switch and if so what routing protocol is being used ? 

I ask because the Paris router example seems to suggest you don't advertise your local subnets to the Verizon MPLS router dynamically, they have to add static routes and they then advertise the local subnets into the MPLS network ? 

Do you run any routing protocol between the core switch and the local firewall ? 

Finally what model(s) are the core switches in these sites and what feature sets ? 

Apologies for all the questions but can't really suggest solution without understanding the full setup although think it is safe to say you may well need to use IP SLA for this which is why I asked about the switches and feature sets. 

Jon

[Richard Burts]

If I am understanding the discussion correctly each of the offices will be going directly to the Internet for non-corporate traffic and to London for corporate (RFC subnets) traffic. So will each of the offices have its own connection to an ISP? My thought would be that if each ISP advertised a default route to its connected office, then each office wold have a default route from its local ISP and another default route from London, if you manipulate the metrics of the default route from London so that the local ISP was preferred then you would pretty much have what you wanted. Each office would use its local ISP default route for Internet traffic, and if the local ISP stops advertising its default route then the office would use the London default route.

One thing to keep in mind is that if each offic will be going directly to its ISP then something needs to be doing address translation for the traffic going to the local ISP. Will you negotiate with the ISP to do the address translation? If I am understanding the description correctly then the offices have switches for connectivity to London (and to the local ISP?). But very few switches support address translation.

[Jon Marshall]

If both default routes are being advertised to the core switch in the remote site then yes easiest thing by far is to manipulate the metrics but it is by no means a given that they are being advertised which is what I am attempting to find out. 

I also don't see the issue with NAT unless the remote sites are hosting services which there is no mention of at the moment. 

Each remote site has a firewall so the NAT can be done there, not sure why you are talking about NAT on switches. 

Jon

[Richard Burts]

Jon

I was reacting to this response:

What do the remote sites look like in terms of routing etc. ie. is the default gateway for the clients the MPLS router, the local firewall or something else.

Remote sites have a core switch acting as the gateway

I should have read the original post more carefully and noted the reference to firewalls. I was working from the assumption that if the switch were the gateway that the connection to new ISP would use the switch. Thanks for catching the fault in my logic.

[BHconsultants88]

Thanks everyone for your assistance so far. I'll go through all of the comments now and try and piece it together a bit more.

I've attached the WAN topology in Word document this time, hopefully you can open this file. On page 2 is a very crude drawing showing the current traffic flow from UK DC and Paris office. Also attached are the routes from the Nexus in UKDC and the WAN router in Paris

Thanks again.

[Georg Pauwen]

Hello,

do you have a topology drawing ? 

[BHconsultants88]

Hi Georg, thank you for your reply. Topology and routing tables attached. Thanks for your time.

[Seroj Hacopian]

Post your topology and the config of one of the sites
If possible.

[MHM Cisco World]

all site connect via BGP to MPLS SP ?

if it send packet with source and destination, how MPLS SP will handle it ?

as I understand you must extend the VRF use in MPLS SP PE to your Site CE.