Internet Edge Re-design

[Joel]

Hopefully posted in the correct community.

A client of ours is currently exploring re-architecting their Internet edge. Currently they have one ISP delivering two bearer across two data centre a few kilometers apart. The bearers have PA address space mapped to each circuit i.e. two address ranges. Neither the ISP nor client are swapping routes and no BGP. In the case of a bearer failure the particular IP address space is lost. The bearers terminate on firewalls designed to handle two ISP. They do this by something known as multi-link technology on a particular firewall vendor. Outbound traffic is not so bad but inbound traffic is a bit of a nightmare as any service published relies on DNS round-robin!! Between DCs there is layer2 connectivity low latency, high bandwidth and VLANs are trunked for the firewalls to form clusters.

Their plan is to keep two Internet bearers split across DC from the same ISP (dual-home) but look to the ISP to handle the routing. From the high-level diagram we've seen the ISPs will be offering a redundant service where they provide two CPEs to the customer and use HSRP to provide fault tolerance across the routers / north bound circuits. The diagram from the ISP indicates a default route via eBGP is received on the CPEs. No iBGP relationship exists between the CPEs, only a FHRP for the customer to default route towards.
 
One concern (many) I have is the edge doesn't just include one layer of FWs, multiple security services exists and majority in a HA setup, mainly A/S. Apart from the trunking of VLANs across sites (extending broadcast and failure domains) , any part of the perimeter may fail to the other DCs and packet flow will transverses the same layer two link multiple times before heading north bound or even south.   

I believe a more effective way is to keep the DCs separate and routable. Each DC does have a campus attached, and campus traffic can exit the nearest edge rather than transversing the layer 2 interconnect all the time. For inbound services could look at global load-balancers to direct traffic to suitable DC. The ISP is happy to provide many PA address space blocks (within reason), company does not want to utilise two ISP and handle the routing. We have to keep the bearers split across two physical locations and the ISP has to the best of their ability ensured the two circuits run to different POPs, from there anyones guess...

Any suggestions? or shall I proceed with their desire to have multiple security appliances spanned across DCs and default route towards a FHRP address? Only a handful of services are published most traffic is outbound.

[Jon Marshall]

Funnily enough I am just watching an internet design presentaton on Cisco Live which goes into all of this and what your options are.

I am happy to discuss but you may find watching that presentation gives you the best options.

One question though is are the CPEs going to be managed by you or the provider ?

If you are using HSRP and the CPEs are only receiving a default route each then all egress traffic uses just one link ie. the HSRP active.

You could run MHSRP (covered in the presentation) and have each DC point to it's CPE and then you would utilse both links.

You would stilll need to span the vlans between DCs though.

Ingress traffic is another matter.

How are you going to be advertising your internal public addressing or is the ISP doing this for you and where is the NAT taking place ie. on your firewalls or the CPEs ?

Jon

[Joel]

Hi Jon,

Please do share the presentation, or point me at the keywords to search for in CiscoLive?

The CPEs will be managed by the provider. We are discussing MHSRP groups as only one link will be in use and we can direct traffic from the DC/Campus to the local CPE, as you say ingress is a problem.

The public address space is to be advertised by the ISP, the company will have very little control over it. NAT will take place on the firewalls.

Joel

[Jon Marshall]

Joel

https://www.ciscolive.com/online/connect/sessionDetail.ww?SESSION_ID=78655&backBtn=true

just in case it doesn't work search for BRKRST-2044.

Ingress may well be an issue.

Ideally it is the customer who should be using BGP to advertise that address space and then they can influence which link is used for what but if the ISP is doing it could go any way.

How exactly are the firewalls setup ie. does each DC point to it's firewall which has it's own addressing or are they running as a pair ?

If a pair then depending on how the ISP advertises your addressing you could get a lot of ingress traffic going across that interconnect.

Jon

[Joel]

Hi Jon,
Thanks for the link.

The current plan is to have a pair  of FWs A/S across the two DCs.With two firewall layers and with other security appliances setup in A/S configuration, when one appliances failover a lot of DCI traffic will occur on-top of the already stateful/sync traffic.

My opinion is it will be better to treat each DC as a separate firewall pair, i.e have four firewalls, 2 per site. With this setup each site utilises it's own public address range and as you stated rely on MHSRP do make sure in normal circumstances outbound traffic routes to the local CPE. Inbound, still unsure how to conquer, need a sit down with the ISP.
I should mention the DCI are 2*10G active wave lengths. Still less than ideal.
Thanks for your help so far.

Joel

[Jon Marshall]

Joel

I would think having each firewall pair separate to each DC and using MHSRP would keep outbound traffic off the interconnect.

In terms of inbound this is obviously where you doing the BGP can help ie. if you could summarise both DC's ranges then each DC could advertise out their own range and the summary so that traffic always uses the more specific route unless the link is down.

Perhaps the ISP could somehow set that up for you ?

Jon

[Joel]

Thanks for your help

Raised the query with the customer who will be speaking to their ISP.

Regards,

Joel

[Jon Marshall]

Joel

Just to add I prefer your solution if it fits with all the other services.

If you run the firewall pair and other services across the interconnect then you are in effect treating one link purely as a backup ie. it never gets used unless the primary site fails.

And if each DC has a campus behind it that interconnect is going to see a lot of traffic in both directions.

Jon