Connecting two Datacenters together - wavelength vs darkfiber

[romanroma]

I am looking at connecting to data center locations together and reviewing the ISP options of Wavelength vs Dark Fiber. Due to cost it looks like Wavelength is the cheaper of the options; however, what I do not understand is when the ISP says 'dedicate link over shared fiber'.

So they are carving up the fiber with different color optics from my understanding (layman's terms); however, are  IPSec vpn still needed for Wavelength connections? Do you usually have to specify 'ethernet' for the mod of transport so I can use ethernet ports on equipment, or since this is fiber from the ISP is this SONET?

Thank you

[Joseph W. Doherty]

Usually, hand-offs to optical networks just appear to you as an Ethernet interface of some specific bandwidth, often gig.  If fact an optical CWDM or DWDM (info) transceiver, like an SFP, is used at the edge much like you would any other optical transceiver, the main difference is, rather than being SX, LX, etc., it's some specific frequency.

To you, such an optical connection usually appears as a p2p link.  When you need something like an IPSec tunnel would be the same as whether you believe, for security, you need such as you would on any other WAN vendor's p2p links.

PS:

BTW, you might ask the vendor, if optical the path is "broken" between your sites, whether your edge device will "see" this break as a link down (on both ends).

[romanroma]

@Joseph W. Doherty 

Hi Mr. Joseph - thank you for your insight. I was also worried about what you speak of due to 'broken' path or a link down in the path. I am just trying to educate myself on the service/concept before I reach out to the vendors. I am currently trying to reach out to Lumens, Cogent and Verizion.

Do you have any experience with these providers?

[Joseph W. Doherty]

Broken path - ah, consider:

Case 1

R1<Ethernet>R2

Cut case 1's Ethernet cable (copper or fiber), both routers see link drop.

Case 2

R1<Ethernet1>SW1<Ethernet2>R2

Cut Ethernet2, SW1 and R2 will see link drop, but R1 will not.

Case 3

R1<Ethernet1>SW1<Ethernet2>SW2<Ethernet3>R2

Cut Ethernet2 neither router will see link drop.

BTW, dynamic routing protocols will detect the "break" but often slower than interface link drop, i.e. often 50ms vs. multiple seconds to even half a minute.  Difference important if you have VoIP and multiple paths, you want very rapid redirection of VoIP traffic.

Optical networks might behave like Case 2 or Case 3, but often seem they can be configured to behave like Case 1, regardless of number of optical transit hops.

As to my experience, have experience with Verizon, but not with their CWDM/DWDM offerings.

Last company I worked at had its own regional optical networks, so I've seen "breaks" in their/out network not reflect to the edge devices unless the optical network was configured to do so.

BTW, you don't always even need CWDM/DWDM transceivers on you devices.  When I first joint the company, above, has a case of one side of a link having SX and the other side having LX transceivers.  Huh??  The SX and LX connected to optical devices (with multiple hops), but to my devices, it appeared to be a directly connected Ethernet link between the two devices.

[Leo Laohoo]

The place I work have over tens of thousand of kms of "dark fibre" -- We own them all.  We dug our own trenches, we put our own fibre optic and we terminate them into our sites.  And we are still growing our dark fibre footprint.  It is a non-stop work.  
Because of this, we dictate where to terminate them and when we do maintenance on them.  A lot of providers say "dark fibre" when it fact it terminates somewhere into their site/circulator and re-patched again to go to the customer's site.  And this is where the "pain" begins.  If there was ever a fibre break (take for example, by a trunk-hunting-backhoe), how long will it take the provider to fix it?  If the path is "kinked" WHO is going to provide the troubleshooting to determine where the problem is occurring.

Because we own everything, we have dedicated team to do the troubleshooting and the maintenance/repair.  

Because we own everything, we dictate what speed we want on each site.  
And finally, because we own everything, about 90% of the sites that are inter-connected with fibre, are dual-homed.  

But here is the catch:  The entire thing costs a fortune.  Not just the amount of cash needed to put the fibre on the ground but the "holistic" part.  If we had to go to a vendor or an ISP to quote for a dedicated 10 Gbps or 100 Gbps point-to-point link, it would cost, at a minimum, about ten times as our investment over a course of two years.  And remember, that is just an "access" fee/subscription.  The bit about "who is going to troubleshoot a layer 1 issue" is not yet added.  

At the end of the day, it is better to run dedicated point-to-point fibre because the investment value will go nowhere but north the moment the link lights up. 

[Joseph W. Doherty]

@Leo Laohoo do I remember correctly, you're down under?  If so, perhaps running your own trenches is a bit different, legally, there.

In the US, soon as you want to trench across anything but your property, you got issues, especially across a "public" road.  So, often in the US, you're "forced" to deal/work with some form of tele-carrier.

OP didn't mention what country he's in, but from the carriers he mentioned, I suspect also in US.

Last company I worked at is a huge (national) cable video provider, which also grew into a sizable ISP on top of their video (digital) links.  They have much of their own fiber, but they too, still often have to deal with the issue of crossing a road.  Even they would still, on occasion, contract out a MAN/WAN link, especially if not part of their core business and/or "temporary".

[Leo Laohoo]

---

@Joseph W. Doherty wrote:

@Leo Laohoo do I remember correctly, you're down under?  If so, perhaps running your own trenches is a bit different, legally, there.

In the US, soon as you want to trench across anything but your property, you got issues, especially across a "public" road.  

---

Yes, down under we have different regulations regarding digging our own trenches.  We do not do the digging ourselves but we hire trusted engineering firms to do the digging.  They in term go through the administrative process to get things done.  They do the digging, they put down the fibre, etc.  At the end of the day, we own the fibre.  In some cases, we share the same pit with ICON and telcos like Telstra or Optus.  Even though we did not do the heavy lifting, we still have a dedicated team who goes around checking/troubleshooting the fibre links.  Since we own the fibre (and does not go into someone else's circulator), we get the kinks fixed in no time.  

We have no problem with going under the road because we keep a database of the path of the fibre.  This database industrty-wide standard and shared among the business that do this kind of job.  Even with this, there is no guarantee that someone will not dig up the link and this happens.  

---

@Joseph W. Doherty wrote:

So, often in the US, you're "forced" to deal/work with some form of tele-carrier.

---

Because ICON has a large footprint, it means MONEY.  There are rumours circulating every eve of an election of Australia's biggest telco telling the would-be prime minister to sell ICON to them.  At any price.  

[Giuseppe Larosa]

Hello @romanroma ,

>> dedicate link over shared fiber'.

I worked on a renewal of a 45 sites MAN  using DWDM for University of Padova Padua Italy in 2015-2016

It was very interesting.

We moved from 1 GE DWDM rings to 10 GE DWDM rings. Using 100 GE ready new DWDM hardware.

with lambda switching also called G MPLS and in other ways you get a "sub carrier" that travels on the fiber together with other lambdas usinf  different wavelength like radio stations in FM. each lamdbda does no interact with the other ones.

For example in my current company we have lambdas between Milan and Rome both for our DCs and some for our customers,

However, depending on the provider and the "DWDM device" the service level can vary.

We need to monitor them  actively.

With dark fiber you own your own fibers but it is much more expensive and you have no guarantee that the fiber is really yours.

For example a 10 GE link in Europe ca be emulated over an MPLS backbone  tthat is running at 100GE or more

Usually you get better SLAs with dark filber solutions the provider is more responsive and it can faster detect falures but at an hgher price and with limited scalability.

How close are the two datacenters ?

How many "lambdas" could you need in the near future ?

if they are in the range of few hundreds of kms and you want to have space for growth go to DWDM

Hope to help

Giuseppe

[romanroma]

I appreciate everyone's support and answers, which is giving me a better understanding. However, I am still a little confused on the terminology since this is new concepts to me. In layman's terms, is it safe to say that usually different lasers/colors are being used to generate my traffic over 'shared' media, with Wavelength offerings, but the path is predicted on specific layer1/layer2 run within the ISP network? Due to the latter, it is 'best' to still run a ipsec tunnel since 'shared' resources still breakup the traffic into segmented parts, which isolates customer traffic off to a different color/laser? (sorry if the terms are not correct, still learning)

I already have a Dark Fiber connection from site A>B; however, this is only 3mi/5km. However, I am looking at running into a colo location that is 40-50mi/64-84km away. The price for Dark Fiber was WAY OUT OF BUDGET, so I am trying to educate myself on offers related to Wavelength ISP connections.

I noticed my location came up. I am located in USA.

[Giuseppe Larosa]

Hello @romanroma ,

>> but the path is predicted on specific layer1/layer2 run within the ISP network? Due to the latter, it is 'best' to still run a ipsec tunnel since 'shared' resources still breakup the traffic into segmented parts, which isolates customer traffic off to a different color/laser? (sorry if the terms are not correct, still learning)

No optical fiber properties makes the different colors indipendent with no chance to interact.no ned for IPSEC tunnnels at all.

The situation is similar to the FM radio modulation with digital improvements when you listen to a radio station you are not disturbed by other FM radio stations unless you go over the highway far enough that the car radio system is not able to track the original radio station.

This is called heterodyne after receiving a colored lamdba the photodiode in the receiver collect noise and signal of that color only.

(note : DWDM use directional couplers to add or remove a color but the concept is still the same colors are indipendent)

DWDM use DWDM CPE to collect this signal and then it deliver it as a non colored base band 10 GE link to the L3 router / switch CPE. In my case they were either ME 3600 IOS XE  with two 10 GE uplinks or Cisco ASR9001 with IOS XR.

Each DWDM ring is is bidirectional we chreated port channels on each L3 CPE.

The central site had a virtual chassis of 2 x ASR9006 running IOS XR .

Also the DWDM had centrall site multi blade equipment of different types.

see

https://www.infinera.com/wp-content/uploads/Innovative-Packet-Optical-Networks-from-Access-to-Core-0031-BR-RevA-0519.pdf

Hope to help

Giuseppe

[Joseph W. Doherty]

Yes, each "channel" uses a different color or frequency of light.  (BTW, the frequencies are so close, it's not like red, yellow, blue, etc., although sometimes explanations of the technology might shown them as such.)

Yes, they all run side by side, sharing media, much as a radio receives different stations at different frequencies on the same antennae.

What's interesting about the technology, on optical segments, different frequencies can be added, and later split. Usually for our purposes, you just see the link as any other end-to-end point-to-point Ethernet connection.

In other words, unless you're doing optional network engineering, "how" doesn't much matter beyond, perhaps,  the need to use a CWDM or DWDM transceiver for a specific frequency and/or channel.

Whether you need to use IPSec would much go along with whether you believe there's a need to do so using a dedicated p2p Ethernet link.  (Generally, if you're worried about your WAN provider reading your transit traffic [more difficult, I believe, with this technology], they you might want to use IPSec.)

[Leo Laohoo]

---

@romanroma wrote:

The price for Dark Fiber was WAY OUT OF BUDGET

---

Putting down private fibre is always expensive.  Like a high-yield investment, look at how fast the value of the link is when it lights up.  

Compare the value, for instance, of a link supplied by an ISP vs dark fibre over 5 years.  

For obvious reasons, do not just put a single pair.  If there is a way to put, say 24 pairs, it would work out even better.

[romanroma]

I was HAM radio operator for a few years, so I understand what you are saying about FM, due to that latter - I appreciate everyone's insight.