VxLAN is here - now what?

[lamav]

Folks:

I'm wondering just how much and how fast this will change the game in the data center. The details of its operation are still sketchy, but from a high level, it promises to allow a return to L3 isolation at the access layer as well as vmotion between subnets. The importance of all the progress that's been made recently in L2 multipathing (TRILL), horizontal scale out with L2 (flattening the data center) may very well become less important and less applicable, except perhaps for non-IP traffic, like FCoE.

No matter what, this is huge. The concept is not a new one for those who have been thinking about this for a while. Personally, I was thinking more along the lines of MPLS, GRE or pseudowires, but this is similar.

Would love to hear some thoughts.

Victor

[Jon Marshall]

Hi Victor

Personally i think it is too early too say as the DC is experiencing quite a rapid change in technologies/infrastructure at the moment.

My initial thoughts though are that the 2 are not direct competitors ie. vxlan and fabricpath (Cisco TRILL) although there does seem to be some overlap.

Fabric path is a L2 technology designed primarily to scale L2 networks way beyond what is currently available, allowing for multiple L2 paths none of which are blocking. It basically creates a much more bandwidth capable fabric between any switch endpoints with some scalability features for mac-addressing ie. the edge switches only need to know about end devices they are actually transmitting traffic for which greatly reduces the cam table per switch. In addition the "core" switches actually don't learn any macs beyond the edge devices, they simply use labels in conjunction with IS-IS.

Interestingly in a recent Ask the Experts session i asked how , if you had one very large L2 domain, you implemented any security ie. how could a firewall for example be integrated into the fabric. The answer i got was that it couldn't simply because no firewall could actually keep up with the bandwidth of fabric.

VxLAN i have seen described as L3 or L2.5. Key thing is it is not described as L2. It is an overlay technology just as OTV is although obviously not the same. And an overlay technology has to be overlaid over something, in this case a fabricpath enabled network, although vxlan does not dictate the use of fabfricpath as far as i am aware. What vxlan seems primarily concerned with is to go beyond the limitations of the 4K vlan limit because in cloud networks there is the potential to need many more vlans than that.

So i don't think the 2 things are dealing with the same problems ie.

fabricpath = create a L2 switch fabric that is more scalable and offers much more bandwidth to edge switches via many L2 equal cost paths

vxlan = scale beyond the traditional limit of useable number of vlans and allow the logical vlan to extend across subnets

Whether or not the 2 technologies will become intergrated is anyone's guess, especially as there seems to be a new protcocol/technology for the DC every other day. The way these things are moving it is quite easy to see the network engineers job in the DC being concerned primarily with the simple switch fabric, with the vlan/subnet management being managed by the server teams.

Those are my initial thoughts. I could of course be talking complete rubbish so please feel free to completely disagree

Jon

[lamav]

Hi, Jon:

I'm glad to see you back on this biard again.

TRILL is a L2 multipathing solution. One of the purposes of VxLAN is to remove the need to span the L2 domain by creating a tunneling construct - an overlay - that gives the illusion of a "stretched" L2 domain. With VxLAN you can create routed links and depend on ECMP for multipathing.

[Jon Marshall]

Victor

Yes but vxlan is still dependant on the underlying L2 network.  You can rely on L3 ECMP for multipathing but you can't use a L3 path if L2 STP has decided to block that path.

L3 only comes after L2 in any network ie. if you have redundant links at L2 then with STP some of them have to be blocked ie. you cannot create a tunnel over a blocked path.

And if you have L2 vlans in that network then STP will block some paths unless you use a technology that does not use STP ie. fabricpath.

So fabricpath creates the underlying network of many L2 equal cost paths, non of them blocking. Then on top of this you can create your L3 tunnels using vxlan.

Again, i may have misunderstood something about vxlan and the way it creates tunnels but as far as i can see the paths at L2 have to be available in the first place.

Edit - it may also be how we view fabricpath.  I'm not aware that fabricpath dictates the use of one L2 vlan, ie. you can run multiple vlans across it. Could be wrong though.

Jon

[Jon Marshall]

Victor

Just read some other paper which describes vxlan as running over a L3 network ?

If this is your understanding then apologies as i did not understand that was the proposed underlying network infrastructure. I did tell you i might be talking rubbish

If so then it seems vxlan is more akin to OTV in the sense it allows the extension of L2 vlans across L3 boundaries. So i can see where you are coming from in your question.

Where i am not clear without reading more is whether they are suggesting that the entire network will be made of up of routed interconnections which i'm not sure they are. I could see a scenario where the core fabric was running fabricpath ie. L2,  and the routing was done at the edges of the fabricpath for the vlan isolation. On top of this you could use vxlan to allow for many more vlans and for vlan extension.

I'm not sure a routed core could scale to fabricpaths bandwidth so you would be losing out on that sense.

Jon

[lamav]

Jon:

Correct, you would overlay the VxLAN over a L3 network. That's what I meant when I mentioned routed uplinks from the access layer. So, you can, not necessarily HAVE to (as far as I understand), actually go back to the days of yesteryear (5 years ago) and deploy a routed access layer and not have to worry about spanning L2 domains. No funky vPC, VSS, vLT, mLAG, TRILL engineering feats to reinvent Ethernet.

The VxLAN encapsulation can take place at the hypervisor, so this is a hypervisor-to-hypervisor tunnel, where each VTEP (Virtual Tunnel End Point) can reside on the virtual switch component of a physical server. The new header will hide the address details of the underlying data packet, which will enable multitenancy applications - a sort of IPVPN in the data center.

Anyway, I was not asking a question, per se. I was trying to engender a discussion and hear some thoughts from people on this board, especially the more brilliant and seasoned minds on here - you all know who you are.

[Jon Marshall]

Victor

I will drop out and let others answer but the way i see it one does not replace the other so in answer to your general question i think we will see both technologies deployed together to not only increase bandwidth but also scale to many 1000s of vlans.

Specifically on the L3 routed network. I don't think that necessarily means L3 routed links ie. at a very high conceptual level

sw1 -> cs1 -> cs2 -> sw2

sw1 and sw2 are doing inter-vlan routing and providing vlan isolation. They are also the vxlan endpoints. cs1 and cs2 are simply there to represent the core of the DC. This could easily be a fabricpath setup.

Unless every single switch interconnect is a L3 routed link ie. "no switchport", and this means your core switch fabric as well then you still need to worry about all the "funky" stuff and this is where fabricpath can help. Unless of course you are actually proposing to have every single interconnect L3 routed and i haven't seen this proposed in terms of vxlan anywhere.

If you have let me know.

Jon

[lamav]

Jon, we're going in circles. There's nothing cryptic about what I'm saying. I didn't say one MUST use routed links throughout the network in a VxLAN environment. I said you CAN. And IF YOU DO, then you don't have to worry about spanning the L2 domain using the technologies I mentioned.

[Jon Marshall]

Victor

I would guess you know a lot more about this than me but compared to your previous posts you seem to be rather defensive on this.

You asked if the advent of vxlan would reduce the need or replace fabricpath. That to me seemed the basic question.

I responded by saying that even in a L3 network you still needed L2 paths. You then suggested that you didn't need to worry about this because it was a routed network. This to me seemed the key point of your argument. And all i was pointing out as that unless every single link was routed (and implict in this was the suggestion that i think this highly unlikely) then you do in fact need to worry about L2 paths being blocked or not.

So i basically answered your question by saying no i don't think vxlan will replace fabricpath and that simply using routing from edge to edge does not mean the core could not be L2.

I have said all along i may well be incorrect but you haven't really explained why you think vxlan would reduce the need or replace technologies such as fabricpath/VSS/vPC etc. You have simply said "it is routed therefore you don't need to worry about L2" which may or may not be factually correct but i thought that is what we were discussing.

Jon

[Jon Marshall]

Hey Victor

Apologies if last post came across a bit strong. Not intended, i just missed the main point of your question.

Jon

[lamav]

Jon, no worries. I wasn't getting defensive, just a bit flabbergasted at the inability to get you to understand what I meant.

So, just to reiterate...I am saying that VxLAN technology, as it is proposed today, can allow you to run a routing protocol in the access layer, thereby removing the need for any L2 inter-switch links in the network or STP, while at the same time providing the "illusion," if you will, of a spanned L2 domain through tunneling. This will allow you to execute vmotions and do other virtualization-based tasks that require a L2 adjacency without having to contend with TRILL or any other band aid to make Ethernet more scalable or to provide full cross-sectional bandwidth. No L2 multi-path. Instead, rely on L3 ECMP from the access layer to the core.

EDIT: So, wrt competing, if someone wants to build a scaled out layer 2  domain, then TRILL is it. If someone wants to design a cloud based on L3  isolation, then VxLAN is it. They provide different solutions to 2  different approaches. Where they do compete is in the approach a network  architect will want to take.

[Jon Marshall]

Victor

EDIT: So, wrt competing, if someone wants to build a scaled out layer 2  domain, then TRILL is it. If someone wants to design a cloud based on L3  isolation, then VxLAN is it. They provide different solutions to 2  different approaches. Where they do compete is in the approach a network  architect will want to take.

Agreed but i would also suggest that you actually only need the L3 isolation at the edge. So you could use a different approach and combine vxlan for vlan isolation and fabricpath for sheer bandwidth within the core. I know TRILL does deal with a scaled out L2 domain as you put it but it does have other advantages ie. sheer bandwidth. I'm not sure how scalable/manageable using L3 routed links everywhere would be and i don't think you need to. As you say vxlan acting as a sort of IPVPN and that is the only point i was trying to make ie. IPVPN doesn't really care how the internal MPLS cloud is built or works and i'm not sure that vxlan does.

I know before you said you weren't saying a vxlan had to be built with L3 P2P links everywhere and i agree. But if it doesn't then you are back to L2 STP etc. which surely means fabricpath is worth a look.  So i can't see it as a direct alternative for a network designer ie. vxlan or TRILL, why not a combination of both ?

I'm not essentially disagreeing, just trying to clarify for myself as much as anything. I get the feeling i am missing something about vxlan that you understand and it would suddenly all make sense

Jon

[lamav]

Jon, you always make sense and you have been my technical advisor for years. So nothing to be sorry about. I respect your opinions - always.

[Giuseppe Larosa]

Hello Victor,

>>

The VxLAN encapsulation can take place at the hypervisor, so this is a hypervisor-to-hypervisor tunnel, where each VTEP (Virtual Tunnel End Point) can reside on the virtual switch component of a physical server. The new header will hide the address details of the underlying data packet, which will enable multitenancy applications - a sort of IPVPN in the data center.

yes indeed it was time that virtualization vendors could join us in a routed L3 world!

there is enormous activity on data centers and here I would like to mention MP BGP based MAC VPN by Juniper (under developement), trill and all these approaches have the defect to force back to L2 flat networks, communication between entities like hypervisors can be seen as an application in OSI model, so if there is an encapsulation to carry info in a way that allows to leave the single broadcast domain limitation it is wellcome.

This does not mean that it will get wide adoption as the drivers for some push actions are not only technical but are ways to attack new market segments

think of FCoE, I have used FCoIP  (actually TCP) in the past successfully, but FCoE allows to put a foot inside the data center storage core.

note: these are my personal opinions and I may be wrong as in other cases

Hope to help

Giuseppe

[lamav]

Giuseppe:

I agree. I'll take a stable routed network with L3 isolation any day over huge L2 networks with band aid technologies to make Ethernet work differently than it has been for the last 30 years. Moreover, the lingering issues that TRILL does not resolve is the flat character of MAC addresses and the fact that the Ethernet control plane is still tied to the data plane (MAC learning as a product of forwarding).

While we're on the subject, I'm no big fan of FCoE either. I have a lot of clients who have Nexus deployed in their data centers, but none of them is using the FCoE functionality. The DCB standards have only recently been ratified and they are still developing and we discover shortcomings. Also, standards like BB5 and ETS leave a lot of room for vendor implementation, like the scheduling algorithm that ETS uses. There are a lot of moving parts, from the virtualized components of the server, to the CNA, to the ToR switch, etc....and without vendor interoperability, vertical integration will have to come at the price of proprietary vendor lock-in.