Port Channel over L2 MPLS links

[Grant Drage]

Hello.

I was hoping that someone could over some suggestions on best practice or recommendations for configuring port channel to bundle to layer 2 MPLS links that we have.

We have racks in two geographically separated data centres, each rack has a stack of Cisco 3850 switches, and there are two 100Mbps layer 2 connections linking these stacks together.  The links are provided by our supplier and runs over their core network (I can get more info on their setup if need be).

I had initially just configured a simple port channel bundling the two ports on "Switch A", and the same on "Switch B".  However I have since noticed that one of the ports was put into error disabled state, and I have my doubts that we ever had 200Mbps throughput over the portchannel.

The logs showed:

UDLD-4-UDLD_PORT_DISABLED: UDLD disabled interface Gix/x/x, aggressive mode failure detected
%PM-4-ERR_DISABLE: udld error detected on Gix/x/x, putting Gix/x/x in err-disable state

How I understand is that one of ports recieved an unexpected (or incorrect) BPDU packet, and shut the port down as part of spanning tree process to stop a loop forming.

For the time being I have removed the port channel config and re-opened all the ports, so I believe one of the ports is now in blocking state.

Whilst redundancy for these links is great, ideally I would like to have the links bundled so I also get the benefit of higher throughput.

Any thoughts are greatly appreciated.

Thank you

[Reza Sharifi]

Hi Grant,

You can not Portchannel the 2 100Mb together as the provider is giving you 2 100Mb links and not 200Mb total. If you want to bundle the 2 100Mb together, you need to talk to your service provider.  Most providers don't want to change their setup for one customer and the stay away from bundling the links but it is worth a talk.

HTH

[rafael_acc]

You can run UDLD in two modes - don't remember both names but one is certainly "aggressive" mode. The running mode must match at both ends and in your case, it was probably not the case. Furthermore, if this is Ethernet, you wouldn't normally need UDLD for this is a feature rather used with fiber links for reasons I won't detail here.

My understanding is you are trying to etherchannel across an MPLS cloud; this is possible by means of Layer2 protocol tunnelling - i'm sure you can find plenty of info using google. However, this option may not be possible since it is an option which must be enabled on all Layer2 devices and this is indeed a big limitation of this feature. And I hope my memory is not failing me on this...

I can't personally think of any other way two get bundle Layer2 links. If it was Layer3, that's a different story. 

And yet, maybe if you could run two GRE tunnels and then use load sharing over Layer3 links instead ... 

This all depends on your requirements, obviously. 

[Grant Drage]

Hi,

Thank you for your reply and suggestions.  I have been reading up on layer 2 protocol tunnelling and I'm not sure if this will work for us.  According to these guidelines for the catalyst 3550 (http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3550/software/release/12-1_19_ea1/configuration/guide/3550scg/swtunnel.html#wp1006657) it states:

•If you enable PAgP or LACP tunneling, we recommend that you also enable UDLD on the interface for faster link-failure detection.

•Tunneling is not supported on trunk ports. If you enter the l2protocol-tunnel interface configuration command on a trunk port, the command is accepted, but Layer 2 tunneling does not take affect unless you change the port to a tunnel port or an access port.

•EtherChannel port groups are compatible with tunnel ports when the 802.1Q configuration is consistent within an EtherChannel port group.

(maybe its a different and is supported on 3850's)

Also, I cant use layer 3 over this link as I am looking to extend VLANs over the link so devices and virtual machines in each of the datacentre are in the same subnet and broadcast domain.

I will speak with the supplier to see what they are willing to do.

Thanks again.

[Bilal Nawaz]

If your service provider are providing you with a L2 service then there is no reason why this cannot work. It should be straight forward. Could you kindly show us the config you applied to Switch A and B. One of the links worked but the other didn't, did you check fibre connections etc..?

[rafael_acc]

it really does depend! Normally, Layer2 services means you can run layer2 dataplane ...

for etherchannel to work the way he wants, we are talking more like control plane, so the etherchannel can come up in the 1st place. Only then, we could talk about data ...

[Grant Drage]

I have uploaded a quick layout of the switches in the datacentres just to show how things are linked, I have also uploaded the current configs of the switches (with certain details removed).

Thank you

[rafael_acc]

To me it really sounds like what you are trying to achieve is Etherchannel over the provider's MPLS core.

They are already running MPLS in their core so your solution will have to use an MPLS based VPN - say VLL or VPLS. In the cisco world, this is called pseudowire - but this feature is hardware dependent. Your switches, wont' support that.

I have written a blog a while back about this:

http://blogbt.net/index.php/2013/07/mpls-applications-l2vpns-pseudowires/

Only in your case you have two pseudowires and to etherchannel them, you would have to hard-set the config at both ends (same as when you hard set trunk ports). You would have to hard-set the config since LACP/PAgP would not run end-to-end, unless you somehow, tunnel those protocols at layer2 - which is why I have initially mentioned it. Having said that, I don't actually think you can do that over an MPLS core.

So, what you are trying to do, I really dont' see how you can achieve it on your own. If at all possible, you would need to liaise with the provider.

I hope this makes sense mate.

* NOTE * Just noticed that particular blog has some display issues in IE. If you check it out, make sure you use FF or Chrome. Hopefully it shows ok then. ...