why would it be advantageous to have 40-GbE instead of four 10-GbE with LACP?

[ciscomdsmagic]

I am trying to get the answer from the document below:

http://www.cisco.com/c/en/us/products/collateral/switches/nexus-3000-series-switches/white_paper_c11-726674.pdf

But, when I click on it, it didn't give me the document itself. 

Can somebody please show me the document ?

[Bryan2015]

The main advantage is that you're using only one single interface on your core or distribution switches, depending on your network you may don't have lots of free interfaces to add 4 interfaces on a LACP bundle going to a single switch or switch stack.  Also is easy to manage and troubleshoot. On my particular network I have 2 10-G from each switch stack to the core using layer 3 routing and not port-channel, it has equal cost load-balance and it work perfect. Please let me know if this was helpful. 

[Edward Clear · Accepted solution]

The primary advantage I see is that any given flow (as defined by your LACP hash setting) is capped at 10Gbps with the 4x10G solution.  If you use a 1x40G, that flow could reach 40Gbps (assuming all other hops support it).  Another potential benefit is that you only need a single physical link, which could be a concern depending on the capacity of your cable plant.  This goes along with @Bryan2015's comment about only using a single interface.

The down side is that there is no redundancy, i.e. you have a single point of failure.   And there is a flip side to the cabling, in that you need better quality fiber to do 40G.  Although I found my 12 year old OM3 fiber plant was fine for 40 and 100G within the data center.  I only exceeded the theoretical link budget for certain cross patching that we didn't anticipate when we originally laid it out.  However there seems to have been plenty of margin as it's working.  You can also run 10G on copper. I don't think they have a non-DAC copper 40G solution. 

[ciscomdsmagic]

@Edward Clear ,

Thanks!

A follow-up: in 4x10G case, what if the flow is capped, would then fragmenting start to happen? 

[Edward Clear]

Not fragmenting, more of a congestion case, possible packet discard.  Same as if you only had one 10G link.  

It's just one of those not so obvious things about link aggregation.  Just because you have  40G in the aggregate doesn't mean that any given flow can use that much bandwidth.  I've lost track of the number of times I've had to explain this to architects and lay (non-network) engineers on my program.

Fragmentation is only when you hit a hop that has a smaller MTU than the packet size, not "smaller bandwidth".

[Joseph W. Doherty]

BTW, to add to the information Edward already posted, when it comes to capacity, LACP (or any Cisco Etherchannel) doesn't dynamically load balance.  So, besides one flow always being limited to the capacity of a single link, some of your links could congest while others have lots of excess capacity.  (Also, a "poor" choice for the hashing algorithm, or an optimal one not being available on the platform, all the traffic might only use a single link.)

As Edward already described, from a redundancy aspect, Etherchannel can be the hands-down winner.  From a capacity aspect, a higher bandwidth single link is better.