FI Uplink Port Channel Questions

[nicholas.heaton]

I am attempting to design the best FI to 5K setup possible given my limited understanding of networking and UCS FIs. I have a few questions that I am hoping someone can help me with.

1. Can I configure a port channel that spans multiple expansion modules? For example, can I configure a PO that uses port 2/1, 2/2 and 3/1, 3/2? Or does a PO have to all reside on the same module?

2. I'm reading the documentation and it states that an Ethernet uplink port channel can be configured up to 16 ports. I've also seen other articles online (going back a few years ago) that state you can configure up to 16 ports, but only 8 will be active at a given time. Assuming connectivity from an FI to a pair of uplink Nexus 5Ks, will all 16 be active, or only 8?

3. Can I configure two separate Ethernet uplink port channels to a pair of 5Ks from the FIs? If yes, is there a good or bad reason to do this?

Ultimately, my goal is to have 12 10Gbps uplinks going to two 5Ks in a VPC setup. If (but not sure) port channels max out at 8 active links, then maybe having two separate port channels will allow you to get around this? Again, I'm just getting my feet wet with this and anything helps.

[Walter Dey]

Hi Nicholas

This documents 

https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/ucs-manager/Reference-Docs/Configuration-Limits/3-1/b_UCS_Configuration_Limits_3_1.html#reference_7009DAE1E4764747B11A8CDE5FF98E99

explains the limits, eg. number of port channels, and number of links per port channel

[nicholas.heaton]

Looking at the referenced article, I see that the max number of links in a port channel is 16. I've already confirmed that. What this article doesn't state is if all 16 links are active at once, or if only 8 are active and the other 8 are in hot standby.

This also doesn't address if you can span a port channel across different extension modules (i.e. using ports 2/1, 2/2, and 3/1, 3/2).

It does indicate that you an have multiple port channel uplinks, but it doesn't indicate if it will forward traffic out both at the same time, or if one of the port channels will be downed for STP.

Can someone else please clarify? Thanks.

[Walter Dey]

This also doesn't address if you can span a port channel across different extension modules (i.e. using ports 2/1, 2/2, and 3/1, 3/2).

I assume you mean Uplink (Northbound) PC's:

I could only find a Statement about Fibrechannel PC's:

To minimize the traffic disruption during system changes, form a Fibre Channeluplink port-channel across

the fixed and expansion modules.

Therefore I'm quite confident, that it applies for Ethernet as well. (another test: just configure it, and if it allows it, ok).

[mojafri · Accepted solution]

Looking at the referenced article, I see that the max number of links in a port channel is 16. I've already confirmed that. What this article doesn't state is if all 16 links are active at once, or if only 8 are active and the other 8 are in hot standby.

All 16 links will be active. Though gen-1 & 2 resembles 5k platform, but there is lot of difference. 

This also doesn't address if you can span a port channel across different extension modules (i.e. using ports 2/1, 2/2, and 3/1, 3/2).

No limitation. You can have span of pc contain different interfaces from diff modules.

It does indicate that you an have multiple port channel uplinks, but it doesn't indicate if it will forward traffic out both at the same time, or if one of the port channels will be downed for STP.

Depends on which mode you are running - End-host or switching mode. In end-host both port-channel will be used as active data path. All your vif/veth will be pinned to uplink port-channel automatically(recommended) or you can use manual pinning policy. 

Hope this helps.

Regards,

MJ

[Walter Dey]

Just another 2c, that is mostly forgotten.

A L2 communication (within same vlan, which exists on fabric A and B), between 2 blades within a UCS Domain: Blade 1 with SP 1 is the initator sending traffic over vethx interface connecting to fabric A; the destination is Blade 2 with SP 2 receiving this traffic over ethy connecting to fabric B. Because UCS FI is a L2 switch only, fabric A and B are isolated: the above traffic has to leave the UCS Domain, and getting L2 switch on the Northbound Ethernet switches, which implies that the vlan traffic between the 2 upstream switches has to be configured accordingly.