FI/IOM : best cabling design

[delahais]

In our UCS system with 2 chassis, we have 2 links between IOM 2208 and FI 6248 :

For a IOM, what is the best cabling design :

- to connect theIOM to Eth1/1-2

- to connect the IOM to Eth1/1 ; Eth1/8

... In mean, it is better to use the same Unified Port Controller (UPC) ASIC [called Carmel, i think] or to use 2 differents UPC ?

and why ?

thx.

Nicolas.

[Robert Burns · Accepted solution]

With the 6200 and 2208 the best design option is actually to ensure all server links (FI <-> IOM) are connected to the SAME UPC (carmel ASIC).  This will provide the max # of vifs to the system.  By spreading your connections across different UPCs may increase redundancy, but it will decrease your max vif count at the same time.

If you want to see the UPC mapping you can log into the CLI, connect to the NXOS context and run the following command:

FI-A-6248-A(nxos)# show hardware internal carmel all-ports

Carmel Port Info:

name   |log|car|mac|flag|adm|opr|m:s:l|ipt|fab|xcar|xpt|if_index|diag|ucVer

-------+---+---+---+----+---+---+-----+---+---+----+---+--------+----+-----

xgb1/1 |0  |0  |0 -|b7  |dis|dn |0:0:f|0  |92 |0   |0  |1a000000|pass| 0.00

xgb1/2 |1  |0  |1 -|b7  |dis|dn |1:1:f|1  |88 |0   |0  |1a001000|pass| 0.00

xgb1/3 |2  |0  |2 -|b7  |dis|dn |2:2:f|2  |93 |0   |0  |1a002000|pass| 0.00

xgb1/4 |3  |0  |3 -|b7  |dis|dn |3:3:f|3  |89 |0   |0  |1a003000|pass| 0.00

xgb1/5 |4  |0  |4 -|b7  |dis|dn |4:4:f|4  |90 |0   |0  |1a004000|pass| 0.00

xgb1/6 |5  |0  |5 -|b7  |dis|dn |5:5:f|5  |94 |0   |0  |1a005000|pass| 0.00

xgb1/7 |6  |0  |6 -|b7  |dis|dn |6:6:f|6  |95 |0   |0  |1a006000|pass| 0.00

xgb1/8 |7  |0  |7 -|b7  |dis|dn |7:7:f|7  |91 |0   |0  |1a007000|pass| 0.00

xgb1/9 |8  |1  |0 -|b7  |dis|dn |0:0:f|0  |80 |0   |0  |1a008000|pass| 0.00

xgb1/10|9  |1  |1 -|b7  |dis|dn |1:1:f|1  |87 |0   |0  |1a009000|pass| 0.00

xgb1/11|10 |1  |2 p|b7  |en |up |2:2:f|2  |81 |0   |0  |1a00a000|pass| 0.00

xgb1/12|11 |1  |3 p|b7  |en |up |3:3:f|3  |86 |0   |0  |1a00b000|pass| 0.00

xgb1/13|12 |1  |4 p|b7  |en |up |4:4:f|4  |82 |0   |0  |1a00c000|pass| 0.00

xgb1/14|13 |1  |5 p|b7  |en |up |5:5:f|5  |85 |0   |0  |1a00d000|pass| 0.00

xgb1/15|14 |1  |6 p|b7  |en |up |6:6:f|6  |83 |0   |0  |1a00e000|pass| 0.00

xgb1/16|15 |1  |7 p|b7  |en |up |7:7:f|7  |84 |0   |0  |1a00f000|pass| 0.00

xgb1/17|16 |2  |0 p|b7  |en |up |0:0:f|0  |75 |0   |0  |1a010000|pass| 0.00

xgb1/18|17 |2  |1 -|b7  |dis|dn |1:1:f|1  |76 |0   |0  |1a011000|pass| 0.00

xgb1/19|18 |2  |2 -|b7  |dis|dn |2:2:f|2  |74 |0   |0  |1a012000|pass| 0.00

xgb1/20|19 |2  |3 -|b7  |dis|dn |3:3:f|3  |77 |0   |0  |1a013000|pass| 0.00

xgb1/21|20 |2  |4 -|b7  |dis|dn |4:4:f|4  |78 |0   |0  |1a014000|pass| 0.00

xgb1/22|21 |2  |5 -|b7  |dis|dn |5:5:f|5  |73 |0   |0  |1a015000|pass| 0.00

xgb1/23|22 |2  |6 -|b7  |dis|dn |6:6:f|6  |72 |0   |0  |1a016000|pass| 0.00

xgb1/24|23 |2  |7 -|b7  |dis|dn |7:7:f|7  |79 |0   |0  |1a017000|pass| 0.00

xgb1/25|24 |3  |0 -|b7  |dis|dn |0:0:f|0  |71 |0   |0  |1a018000|pass| 0.00

xgb1/26|25 |3  |1 -|b7  |dis|dn |1:1:f|1  |64 |0   |0  |1a019000|pass| 0.00

xgb1/27|26 |3  |2 -|b7  |dis|dn |2:2:f|2  |65 |0   |0  |1a01a000|pass| 0.00

xgb1/28|27 |3  |3 -|b7  |dis|dn |3:3:f|3  |70 |0   |0  |1a01b000|pass| 0.00

mfc1/29|28 |3  |4 -|b7  |dis|dn |4:4:0|4  |66 |0   |0  |0101c000|pass| 0.00

mfc1/30|29 |3  |5 -|b7  |dis|dn |5:5:1|5  |69 |0   |0  |0101d000|pass| 0.00

mfc1/31|30 |3  |6 -|b7  |dis|dn |6:6:2|6  |67 |0   |0  |0101e000|pass| 0.00

mfc1/32|31 |3  |7 -|b7  |dis|dn |7:7:3|7  |68 |0   |0  |0101f000|pass| 0.00

sup0   |32 |4  |4 -|b7  |en |dn |4:4:0|4  |62 |0   |0  |15020000|pass| 0.00

sup1   |33 |4  |5 -|b7  |en |dn |5:5:1|5  |59 |0   |0  |15010000|pass| 0.00

The column you're concerned with is the one labeled "car" (for carmel).  As you probably can see the first column is the Interface #.

FYI - I have yet to see a failed port controller ASIC.  Their MTBF is far beyond many other components including SPFs, cables, etc.  I wouldn't sweat worrying about ASIC level redundancy in this regard.

Regards,

Robert

[delahais]

Hello Robert,

Many thank for your reply.

Do you have more information about Carmel UPC : what is the max vif count ? how could we reach them.... which kind of configuration consume vif ?

Do you have any more detail in order to fully understanding of the table "show hardware internal carmel all-ports"

Many thx.

Nicolas.

[Robert Burns]

Nicolas,

We don't publish information about the UPC.  Much of that information is proprietary.

I will tell you how the vif count is accumulated - it's not a simple calculation.

A VIF is any virtual interface - vfc or veth.  This is the logical endpoint where a vNIC or vHBA terminates to on the Fabric Interconnect.  VIFs are limitated by a few factors: Network, OS/BIOS and the system. 

First Generation HW Considerations

With the first generation of interconnects and IOMs (6100 and 2100) the calculation was simple:

Max VIFs per adaptor = (15*n) - 2

[where n = the # of acknowledged IOM uplinks]

The first generation hardware couldn't do any link aggregation so it's fairly straight-forward.

Second Generation HW Considerations

With the second generation of interconnects & IOMs (6200 & 2200) the calculation is dependent on whether or not you're using Port channeling between the FI and IOMs:

Using Individual Links:

Max VIFs per adaptor = (63*n) - 2

[where n = the # of acknowledged IOM uplinks]

Using Port Channel Between FI & IOM:

MAX VIFs per adaptor = (63*n) -2

[where n is the lowest # of IOM uplinks going to a single carmel ASIC]

For example if you had two port-channeled IOM uplinks, and they went to the same carmel ASIC on the FI, then you would have 124 available VIFs for that adapter. (63*2) - 2 = 124

Alternately if you had four port-channeled IOM uplinks, and they were split between two carmel ASICs on the FI, then you would have 61 available VIFs per adapter.  (63*1) - 2 = 61

Now taking these limitations into consideration, you also have to consider how many vNICs/vHBAs the OS itself will support. 

Ex.

Windows 2008 R2 = 20 (Eth + FC)

RHEL = 32 (Eth + FC)

ESX 5 = 32 (Eth + FC)

The most common limitation you will probably run into would be in VM-FEX deployments.  This is where dynamic vNICs are assigned directly to VMs (bypassing the Hypervisor).  With VM-FEX each OS supports the following amount of dynamic vNICs:

ESX 4.0U1 - 54

ESX 4.1 - 54

ESX 5 - 112

RHEL KVM - 112

Hope this helps.  If you have specific questions let me know and I'll do my best to explain.

Regards,

Robert

[delahais]

Hi Robert,

Many thx for all explanation. I do pretty understand.

Just a question on this way. Do we have the same best pratices to do a vPC on Nx 5k : use the same UPC to configurate a vPC ? and why ?

Many thx.

Nicolas.

[Robert Burns]

I don't understand your question.

Are you asking about configuring a VPC between a UCS FI and Nexus 5000?

Robert

[delahais]

Sorry.

I mean, on a vPC domain with 2 Nx 5500 (any connection with UCS), do i have the same best practice ?

For exemple, if i want connect 2 couples of NX 5500 with a vPC 40 giga. on each 5500, It is better to get 2 ports from the same UPC (eth 1/1-2) or 2 ports from different UPC (eth1/1 ; eth 1/9)

and why ?

best regards.

Nicolas.

[Robert Burns]

Since there's no VIF limitation in the same way as UCSM, the same consideration do not apply.  You're free to use the same or different carmel ASICs.  This being the case you can spread the VPC interfaces across carmel ASICs for better redundancy.

Regards,

Robert

[delahais]

ok. I get the point ! many thx for all your explanation.

have a good night (or day)

Best regards.

nicolas.