Best practices for Server Port cabling (FI<->IOM)

[bruceperttunen]

I currently have two chassis(5108) connected to two Fiber Intrerconnects (6248).  Two cables from each IOM to each FI. 

Chassis 1:

IOM1/1 - FI-A port 1

IOM1/2 - FI-A port 2

IOM2/1 - FI-B port 1

IOM2/2 - FI-B port 2

Chassis 2:

IOM1/1 - FI-A port 9

IOM1/2 - FI-A port 10

IOM2/1 - FI-B port 9

IOM2/2 - FI-B port 10

I now have a third chassis I need to connect. What is the best way to do this?  Can I use port 3 and 4 on the FI?  Or 11 and 12?  Does it matter the order the server ports on the FI get used?

Thank you.

[smcquerr]

Bruce,

What type of IOM do you have installed in the Chassis?  Are you using port channling from the IOM to the FI?

Do you have an expansion module or are you using the base ports for uplinks and where are your uplinks connected?

With the 6248, the ports you choose technically do not matter from a pure connectivity standpoint but you may limit the number of VIF paths avalible for your servers if you choose ports that are not in the same ASIC port group (8 ports per port group).

Typically we would recomend that you connect your IOMs across a contiguous port group and give yourself room for growth within the same ASIC group.  For example if you were using a 2208 IOM and wanted to start with 2 ports and have room to add additional links for BW the recomendation would be to start with ports 1 and 2 and leave 3 - 7 open to expand that module to it's maximum.  Then ports 9 and 10 and leave ports 11-16 for expansion.  Next ports 17 and 18 and leave 19-24 open for expansion.  This would also allow you to take advantage of ASIC buffer space on the FI for each chassis/IOM.

If you are only going to use 2204s or 2104s then you could leave only 2 ports open between connections.  The key is making sure that you give yourself the ability to grow without splitting the connection between 2 ASICs.  If you start out with 2 ports connected you are at the maximim VIFs (118 per server) but if you expnaded had 2 connections in ports 1 and 2 and then a single connection in port 11 (assuming a channel group) for that IOM then you would limit yourself to 63 VIFs per server.

My suggestion would be if you have 17 and 18 avalible that would be a good place to connect the third Chassis as this would allow you to split the load from the three chassis across multiple ASIC groupls and provide the ability for you to grow your BW from the chassis to the FI with the best scaliblity in the future.

Take a look at the following documentation for more detials.

http://www.cisco.com/en/US/partner/docs/unified_computing/ucs/hw/chassis/install/ucs5108_install_chapter_010.html#concept_C4275AF7713D4231842F86B80288D633

Steve McQuerry

UCS - Technical Marketing Engnieer

[bruceperttunen]

Steve,

The two existing chassis have 2208 but the new chassis has 2204.  I am using port channeling.

I do not have an expansion module on the 6248.  I have it configured with Eth ports on 1-24 and FC on 25-32.  Server ports are 1,2 and 9,10.  Uplink Eth is 17 and 18.  Uplink FC is 25 and 26.

So I have to work within the 1-16 range, unless I can put server ports on 19 thru 24 which is above existing Eth uplinks 17 and 18.  Would it be a good idea to put Uplink Eth and Server Ports on the same ASIC?  Or do I use ports 5 and 6 and realize that I will limit my possible future BW on the existing chassis(which is not a concern at this time).

Thank you.

Bruce

[smcquerr]

You could connect above the server ports.  There is enough buffer space to handle all the ports being active my suggestion was based on if you weren't using any ports in a specific group then its not a bad idea to take advantage of as much of the HW capabilities as possible. 

If you think you will expand the 2208s to all 8 ports you may want to consider using ports 20 and 21.  This would give you 2 ports for exapndability of your uplinks keeping them contiguous as well as 2 ports for the 2204.  It would also keep all your IOMs connected in the same ASIC groups for future expandability.

If you don't think you will ever expand your 2208s over 4 uplinks then you could start with ports  5 and 6.

Kenny makes an excellent point from a licensing standpoint.  Whichever ports you use make sure that you have licensing avalible for them.

Steve McQuerry

UCS - Technical Marketing Engineer

[Keny Perez]

Bruce,

Steve gave you a really good piece of information and starting point.

My two cents would be that you have to remember the licensing part, remember that on our FIs, only some of the ports are permanently licensed, the others will have a grace period of 120 days, so be sure you have licenses for all the ports you are going to use, otherwise, the grace period will eventually expired and you will lose functionality.

Dont forget to mark Steve's answer as helpful if it did help you

-Kenny