Introduction
Fibre-channel SAN is a great design choice for UCS because it enables high-performance, reliable, stateless computing.
High-performance because fibre-channel is the best performing block-storage device, and stateless because Boot-from-SAN is an ideal way to abstract the operating system (e.g. ESX4) from the physical blade.
However, Fibre-Channel requires good planning and preparation before implementation - that's where this document comes in.
This document explains how to build UCS on a fibre-channel SAN by covering the following topics:
- Build Procedure
- Bill of Materials
- Cabling
- Configure SAN Array
- Configure SAN Switches
- Configure UCS
- Boot from SAN
Intended Audience
This document shoudl be use collaboratively between architects, engineers and administrators to design, build and operate UCS on a fibre channel SAN.
Audience
Steve Chambers, Cisco Unified Computing Practice.
Resources
- You can find this doc on the web at https://supportforums.cisco.com/docs/DOC-6223
Disclaimer
This is not a comprehensive guide and your mileage may vary. This is a sample to show the outline of a successful practice.
1. Build Procedure
The following procedure is a proven way of building a successful UCS-on-FC-SAN system.
This procedure assumes existing SAN Array and SAN Switch devices (most common scenario). You may need to extend this procedure to include the build of either/both Array and Switch devices, but the procedure remains in the same order.
- Check all inventory items on Bill-of-Materials and check integration compatibility between Arrays and Switches, Arrays and CNAs, Switches and CNAs, OS and Array and OS and CNAs.
- Configure physical systems (rack, cabling) as per best practice.- covering Array, Switches and UCS.
- Logical configuration of SAN Array - storage processors.
- Logical configuration of SAN Switch - zonesets, zones and VSANs
- Logical configuration of UCS - uplinks, blade CNAs, service profiles
- Connect the dots - Boot from SAN and test.
2. Bill of Materials
This is the list of equipment required:
- If your SAN Array supports FC then it will work with UCS (not an official support statement).
- If your SAN Switches support basic FC then they will work with UCS. If they support VSANs (Cisco MDS only) then you can configure each port to be on a different VSAN.
- You need a UCS Gigabit Expansion Module (GEM) for Fibre-channel, which gives you 6 x 4Gbps.
3. Cabling
Cabling is easy...to get wrong! While there are a few different ways of doing this, there is one best practice and this is it:
- Assuming you have a Tier 1 array like an EMC DMX-4 with two storage processors exposed per VSAN (or fabric)
- Assuming you have separate VSANs (or physical fabrics).
- Each UCS 6120 GEM should have two ports connecting to the same VSAN (or fabric) - two ports provide high-availability for the connections with dynamic repinning of traffic should one of the connections be broken.
- There are no connections required from blades or fabric extenders - it's all Fibre Channel over Ethernet (FCoE) using existing cabling.
4. Configure SAN Array
Assuming your SAN is a Tier-1 array and has two storage processors per VSAN (or fabric) then each SAN Switch should be able to see two Array Storage Processor port WWNs - there is no cross-over here, unlike LAN. Each pair of storage processors connect to distinct VSANs/Fabrics.
A LUN can be presented to all storage processors on a Tier 1 array because they are normally active/active arrays (meaning LUNs are available on all storage processors).
If you have a Tier-2 array or lower then you will have to consider the assigning of LUNs to storage processors if they are active/passive arrays.
You can't create a LUN and mask/map it to a blade yet because the UCS isn't configured. At this point, make a note of each fabric's storage array processor port world wide names (deep breath):
VSAN/Fabric A
- SP A - pWWN 50:00:01:02:03:04:05:06
- SP B - pWWN 50:00:01:02:03:04:06:07
VSAN/Fabric B
- SP A - pWWN 50:10:01:02:03:04:05:06
- SP B - pWWN 50:10:01:02:03:04:05:06
Why do you need these addresses? Because you will be telling a blade to Boot from SAN and for that you need to tell the Initiator (blade HBA) the target pWWN and LUN ID. More of that later.
5. Configure SAN Switches
The SAN switches are critical to UCS on Fibre Channel. Cisco provide MDS switches that provide functionality like VSANs, NPV and Zones.
VSANs are not essential to to connect UCS to an FC SAN, but VSANs are essential to have an efficient SAN. Without VSANs, you have to have separate physical SAN switches for every FC fabric network. With VSANs you can collapse them onto few devices like Nexus 5K/7K and MDS 9K devices.
You or your storage experts may have reasons against VSANs - please let me (the author) know what they are.
NPV is essential for UCS, because UCS runs in NPIV mode and needs an NPV/NPIV enabled switch to talk to. NPIV enables UCS to present multiple virtual HBAs over one physical HBAs. Think of virtual MAC addresses on ESX out of one physical NIC. Same thing.
Whereas on the SAN Array you can mask/'map LUNs to specific initiators (blades), the Zones of the fabric/VSAN switch enable you to match initiators (blade HBA pWWNs) to targets (array storage processors pWWNs). The goal is to create "single-initiator zones" where each vHBA (ie. the actual blade HBA, which might be virtual and _not_ the CNA pWWN) is in a zone of its own with the target SP pWWNs on that fabric:
Single Initiator Zone 1 contains:
- Service Profile vHBA 1 pWWN (on Fabric A/VSAN100)
- Storage Processor A pWWN (on Fabric A/VSAN100)
- Storage Processor B pWWN (on Fabric A/VSAN100)
6. Configure UCS
If you've got a well configured SAN Array, and well configured SAN Switches, then you are ready to connect up UCS.
Assuming you've cabled both 6120s to _different_ VSANs/Fabrics, then that's all you need to do for the Fabric Interconnects.
The next step is to create a Service Profile that will configure a blade to connect to FC via vHBAs.
7. Boot from SAN
