MDS 9124 connecting to two 9124e in HP c7000 Bladesystem

[nickhesson]

Hi all,

We got two MDS 9124e switches for our new HP c7000 Blade system. We currently have one MDS 9124 switch for our SAN environment now. We are thinking of connecting both 9124e switches via 2-port port channel to the main 9124 switch. I have a host of questions. First, all vsans are manually configured on all switches (meaning there is no VTP like function for VSANS)? I have read other threads talking about Domain ID's, and related configuration changes; do I have to be concerned with this? Through HP documentation I have read that these switches can be configuring for redundancy, how? And is this an active/active or active/passive configuration?

So I believe my tasks are configure the vsans, create the port channel, configure the port ch. interface for ISL and configure trunking (?), cable the switches and the assign the FC interface to their VSAN. Is there anything else I'm missing or should be concerned with?

As you can see, I'm pretty new with SAN configuration.

Thanks for all your time and help.

Nick

[Michael Brown]

Hello Nick and welcome to the world of SAN configuration. You are correct, there is no feature like VTP for VSANs. They are configured per switch. Fabric Manager has the ability to configure a VSAN across multiple switches. As for domain IDs, they are assigned per VSAN per swtich. For example, for VSAN 100, if you have 2 switches connected via and ISL and both are in VSAN 100, each switch would have it's own domain ID. Normally when switches connect to each other they sort this out on their own so that each one is unique. Ad far as port channels, yes you can create a port channel between the 9124e and the 9124. The term trunking means the ISL or port channel is carrying multiple VSANs. Think of a port channel as comparative to an etherchannel, and a trunk as comparative to an 802.1Q trunk. Once you have the VSANs configured, the port channel created, the ports in the correct VSAN, the next step would be to perform zoning. That is done per VSAN, and it is how you permit the blade servers to access the desired storage ports.

Hope this helps,

Mike

[stephen2615]

Nick,

There are so many variables here that a simple response might not help. As you have only one 9124, this limits your redundancy a lot. They are a good switch but not a really robust business critical switch by themselves. To gain a redundant solution, you should have another 9124 switch.

Your current 9124 should have at least two VSANs for some sort of redundancy. Almost all SAN equipment is dual pathed. Your blades will have at least a dual port mezzanine card in them. Your storage array will also have at least two paths configured. Lets say port 0a and 1a. Port 0 on your server should go to 0a on your array and so on. This is all done via your switch.

Enterprise class arrays are active active which means they move data down both paths in parallel. They tend to be very expensive and smaller arrays don't offer this type of feature. They are normally active passive. This means that if you have a problem with your active path, the passive one will take over.

Having the two 9124e in your enclosure is normally to have one path go to one switch and the other to the other switch. Switch could be a VSAN in this case. Its a standard for the industry.

Each of those enclosures can have 16 blades. If you are going to use all of them for SAN, a 2 port port channel could be stretching the solution a bit. It will only offer 8 Gbps max. Your blades each have a 4 Gbps HBA port. 16 x 4 = 68 Gpbs.... If you are going to be using VMware, you might need bandwidth.

I tend to allow for around 1 Gbps bandwidth for each server that does not do a lot of work. Heavy usage VMware servers will want 2 Gbps or better.

There are some interesting things happening right now with the MDS so I would get some advice on which way to go before making a decision like you have now.

I would consider having a look at NPV in this instance instead of getting into a complex solution if no one has the skills to support it. NPV takes away the need to worry about whats in the blade enclosure.

Stephen

[nickhesson]

Hi Stephen,

I do thank you for your reply. But sorry to say, I don't believe you understood my questions.

I know having only one 9124 for our main switch is does not offer any redundancy. We do have multi VSANS, that is why I was asking about the configuration of Vsans across all switches. I know what act/act and act/pass failover means. I wanted to know if the two 9124e switches inside the chassis had some type of internal link or redundant path that I can create one virtual switch out of both of them. (But never mind I got that question answered of a HP forum).

Thanks for your concern about possible bandwidth issues, but we will have two port-channel links coming from both switches giving us a total of 16 Gb.

To tell the truth, mikbrown answered most of thoughts. The only thing I am still unclear on is the use of Domain ID and the whole Fclogi process after reading mikbrown post. But I believe I get need to do some good reading on the web..

Again, thanks for help.

Nick

[Michael Brown]

Nick,

The domain ID is assigned per VSAN, per switch. So if you have 3 VSANs in a single switch, each VSAN can have the same Domain ID, or unique Domain IDs. If you trunk this switch to another switch, and it also has the same 3 VSANs, that switch must have different Domain IDs for the VSAN. IE: Switch 1 can have domains 2, 3 and 4 for VSAN a, b, and c respectively. When you trunk the VSANs to a second switch, the domain in the second switch for vsan a can not be 2. It must be some other ID between 01 and EF, Likewise for vsan b, the new switch can not use Domain ID 3. The default configuration lets the switches change the domain ID as needed to permit the VSAN to merge. You can statically set the domain ID in each switch, for each VSAN, but those IDs must be unique across the VSAN. Normally you dont have to worry about the domain IDs. The domain ID is used to generate the first byte of the FCID, which is assigned at FLOGI time. So if your server is attached to VSAN 100, with domain ID hex 55, the server will be assigned an FCID 55xxxx. If you move the server to a port in VSAN 200, with domain ID hex 22, it will FLOGI and be assigned an FCID like 22xxxx. Normally the FCID is not something to concern yourself with, but with some AIX and HP-UX operating systems, you really don't want the FCID to change. With those operating systems you should configure the domain ID as static, which means it will not change.

Hope this helps,

Mike

[nickhesson]

Thanks mike, I pretty much got that in your first post. But you confuse me a bit when you say:

"So if you have 3 VSANs in a single switch, each VSAN can have the same Domain ID, or unique Domain IDs. If you trunk this switch to another switch, and it also has the same 3 VSANs, that switch must have different Domain IDs for the VSAN."

e.g. If you only have two storage targets (two HP EVA's) one configured for VSAN 2 and the other configured for VSAN 40 both connected to a single switch, and then trunk that switch to another new switch, The new switch HAS to be configured for both VSANS 2 and 40 to be able to see both storage targets? I'm I correct? If so, then the switches will on their own will dynamical assign the correct Domain ID's?

In your sentence above you make it sound like you can have different VSANs on the different switches. But if you have, let's say one storage on your SAN all switch must have the same VSAN, and all host must be in that same VSAN (ie VSANs 40) to be able to see that storage?

Thanks for your help and getting my head on track. Thanks again,

Nick

[Michael Brown]

Nick,

I to clarify.

MDS1 can have VSAN 2 with domain ID 20. MDS1 can also have VSAN 40 with domain ID 20.

If MDS1 is trunked to MDS2, and MDS2 has VSAN 2, the domain ID on MDS2 for VSAN 2 can not be 20. Same theory for VSAN 40 on MDS2.

You are correct, that if you have a single storage device in VSAN 40, the hosts wishing to use that storage must be in VSAN 40 on that same switch, or another MDS that has VSAN 40 trunked to this switch.

The exception would be to use IVR (inter vsan routing) where the MDS (not supported by the 9124) can route between VSANs.

- Mike

[Chhabrag87]

Hi,

Can anyone suggest rather than creating ISL between Cisco MDS9706 and MDS9124e (IBM Blade Switch), is there is an option to configure access gateway like we do in Brocade where we don't want to follow ISL.

Second, is it mandatory to configure port channel while doing ISL between cisco switches ?

Thanks

Gaurav

[Michael Brown]

If you connect 2 FC switches, that link is an ISL.  No way around it.  Now regarding using a port-channel, that is not mandatory.  You can have multiple links between 2 FC switches and let them operate in dependently, or bundle them into a single logical link A.K.A a port-channel.  FSPF can load balance across 16 links between 2 FC switches (assuming they are all the same speed). 

Onto the second part of your question.  Yes, the 9706 can act as gateway.  It is know in Cisco as feature NPIV, while the 9124e operates as NPV switch.  This mode uses the port modes of NP on 9124e side, and F-port on the 9706 side. (I still consider this as an ISL since it connects 2 FC switches).  If the 9124e is not currently running in NPIV mode, to make the change, it will need to be reloaded.   The change from non-NPV to NPV mode is disruptive.  Enabling NPIV feature on the 9706 is not disruptive.

Here is a link on NPIV and NPV operation that hopefully helps.

http://www.cisco.com/c/en/us/support/docs/storage-networking/mds-9000-series-multilayer-switches/200127-N-Port-Identifier-Virtualization-NPIV.html

[Chhabrag87]

Thank you for this information.

Have enabled NPV on blade 9124 switches and created port channel on external ports which are connecting to MDS9706. All server internal port mapping has been enabled to port-channel external port.

Now question comes to my mind is do i need to create port channel at MDS9706 side where 9124 uplink ports are connected ? if yes then how they will communicate in different VSAN as MDS9706 ports are in VSAN 301 and 9124 ports are VSAN 401 ?

[Michael Brown]

The feature on the 9706 is called f-port-channel and here is an example of how to configure it.  I would put the NP ports on 9214e and F ports in 9706 in VSAN 1, and then enable the port-channel and f-port-channel to be a trunk, and add the VSAN that the hosts in the 9124e will need to be in, on the trunk.

feature fport-channel-trunk
 feature npiv
 
 interface port-channel 1
   channel mode active
   switchport mode F
   switchport trunk allowed vsan 1
   switchport trunk allowed vsan add 20
   switchport rate-mode dedicated
 
 interface fc2/2
   switchport rate-mode dedicated
   switchport mode F
   channel-group 1 force
   no shutdown
 
 interface fc4/8
   switchport rate-mode dedicated
   switchport mode F
   channel-group 1 force
   no shutdown

[Chhabrag87]

Thank you Michael for detailed information.

Have doubt here.

In 9124e switch has created port channel 01 with 4 ext0-17 ports and added that port channel in VSAN 1 and all server ports bay1-14 are also in VSAN 1 and done external traffic map to port channel 01.

In MDS9706 where these ext0-17 uplink ports are connected are in VSAN 100 which is production VSAN and storage ports in this VSAN need to be zoned with servers in Blade centre connected to Bay1-14.

Query

1. Shall I create VSAN 1 in 9706 and add up-link ports and port channel into that VSAN 1 and both switch VSAN 1 will establish connection and can communicate ?

2. what will be next how i can enable zoning between ports in VSAN 100 and blade ports which will come online in VSAN 1

If my understanding is wrong somewhere please do correct.

[anantha.dommeti]

"I have read that these switches can be configuring for redundancy, how? And is this an active/active or active/passive configuration?"

If you have two Core switches(9124 standalone in your case), you can create two separate fabrics for higher redundancy...Create a fabric from each of your Blade Chassis 9124e and Standalone 9124 and you get the best availabilty(I hope you have some kind of multipathing software in your hosts)

It's an active-active configuration.