NPV mode - Switch Mode Questions

Pete89 · ‎08-18-2011

Hello,

I have a 5596 in my lab for FCoE. I am going to set up a new host that is running Vmware ESXi4.0 and has a Emulex CNA in it.

Questions

1. The switch has to be in NPV mode correct?

2. I have other servers (non-VMs) running on ports on the 5596 already. Does changing the mode of the switch affect these hosts?

Thanks,

P

dynamoxxx · ‎08-19-2011

why do you think you need NPV mode ? I can see you enabling NPIV on the switch so you can do some fancy stuff with ESXi (present storage directly to VMs where each VM has its own WWN). When you change mode to NPV, switch becomes a "pass-thru" if you will, it does not name server or anything like that. It's a great mode for blade-enclosure switches.

@dynamoxxx

Pete89 · ‎08-19-2011

Dynamoxx,

Thanks for the great input!

This is a ROOKIE question I know. I have been tossed into the deep end of the FCoE pool and I am going through a steep learning curve.

I saw NPV mode and just thought...."ok this must mean you need it for VMs". So what you are saying is I can keep the 5596 in Switch mode and each VM will do a FLOGI and I can set up multiple vfcs on a single ethernet interface?

Sorry for the confusion.

P.

dynamoxxx · ‎08-19-2011

no problem at all. Take a look at this link

http://www.simonlong.co.uk/blog/2009/07/27/npiv-support-in-vmware-esx4/

@dynamoxxx

Pete89 · ‎08-21-2011

Great Article. Thanks a lot.

We will be using ESX4.0 on for our VMs, and I have been told by my Cisco rep that indeed I do want my switch to be in NPV mode. He writes:

The switch has to be in NPV mode correct?
- Yes, the switch should be placed in NPV mode.
I have other servers (non-VMs) running on ports on the 5596 already. Does changing the mode of the switch affect these hosts?
- Those hosts will still function even when the mode is changed to NPV mode. Note that there is a reboot when the mode is modified.
Are the two modes exclusive to each other? Can you mix port types on the same switch. If not this is a deal killer.
- Yes, the two modes are mutually exclusive, thus you cannot mix NPV/fabric switch mode, but NPIV, N-port Interface Virtualization, can be configured per-port, so you’ll be able to support both server types in your environment.

In your original post you made it seem like it was unnescessary for me to have the switch in NPV mode and that I would lose some kind of funtionality. You mentioned "named server". Could you tell me more?

Thank you!

P.

dynamoxxx · ‎08-21-2011

take a look at this link

http://www.cisco.com/en/US/products/ps5989/products_configuration_guide_chapter09186a008089f988.html

you enable NPV mode on a device that will be connected to your core switch that has NPIV enabled, for example:

MDS 9124e ---- MDS 9513

My 9124e is a blade enclosure switch, and i have multiple blade enclosures. I wanted to keep my fabric simple so i enabled NPIV feature on my 9513 (that is non-disruptive). Then i enabled NPV mode on my 9124e switch, when i did that the switch reboots itself and becomes this simple "pass-through" kind of device. Is your 5596 going to be connected to other fiber channel directory class switches ? If your 5596 is your core switch, i don't think you want to enable NPV mode ..but simply enable NPIV feature.

@dynamoxxx

Pete89 · ‎08-21-2011

Thanks for answering my post. Here is a drawing of my setup. This is what we eventually want to do:

S for Server with a two port CNA

Two Top of Rack 2232s in each rack

Red Circles are Vitual Port Channels

From what you said I would say the 5ks are my core switches as far as FCoE goes. You say "

i don't think you want to enable NPV mode ..but simply enable NPIV feature"

Is the NPIV a feature part of the ESX?

Thanks again,

P.

Pete89 · ‎08-24-2011

I know that answering your own posts might be seen as a bit strange but for the poor soul, who like me was asked to make FCoE work without any training or help:

When Dynamoxx told me I should just enable NPIV and that putting the 5k into NVP mode was not necessary I was confused. I didnt understand the difference between NPV and NPIV. Then I read a few articles:

http://www.dasblinkenlichten.com/?p=292

http://blog.scottlowe.org/2009/11/27/understanding-npiv-and-npv/

So, for my setup where I have a ESXi host servering up a bunch of VMs enabling NPIV on the 5k is what I want to do:

This link explains how to do that:

http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/configuration/guide/cli_rel_4_0_1a/FCintf.html#wpxref87357

Hope this helps somebody someday

dynamoxxx · ‎08-24-2011

do you plan on provisioning block storage to individual VMs ? If you simply want to provision some storage from your array to ESX hosts, create datastores and build VMs on top of that ..you don't need to enabled NPIV and complicate things. Just zone your ESX hosts to the array, present some LUNs, create VMFS datastores and build VMs. If you have the need to present "native" storage, you can use RDM functionality.

@dynamoxxx

Kris Vandecruys · ‎08-24-2011

Hi pener1963,

Do you have a pure FC switch (like a Cisco MDS switch) in there and is it connected to a FC port in your 5596?

In that case, NPV would make sense. You would do FCoE to the n5k, which would turn your frames into pure FC and then forward them on through the uplinks to the MDS. The advantange here is that your n5k would not use a fc domain.

Using NPV only makes sense if there are some pure FC ports involved here. It's a (over)simplification of what is happening but as a rule of thumb it makes your design decisions easier.

hth,

Kris

Pete89 · ‎08-25-2011

Gentlemen,

Many thanks for the feedback.

>>Do you plan on provisioning block storage to individual VMs ?

Good question. This popped up yesterday when my VMWare guy told me he would only be presenting the 5596 one wwn to the switch and that VMWare "would take care of the rest on the backend". That got me thinking. Is that secure enough? I know this is out of scope for the question at hand, but how sure can I be that VM hosts cant get into each others datastores?

>>Do you have a pure FC switch (like a Cisco MDS switch) in there and is it connected to a FC port in your 5596?

No. We will eventually have what is represented in the diagram I pinned up on the 21 of August in this thread. We do have an MDS switch in house but I believe in will stay in the Enginneering environment, however if the possiblity of the MDS ever getting jacked into that 5k exists what would it hurt to have it in NVP mode? It wouldnt change the config I have on it now right? Having it in NPV mode buys me not having to reboot that 5596 someday.

Thanks again,

P.

dynamoxxx · ‎08-27-2011

Good question. This popped up yesterday when my VMWare guy told me he would only be presenting the 5596 one wwn to the switch and that VMWare "would take care of the rest on the backend". That got me thinking. Is that secure enough? I know this is out of scope for the question at hand, but how sure can I be that VM hosts cant get into each others datastores?

that is a function of vSphere how it manages who has access to what. I am not understanding reply from your VM guy though. I'll give you an example what you would do with a Clariion. On Clariion you have two storage processors, each SP has multiple ports. So when it's time to do your zoning you would do it like this:

Fabric A:

zone1 - host-HBA0

Clariion-port-SPA0

Clariion-port-SPB1

Fabric B:

zone2 - host-HBA1

Clariion-port-SPB0

Clariion-port-SPA1

i don't know how NetApp lays out their ports on the filers/controllers but it's probably very similar recommendation.

No. We will eventually have what is represented in the diagram I pinned up on the 21 of August in this thread. We do have an MDS switch in house but I believe in will stay in the Enginneering environment, however if the possiblity of the MDS ever getting jacked into that 5k exists what would it hurt to have it in NVP mode? It wouldnt change the config I have on it now right? Having it in NPV mode buys me not having to reboot that 5596 someday.

Thanks again,

P.

If you don't plan on connecting to any FC switches right away, then you don't want to put 5596 into NPV mode, because it will not be a "full" blown FC switch at that point. You will not be able to do any zoning or anything like that. So if 5596 is the only switch you have right now, leave it as it is. If you later on decide to connect 5596 to another MDS or Brocade ..whatever, you don't have to enable NPV ..you can simply connect it to MDS and setup a portchannel between the two. You don't need downtime, you don't need to enable NPV (it's disruptive). In my opinion NPV is only useful when you have huge fabrics and you want to simplify the topology.

@dynamoxxx

Kris Vandecruys · ‎08-30-2011

There's basically 2 ways to setup storage on virtual infrastructures:

1. Assign a lun to your ESX, create vmdk files on that LUN and attach them to th virtual machines which will see them as local disks without any idea of the SAN below

2. Assign virtual WWNs to your virtual machines, and zone/map/mask these WWNs with the LUNs and ports on your storage array. This requires NPIV at one level or another

Either way will isolate data in one VM from other VMs, but the authority for this separation lies with different parties. in Scenario 1, ESX should do this, in the second, the Storage Network takes care of this. And both of them rely on the admin not to cross configure the LUNs ;-)