How to deploy UCS Nexus 5000 & SAN storage

[sjahirud]

Hi ,

I dont have good knowledge on nexus 5000 , but after going through lot of doc I am trying to setup UCS with nexus 5000 and san storage.

My objective is to boot server from san storage.

My setup :

--------

UCS------>Nexus5000---------SAN storage

As per the document when I am configuring UCS for FCOE Uplink , there is no option to set the VSAN id.

I have created the VSAN 500 and then making one of the port in FI for FCOE uplink.Screenshot attached.

Plz let me know FI should be in endhost mode or switching mode..?I am running ucs 2.2(3d) version.

What kind of SFP should I use to connect from UCS & Nexus.

THanx & Regards

Shaikh

[Walter Dey · Accepted solution]

Hi Shaikh

Which protocol is your storage array using ? FC or FCoE ?

[sjahirud]

Hi Walter ,

Storage array is using FC.

I need to clear few things here.

             (FCoE)                                                  (FC)

UCS-----------------------------------Nexus5000-------------------------SAN Storage

1.FI should be in EHM(NPV).I selectthe EHM and FI rebooted.

2.But in FI I can see NPIV is enabled.After FI comes up still NPIV is enabled

agt-lab-A(nxos)# sh npv status

npiv is enabled

disruptive load balancing is disabled

External Interfaces:
====================
  Interface: vfc1054, State: Down
  Interface: vfc1055, State: Down
  Interface: vfc1062, State: Waiting For VSAN Up
  Interface: vfc1063, State: Waiting For VSAN Up

  Number of External Interfaces: 4

Server Interfaces:
==================
  Interface: vfc892, VSAN: 1000, State: Waiting for External Interface
  Interface: vfc896, VSAN: 1000, State: Waiting for External Interface
  Interface: vfc900, VSAN: 1000, State: Waiting for External Interface
  Interface: vfc1020, VSAN: 1000, State: Waiting for External Interface
  Interface: vfc1028, VSAN: 1000, State: Waiting for External Interface
  Interface: vfc1032, VSAN: 1000, State: Waiting for External Interface
  Interface: vfc1036, VSAN: 1000, State: Waiting for External Interface

  Number of Server Interfaces: 7

agt-lab-A(nxos)#
agt-lab-A(nxos)#

agt-lab-A# connect nxos
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2014, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php
agt-lab-A(nxos)# sh running-config | inc feature
feature fcoe
install feature-set virtualization
feature-set virtualization
no feature telnet
feature tacacs+
feature private-vlan
feature port-security
feature udld
feature interface-vlan
feature netflow
feature lacp
feature vmfex
feature lldp
feature fex
feature network-segmentation-manager
  rule 3 permit read-write feature sam-pn-maintenance
  rule 2 permit read-write feature sam-pn-policy
  rule 1 permit read-write feature sam-pn-equipment
  rule 1 permit read-write feature sam-power-mgmt
  rule 3 permit read-write feature sam-ls-security-policy
  rule 2 permit read-write feature sam-ls-security
  rule 1 permit read-write feature sam-pn-security
  rule 3 permit read-write feature sam-ls-compute
  rule 2 permit read-write feature sam-ls-server-oper
  rule 1 permit read-write feature sam-ls-server-policy
  rule 6 permit read-write feature sam-ls-server-oper
  rule 5 permit read-write feature sam-ls-ext-access
  rule 4 permit read-write feature sam-ls-server-policy
  rule 3 permit read-write feature sam-ls-config-policy
  rule 2 permit read-write feature sam-ls-server
  rule 1 permit read-write feature sam-ls-config
  rule 2 permit read-write feature sam-fault
  rule 1 permit read-write feature sam-operations
  rule 1 permit read-write feature sam-read-only
  rule 12 permit read-write feature sam-ls-qos-policy
  rule 11 permit read-write feature sam-ls-network-policy
  rule 10 permit read-write feature sam-ls-qos
  rule 9 permit read-write feature sam-ls-network
  rule 8 permit read-write feature sam-ext-lan-qos
  rule 7 permit read-write feature sam-ext-lan-security
  rule 6 permit read-write feature sam-ext-lan-policy
  rule 5 permit read-write feature sam-ext-lan-config
  rule 4 permit read-write feature sam-pod-qos
  rule 3 permit read-write feature sam-pod-security
  rule 2 permit read-write feature sam-pod-policy
  rule 1 permit read-write feature sam-pod-config
  rule 6 permit read-write feature sam-ls-storage-policy
  rule 5 permit read-write feature sam-ls-storage
  rule 4 permit read-write feature sam-ext-san-qos
  rule 3 permit read-write feature sam-ext-san-security
  rule 2 permit read-write feature sam-ext-san-policy
  rule 1 permit read-write feature sam-ext-san-config
  rule 1 permit read-write feature sam-admin
  rule 1 permit read-write feature sam-aaa
no snmp-server enable traps license notify-no-license-for-feature
feature npv
feature npiv
agt-lab-A(nxos)#

But FI should be in EHM and nexus 5000 in NPIV mode.

I want to use FCoE between UCS & Nexus and FC between nexus & Storage array.

When I am selecting the FCoE uplink it should give me the option for SAN.

In the Properties area, specify the VSAN in the VSAN field.But there is no option to configure VSAN in FCoE uplink properties.

Please help me.

I am running 2.2(3d) version

I am running ucs 2.2(3d) version

I am running ucs 2.2(3d) version

I am running ucs 2.2(3d) version

[Walter Dey]

Because you use FC on the N5k anyway, I would propose that you use FC between FI and N5k; this makes your setup a little bit easier. This doesn't require any additional license.

[sjahirud]

Hi Walter ,

THanx a lot..

Can you plz provide some info how can I use FC between FI & N5k.

What need to be done on FI & N5k

Regards

Shaikh

[Walter Dey]

Hi Shaikh

- On the FI, you simply have to configure FC EHM (Ethernet in EHM as well); EHM for FC and Ethernet are the default setting. No other configuration is necessary.

- Then you create VSAN for fabric A and B (separate ones, e.g. 10 and 11).

- You create FC ports on the FI (see eg. https://www.youtube.com/watch?v=QbaTfBCCZzE)

- You cable up one port from each FI (or more) and connect it to a FC port on the N5k

- Put this FC ports on the FI in VSAN 10 resp. 11

- setup N5k for FC like I mentioned above (enable feature NPIV and F port channel trunking)

- On N5k: create vsan 10 for fabric A and vsan 11 for fabric B

- connect your storage

---------------------

check "show flogi database vsan 10 resp 11" which shows you UCS FI and storage

then you need to do the zoning

[sjahirud]

THanx Walter,

Zoning should be on n5k and local zoing should be disabled on FI.Is it so..?

THanx & Regards

Shaikh

[Walter Dey]

Correct ! Zoning is done on N5k.

Local zoning (on UCS FI) is only done, if storage is directly connected to UCS FI.

In you case, you have a SAN (N5k !) and storage is connected to the SAN.

have a look at

http://www.alcatron.net/Cisco%20Live%202013%20Melbourne/Cisco%20Live%20Content/Data%20Centre%20And%20Virtualisation/BRKCOM-2002%20%20UCS%20Supported%20Storage%20Architectures%20and%20Best%20Practices%20with%20Storage.pdf

which should answer most of your questions.

Walter.

[sjahirud]

Hi walter ,

One doubt ..what sfp sould I use..?

DS-SFP-FC4G-SW or SFP-10G-SR

Plz clarify

[Walter Dey]

For the configured FC ports on the FI as well as N5k, you need to use FC transceivers, e.g. DS-SFP-FC-FC4G-SW

[sjahirud]

Hi walter ,

Thanx for your support , but little bit confused about "F port channel trunking"

Can you plz tel me how to do this F port channel trunking

Regards

Shaikh

[Walter Dey]

This is only used, if you would create a FC port channel between UCS FI and N5k.

see e.g. https://www.youtube.com/watch?v=PpzKPguRTXc

[sjahirud]

Hi walter,

Do I need to create vfc in nexus..?

[Walter Dey]

Hi

No ! FCoE runs between blade and FI; FI Northbound to N5k is native FC; same from N5k to Storage.

Walter.

[sjahirud]

Hi Walter ,

As per my previous setup , can you please help me on

             (FCoE)                                                  (FC)

UCS-----------------------------------Nexus5000-------------------------SAN Storage

In fabric interconnect I am not able to see the san config option once make it to FCoE port.

Regards

Shaikh