Server 2012 R2 Boot from SAN with ISCSI - Can't recognize

[brandonlucas]

Hi All -

I am trying to configure boot from SAN for a Windows 2012 R2 server running on a B22 M3.  I have previously configured Cisco blades running ESXi 5.5 with no issues in regards to boot from SAN.  However, I just can't get this to work.  I have seen several posts on this issue but no one has had a solution.

I am using ONE iSCSI vNIC.  Here is the output after initial boot to the ISO which I believe looks to be normal:

adapter 2/8/1 (mcp):9# iscsi_get_config

vnic iSCSI Configuration:
----------------------------
        

vnic_id: 15
          link_state: Up

       Initiator Cfg:
     initiator_state: ISCSI_INITIATOR_READY
initiator_error_code: ISCSI_BOOT_NIC_NO_ERROR
                vlan: 0
         dhcp status: false
                 IQN: iqn:ucspd01:29
             IP Addr: removed
         Subnet Mask: 255.255.255.0
             Gateway: removed

          Target Cfg:
          Target Idx: 0
               State: ISCSI_TARGET_LUN_INITIALIZE
          Prev State: ISCSI_TARGET_DISABLED
        Target Error: ISCSI_TARGET_LUN_INIT_ERROR
                 IQN: iqn.1992-04.com.emc:cx.apm00130724582.a4
             IP Addr: removed
                Port: 3260
            Boot Lun: 0
          Ping Stats: Success (3000.365ms)
          Target Idx: 1
               State: ISCSI_TARGET_LUN_INITIALIZE
          Prev State: ISCSI_TARGET_DISABLED
        Target Error: ISCSI_TARGET_LUN_INIT_ERROR
                 IQN: iqn.1992-04.com.emc:cx.apm00130724582.b4
             IP Addr: removed
                Port: 3260
            Boot Lun: 0
          Ping Stats: Success (3009.012ms)

I see the initiators logged in and registered on the SAN.  When I install the drivers, Windows just doesn't find the LUNs.  My environment is running on 2.2(1d), but I have tried the 2.2.1 drivers and the 2.2.5 drivers.  Both do not make a difference.

I have tried running the LUNLIST command but it seems to not work when using attach-mcp.  attach-fls does not provide any output.  Any ideas?

[Marcin Latosiewicz]

You're able to communicate with storage on IP level, but it looks like LUN masking or similar is not setup correctly.

Also  Ping Stats: Success (3009.012ms)  that's a HUGE latency.

[brandonlucas]

I agree it is a huge latency.  I have verified the lun masking.  The host is set to look for LUN ID 0 on boot.  The HOST LUN ID on our VNX 5500 is set to 0.  The iSCSI boot adapter uses native VLAN.  The VLAN is set correctly.  I have tried multiple drivers (although initially I think I was using the wrong ones).

I have attached screenshots of my configuration.  Please take a look and let me know if you see anything that is awry?

One thing that is curious to me: when I boot the blade, I see the connections on the SAN.  However, when I load the driver and click "refresh" from the Windows 2k12 setup, the connections on the SAN go away.  Can this be any indication other than a driver problem?

Also, why do I not get output from lunlist or lunmap?  Would I be better off opening a support ticket on this issue?

ssch56ufi01-A# connect adapter 2/8/1
adapter 2/8/1 # connect
adapter 2/8/1 (top):1# attach-mcp
adapter 2/8/1 (mcp):1# vnic
vnic id       : internal id of vnic, use for other vnic cmds
vnic name/mac : ucsm provisioned name (-n) or mac address (-m)
vnic type     : enet=ethernet, enet_pt=dynamic ethernet, fc=fcoe
vnic bb:dd.f  : host pci bus/device/function id
vnic state    : state of vnic
lif           : internal logical if id, use for other lif/vif cmds
lif state     : state of lif
vif uif       : bound uplink 0 or 1, =:primary, -:secondary, >:current
vif ucsm      : ucsm id for this vif
vif idx       : switch id for this vif
vif vlan      : default vlan for traffic
vif state     : state of vif
---------------------------------------- --------- --------------------------
                v n i c                    l i f             v i f          
id  name           type    bb:dd.f state lif state uif  ucsm   idx vlan state
--- -------------- ------- ------- ----- --- ----- --- ----- ----- ---- -----
 13 vnic_1         enet    08:00.0 INIT    2 INIT  =>0  1091    -1    0 INIT  
 14 vnic_2         enet    09:00.0 INIT    3 INIT  =>1  1092    -1    0 INIT  
 15 vnic_3         enet    0a:00.0 UP      4 UP    =>0  1093    90   44 UP    
 16 vnic_4         enet    0b:00.0 INIT    5 INIT  =>1  1094    -1    0 INIT  
 17 vnic_5         enet    0c:00.0 INIT    6 INIT  =>0  1095    -1    0 INIT  
 18 vnic_6         enet    0d:00.0 INIT    7 INIT  =>0  1096    -1    0 INIT  
 19 vnic_7         enet    0e:00.0 INIT    8 INIT  =>0  1097    -1    0 INIT  
 20 vnic_8         enet    0f:00.0 INIT    9 INIT  =>1  1098    -1    0 INIT  
adapter 2/8/1 (mcp):2# lunlist 15
% Command not found
adapter 2/8/1 (mcp):3# lunmap
% Command not found
adapter 2/8/1 (mcp):4#

[Marcin Latosiewicz]

Cursory glance at the config doesn't show anything odd. 

A couple of notes:

- loading drivers causing a disconnect - could be expected (I've only worked with UNIX and iSCSI), but the "refresh" button should fix this, or at least try to reconnect. Do you see disconnect also happen in MCP with iscsi_get_config ?

- lunlist/lunmap are specific to FC/FCoE world - thus is only present in FLS not in MCP. (At least that I'm aware of).

- multipathing, with FC and FCoE when booting initially we supply only a single path to install OS and only once MPIO is installed we're adding more path to boot policy. At least we used to, vide:

http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/gui/config/guide/2-0/b_UCSM_GUI_Configuration_Guide_2_0/b_UCSM_GUI_Configuration_Guide_2_0_chapter_011101.html#task_B2B9F86E220545659E5072387E988C27

- use the ISO for your version of firmware (for sake of clarity). 

If nothing works, open up a TAC ticket, have the guys investigate thoroughly. I'm quite surprised nobody wrote up a document about it. 

[brandonlucas]

I'm going to open up a TAC case.  I tried even eliminating the redundant paths I had set up so that I am only connecting a single path and it is not working.  I will update this thread once resolved.

[Walter Dey]

Hi Brandon

I see a problem:

Global Boot Policy

Enforce vNIC/vHBA/iSCSI Name: Yes

The vnic Name is ISCSIBoot0

Somewhere else I see Boot-A vnic which is not same !

Change the name in the Boot Policy to Boot-A.

Also; you have only one interface, therefore enable "failover flag"

Walter.

[brandonlucas]

I think that setting is right, isn't it?

The ISCSI vNIC in the service profile template is ISCSIBoot0.  The boot policy is set to enforce vNic and it labels the vnic as ISCSIBoot0.  The overlay NIC in the template for the iSCSI vNIC, however, is Boot-A.  That shouldn't matter should it?  This is the same configuration as all of my ESXi machines which have not had any issues.

[Walter Dey]

Of course you are 100% correct ! 

I didn't see all the information. And what about the missing hardware failover flag ? your interface is only connected to one fabric ? correct ?

Btw. what is your hardware configuration ?

B22-M3 with 1240 and 1280 or 1240 and VIC Port Expander or only 1240 ?

Which IOM is used ?

[brandonlucas]

Well, my desire ultimately is to use 2 vNICs for booting just like I have for my ESXi environment, but I have read in various locations online that for WK2012 boot from SAN you should only have 1 connection initially for the install, then put in your second vNIC for ISCSI boot after the fact.  My end-state should be a vNIC that is connected specifically to each fabric.  I have no idea if the failover flag would affect the initial install if that is the case.  I can't imagine it would.  What do you think?  It seems silly to think that I can only have one ISCSI boot NIC configured for the install, but at this point I'm just trying to get it to work. :P

We are using B22 M3 and it is just the VIC 1240.

[Walter Dey]

If I understand you correctly, you cannot even install W2012 R2, because after the installations of the drivers, you don't see any lun's ?

Did you install this drivers ?

Adapter Driver   = 2.4.0.8 (FNIC) / 2.4.0.15 (ENIC)
Adapter Firmware = 2.2(1)

Is your storage directly connected to UCS FI ? or via some North bound switch ?

Important question: did you disable the W2012 R2 firewall ?

[brandonlucas]

Yes, you are correct about my issue.

One interesting test I just did is to change the boot mode to UEFI.  When I do that, I actually see the exact LUN that was provisioned during BOOT.  Then again, I am back to the same problem: Windows installer will not recognize it.

The adapter/firmware across the board is 2.2(1d).  I am trying to use the following version:  DriverVer   = 11/20/2013,2.4.0.15

This was retrieved from the ucs-bxxx-drivers.2.2.1.iso package.

My storage is connected directly to a Nexus 5548 chassis environment, and my fabric interconnects are directly connected to that.  So there could possibly be some issue in the Nexus environment that Windows does not like but that ESXi has no problem with.   Here is more information about firmware levels and so forth on the Nexus:

Software

  BIOS:      version 3.6.0

  loader:    version N/A

  kickstart: version 5.2(1)N1(1)

  system:    version 5.2(1)N1(1)

  power-seq: Module 1: version v1.0

             Module 2: version v1.0

             Module 3: version v2.0

  uC:        version v1.2.0.1

  SFP uC:    Module 1: v1.0.0.0

  BIOS compile time:       05/09/2012

  kickstart image file is: bootflash:///n5000-uk9-kickstart.5.2.1.N1.1.bin

  kickstart compile time:  7/12/2012 19:00:00 [07/13/2012 02:08:13]

  system image file is:    bootflash:///n5000-uk9.5.2.1.N1.1.bin

  system compile time:     7/12/2012 19:00:00 [07/13/2012 03:23:16]

Hardware

  cisco Nexus5548 Chassis ("O2 32X10GE/Modular Universal Platform Supervisor")

  Intel(R) Xeon(R) CPU         with 8263848 kB of memory.

  Processor Board ID FOC16192WJG

[Walter Dey]

We should separate 2 problems:

1) you don't see the boot lun after installing the enic/fnic drivers

2) Boot from Lun after we have installed successfully W2012 in 1).

Therefore as long as 1) is not solved, forget Boot ? It will end the woods, that's why you see UEFI boot, which is the last possibility in the chain.

Did you disable the W2012 firewall ??

[brandonlucas]

I am not making it that far.  My comment about UEFI was simply that when I turn on UEFI in the BIOS, the BIOS is able to identify the specific LUN during the POST process (rather than just generic storage connectivity shown by Option ROM).  It shows me the specific LUN it is trying to connect to so I know the storage configuration is right.

I have never been able to install Windows on the LUN. This is an initial install problem.  The problem has always been the same:

BIOS post indicates that is successfully connecting to the LUN.  After installing the drivers in Windows setup, the initiators log out and the Windows setup process is unable to see the LUNs (they can't see them before the driver install either, but at least the SAN is showing the initiators as logged in).

There is no firewall to disallow because I am not getting the OS installed.

[Walter Dey]

Please check

http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/os-install-guides/windows/b_B-Series_Windows_Install/BSERIES-WINDOWS_chapter_011.html

[Walter Dey]

Your problem is  ISCSI_TARGET_LUN_INIT_ERROR

Have a look at

http://www.cisco.com/c/en/us/support/docs/servers-unified-computing/ucs-manager/116003-iscsi-ucs-config-00.html

did you configure any authentication ?

Is the initiator iqn ok (meaning does the storage array understand / accept it)?

you see 2 iSCSI targets ? why ?