UCS B200 M5 - ESXi/HBA FNIC issue

[NMDSIO73]

Hello,

I'm setting up a new UCS chassis with some B200M5 blades VIC1440.

Connecting these blades for boot on san on VNX backend worked with no issue and setup of esxi 6.5 worked perfectly.

Just facing right now an issue for connecting those blades to VPLEX SAN. I've spend lot of time searching for an answer on my issue with no succes. Might be related to Cisco directly so I'm trying here ..... here is my problem :

UCS 4.0.4c bundle installed

B200M5 ESXi 6.5 Custom Cisco installed with boot on san.

When scannign HBA to connect VPLEX here is what I got :

2019-08-16T14:34:30.233Z cpu8:66287)ALERT: LinScsi: SCSILinuxProcessCompletions:855: Error BytesXferred > Requested Length but HOST_OK/DEVICE_GOOD!vmhba = vmhba2, Driver Name = fnic, Requested length = 66, Resid = 512
2019-08-16T14:34:30.233Z cpu8:66287)ALERT: LinScsi: SCSILinuxProcessCompletions:855: Error BytesXferred > Requested Length but HOST_OK/DEVICE_GOOD!vmhba = vmhba2, Driver Name = fnic, Requested length = 66, Resid = 512
2019-08-16T14:34:30.233Z cpu8:66287)ALERT: LinScsi: SCSILinuxProcessCompletions:855: Error BytesXferred > Requested Length but HOST_OK/DEVICE_GOOD!vmhba = vmhba2, Driver Name = fnic, Requested length = 66, Resid = 512
2019-08-16T14:34:30.233Z cpu8:66287)ALERT: LinScsi: SCSILinuxProcessCompletions:855: Error BytesXferred > Requested Length but HOST_OK/DEVICE_GOOD!vmhba = vmhba2, Driver Name = fnic, Requested length = 66, Resid = 512
2019-08-16T14:34:30.233Z cpu16:65584)ALERT: LinScsi: SCSILinuxProcessCompletions:855: Error BytesXferred > Requested Length but HOST_OK/DEVICE_GOOD!vmhba = vmhba1, Driver Name = fnic, Requested length = 66, Resid = 512
2019-08-16T14:34:30.234Z cpu16:65584)ALERT: LinScsi: SCSILinuxProcessCompletions:855: Error BytesXferred > Requested Length but HOST_OK/DEVICE_GOOD!vmhba = vmhba1, Driver Name = fnic, Requested length = 66, Resid = 512
2019-08-16T14:34:30.234Z cpu16:65584)ALERT: LinScsi: SCSILinuxProcessCompletions:855: Error BytesXferred > Requested Length but HOST_OK/DEVICE_GOOD!vmhba = vmhba1, Driver Name = fnic, Requested length = 66, Resid = 512
2019-08-16T14:34:30.234Z cpu16:65584)ALERT: LinScsi: SCSILinuxProcessCompletions:855: Error BytesXferred > Requested Length but HOST_OK/DEVICE_GOOD!vmhba = vmhba1, Driver Name = fnic, Requested length = 66, Resid = 512

Talking about the fnic driver, which is currently running to last last version present in the compatibility table :

vmkload_mod module information
input file: /usr/lib/vmware/vmkmod/fnic
Version: Version 1.6.0.50, Build: 2494585, Interface: 9.2 Built on: Mar 14 2019
Build Type: release
License: GPLv2

VIC1440 firmware is currently set to 5.0(3c).

Does anyone already faced this issue ? Is it FNIC/VIC related ?

Many thanks,

Best regards

[Kirk J]

So you have 2 storage array systems connected to FC switches above the FIs (VNX, VPLEX) connected to the same blades?

Are you using same vHBA, or do you have one set of vHBAs defined for each storage system?

Different Vsans used?

Are you seeing flogi for wwpn from vHBA you are expecting to to connect to VPLEX, on the FIs (nxos#show npv flogi) ?

What FC switches are above the FI?  If they are Cisco you should be able to issue command # show zoneset active vsan xx

Above command should confirm if you are seeing your wwpn storage targets and wwpn initiators active in the correct zones/vsans.

Kirk...

[NMDSIO73]

Hi Kirk,

So you have 2 storage array systems connected to FC switches above the FIs (VNX, VPLEX) connected to the same blades?

- Yes that's correct, with one working perfectly (VNX)

Are you using same vHBA, or do you have one set of vHBAs defined for each storage system?

- Same vHBA (x2, one per vsan/FI) for both storage arrays

Different Vsans used?

- 2 Vsans used, one per vHBA/FI 

Are you seeing flogi for wwpn from vHBA you are expecting to to connect to VPLEX, on the FIs (nxos#show npv flogi) ?

- I'm seeing the correct WWPN per each vHBA on the right FI with the correct Vsan

What FC switches are above the FI?  If they are Cisco you should be able to issue command # show zoneset active vsan xx

- 2xNexus 5672UP are above the 2xFI (One VSAN per vHBA-FI-Nexus). FcOE is used to interconnect FI and Nexus. Zoning is activated and correct. Triple checked that last week.

Still working with storage array VNX but not VPLEX. 

Many thanks for you help,

Best regards

NB : just in case, forgot to mention that the VPLEX is running in production with B200M2-M3 and B22M3 (several different VICs) blades running on esxi 5.5 1892794 cisco custom.

[Kirk J]

Please post the output from show zoneset active vsan XX that is relevant to your vplex vsan.

Thanks,

Kirk...

[NMDSIO73]

Here it is :

First Nexus :

zone name Z_ESXi-PR-A_VPLEX-1 vsan 411
* fcid 0x5d01a1 [pwwn 20:00:00:25:b5:10:0c:04] [ESXi-PR-A]
* fcid 0x5d0080 [pwwn 50:00:14:42:c0:39:78:01] [VPLEX-1-A1]
* fcid 0x5d00a0 [pwwn 50:00:14:42:d0:39:78:01] [VPLEX-1-B1]

Second Nexus :

zone name Z_ESXi-PR-B_VPLEX-1 vsan 422
* fcid 0xc401a1 [pwwn 20:00:00:25:b5:11:0c:04] [ESXi-PR-B]
* fcid 0x9e0006 [pwwn 50:00:14:42:c0:39:78:00] [VPLEX-1-A0]
* fcid 0x9e0007 [pwwn 50:00:14:42:d0:39:78:00] [VPLEX-1-B0]

Thanks

[NMDSIO73]

I have installed esxi 5.5 U3 cisco custom on that blade instead of 6.5, same result.

I have then migrate the profile to an other blade but a B200M3 with VIC1340 and fnic Version 1.6.0.37 and suprise ! I can see my VPLEX with no errors ....

So :

B200M5 VIC 1440 (5.0(3c)) ESXI 5.5/6.5 Custom Cisco with fnic 1.6.0.50 not working

B200M3 VIC 1340 (4.2(2b)) ESXI 5.5 Custom Cisco with fnic 1.6.0.37 working

Same profile used for both as the esxi install because of boot on san....

Does cisco have a clue on that ? I'm stuck with my B200M5 right now ....

Many thanks,

Best regards

[Wes Austin · Accepted solution]

Do you have a case opened with TAC? It would probably be easier to just take a look at everything over a Webex as a whole.

[NMDSIO73]

We are going to open one.

Just to confirm, switch VIC 1440 to VIC 1340 on the B200M5 works with vplex.

Thanks for you help.

[NMDSIO73]

Hi Kirk,

So you have 2 storage array systems connected to FC switches above the FIs (VNX, VPLEX) connected to the same blades?

- Yesy that's right with VNX working

Are you using same vHBA, or do you have one set of vHBAs defined for each storage system?

- Same vHBA are used (2vHBA) for both

Different Vsans used?

- Yes 2 vSANS are used (one per vHBA/FI).

Are you seeing flogi for wwpn from vHBA you are expecting to to connect to VPLEX, on the FIs (nxos#show npv flogi) ?

- I can see the expected wwpn on each FIs

What FC switches are above the FI?  If they are Cisco you should be able to issue command # show zoneset active vsan xx

Above command should confirm if you are seeing your wwpn storage targets and wwpn initiators active in the correct zones/vsans.

- Nexus 5672UP are above the FI, zoning and zoneset are correct. Triple checked. Everything is fine with the VNX though. FCoE is used between FI and Nexus

I forgot to mention that the VPLEX is running in production with B200M2-M3 and B22 blades (different VICs and FW versions) with ESXi5.5 cisco custom with no problem on other chassis and ucs version.

Many thanks for your help,

Best regards