Cisco C240-M4SX / 12G SAS Embedded Raid Controller Queue Depth Reporting issue in ESXi 6.0

[rgrim2lghealth]

All,

We've been a VMware vSAN 6.0 customer for the past 6 months running our environment entirely on Cisco C240-M4SX servers with the Cisco 12G SAS Embedded Raid controller.   Everything in the environment was running well until we began to bring in data warehouse workloads into the environment and began to notice performance issues around disk latency and more importantly high outstanding IO.  After reviewing ESXtop and ESXCFG we found that the queue length advertised to ESXi from the adapter was only 234 however the VMware HCL advertises 895; 234 is below minimum spec of 256 to properly operationalize vSAN.  We've worked diligently with VMware on this trying different certified and non-certified versions of the async driver for this raid controller in addition to newer firmware for the raid controller.  No matter the change the queue depth remained 234.  Does the presence of FBWC impact the queue depth reported to the OS?  VMware support has indicated that this is squarely a "hardware" issue.  Any ideas as to what may be causing this?

Our environment:

C240-M4SX

12G SAS Embedded Raid Controller (Operating JBOD / Pass-through, No FBWC)

UCSM 2.2(6c)

VMware driver for controller: 6.606.06.00-1OEM.550.0.0.1331820.x86_64.vib

VMware vSphere 6.0U1

[Kirk J]

Greetings.

I checked this out in the lab tonight on a C240M4 with a 12Gb SAS controller with 1GB cache/w Supercap.

With 8 physical disks in jbod mod, I installed ESXi 6.0 and ran esxtop, clicked 'f', then 'd' to select queue depth, and get the following on the 12Gb sas card:

vmhba2 is my 12Gb LSI sas controller, and advertises the 895 depth.

I noticed all the Cisco Bundles on the VMware site included the FWBC 1GB as part of the SKU.

I believe the FWBC may assist in providing the queue depth, as you are asking.

Can you confirm the qdepth on your setup from the native esxtop output?

Also, do you have access to the 1GB cache module to test in your setup?  PID: UCSC-MRAID12G-1G

Thanks,

Kirk..

[rgrim2lghealth]

Kirk,

Thank you for taking the time to mock this up in the lab and trial it.  Attached is output from ESXTOP showing vmhba3 (12GB SAS Raid controller) has a queue length of 234.   Unfortunately I do not have access to a cache module to add to the server.  Would you be able to remove your cache module in order to validate our working theory? 

Thanks,

Ryan

[Kirk J · Accepted solution]

Greetings.

Got a chance to spend some time in the lab, and removed the cache module.

After this, I now get '207', so I would say it is confirmed that the 1,2,4 GB cache modules are used to increase queue depth/length for these controllers.

Thanks,

Kirk

[rgrim2lghealth]

I can't thank you enough for taking the time to validate this.  This finally puts to rest a 2 month long problem discovery.  I wonder if there is anyway this can be added to Cisco's product documentation as I was unable to find any data indicating that FBWC has an impact on the adapters queue depth.   Either way .. thank you again and Happy New Year!

Cheers,

Ryan