Update - January 20, 2018
Intel released a statement today with additional information about a reboot issue associated with the microcode update for Broadwell, Haswell, Skylake and other processors, and is advising customers to not deploy select versions of the previously provided microcode. As a result Cisco removed UCS Manager 3.2(2e) and Cisco IMC 3.1(2e) from Cisco.com over the weekend. Intel also confirmed that they have identified the root cause for the reboot issues on these platforms, and have provided the first beta version to vendors, including Cisco, for testing. While we work closely with Intel to finalize a solution to remediate the involved issues as quickly as possible, Cisco will hold deploying new releases of microcode updates for UCS Manager and Cisco IMC software. For further updates from Intel click here. For further updates from Cisco, please see the Cisco Security Advisory.
Update - January 17th
Intel has just announced that the M5 firmware that Cisco released on January 16th is also impacted by the reboot issue that was previously seen on M4 platforms. It also impacts M3 platforms. Customers should consider the reboot impact prior to deploying the new UCS Manager 3.2(2e) or Cisco IMC Software 3.1(2e) versions. We will update our guidance as we receive additional information.
The Intel advisory also provides some performance impact data, and specifically highlights the minimal impact on CPU and memory intensive workloads. The Intel advisory is available at https://newsroom.intel.com/news/firmware-updates-and-initial-performance-data-for-data-center-systems/
January 16th
Earlier today, the Cisco UCS team released UCS Manager 3.2(2e) and Cisco IMC Software 3.1(2e) (and here and here). These releases include the Intel microcode 0x0200003A for Intel Xeon Scalable (Skylake) processors used in the Cisco UCS M5 generation B-Series, C-Series, and Hyperflex M5 servers. Operating System patches together with this Intel CPU microcode mitigate CVE-2017-5715, also known as Spectre/Variant 2. For further details, please see the UCS Manager 3.2 Release Notes or the Cisco IMC Software 3.1(2) Release Notes.
Cisco preferred to release a single UCS Manager patch that supports updated microcode for all of the Cisco UCS and Hyperflex M5, M4, M3 and M2 platforms. However, due to the Intel disclosure on issues with the microcode vulnerability fix for the Broadwell and Haswell processors used in Cisco UCS and Hyperflex M4 servers causing servers to potentially reboot, Cisco will not release the previously provided microcode for M4 servers but wait for an updated version of the microcode. There will also be a follow-on impact to the availability of microcode for Cisco UCS M3 and M2 platforms. We do not have detailed dates at this time to provide firmware for Cisco UCS and Hyperflex M4, M3, and M2 systems. Cisco will update the Cisco Security Advisory at https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20180104-cpusidechannel with revised dates as they become available.
Please note that as has been widely reported in the industry, the mitigation for Spectre/Variant 2 [CVE-2017-5715] may impact the observed performance of servers. In Cisco’s testing, we are seeing little impact for CPU and memory constrained workloads. Where we are seeing an impact is with I/O intense workloads, especially with local or SAN-based storage with sequential reads and sequential writes. While there is a significant amount of variation on the various tests that we’ve run, we are often seeing a 10-20% decrease in IOPs where there is both sequential reads and writes, an increase in processor utilization on average below 10%, and at times, an increase in latency. Customer and partner discussions over the past week have provided similar performance feedback. Other posts on performance impacts from RedHat and Microsoft may also be useful to understand the industry wide performance impacts. As always, your results will vary for your workloads, system overhead and applications.
We hope that this blog post provides further information to supplement the Cisco PSIRT Advisory and the initial UCS Response to Meltdown and Spectre Vulnerabilities blog post. UCS Manager 3.2(2e) and Cisco IMC Software are available for download now.
