Re: All Paths Down Upgrading Firmware from 4.2(1L) to 4.2(2a)

SteveT12 · ‎09-12-2022

I ran into and APD after I migrated from 4.2L to 4.2(2a). Here's the info:

APD Background

After the upgrade on Friday (9/9/22), the “Firmware Auto-Install” process did not turn “Auto Evac” off. I had to manually turn off “Auto Evac” after the firmware upgrade was complete. After I turned off “Auto Evac” I manually moved the primary Fabric Interconnect back to Fabric Interconnect A. I needed to do this because FI-A allows the networking capability to do our disaster recovery tests. However, once I did this, we experienced an “APD” in VMware. I had to hard restart all of the hosts in the cluster to get the machines back up and running.

UCS Firmware Upgrade Information

UCS Firmware version upgrading from: 4.2(1L)

UCS Firmware version upgrading to: 4.2(2a)

I only upgraded the UCS Management portion to this level. I did not upgrade the servers with this version of the firmware since it’s no longer supported.

Steven Tardy · ‎09-12-2022

Sounds like your UCS environment is not configured properly. Both Fabric Interconnects should allow all VLANs/traffic for full redundancy. Yet statements like this make me think otherwise:

@SteveT12 wrote:

I needed to do this because FI-A allows the networking capability to do our disaster recovery tests.

"APD" in VMware. . . . From what to what? Where is the datastore? What protocol? Is that protocol configured with full redundancy? What is the upstream device(s)/configuration?

Seems like you are looking for an RCA which may be a bit much for a community post so should probably engage Cisco TAC to fully investigate what happened.

@SteveT12 wrote:

UCS Firmware Upgrade Information

UCS Firmware version upgrading from: 4.2(1L)

UCS Firmware version upgrading to: 4.2(2a)

I only upgraded the UCS Management portion to this level. I did not upgrade the servers with this version of the firmware since it’s no longer supported.

If your servers are not supported on UCSM 4.2(2x) then they are not supported on UCSM 4.2(2x). In other words upgrading to UCS Management 4.2(2x) is by definition not supported.

SteveT12 · ‎09-13-2022

@Steven Tardy The datastores are located on a FC SAN. The Cisco Mini (6324) connects two fiber connections to a SAN switch. The UCS infrastructure software is compatible with 4.2(2a) and I upgraded the servers to 4.2(1n). I verified the infrastructure firmware (4.2(2a)) for the 6324 is compatible with my particular model of server with this firmware (4.2(1n)) on the Cisco UCS Hardware Compatibility List. When I manually made the Fi-A the lead, it dropped all connections. However, after the reboot of the hosts they came back. I'm just trying to figure out why this happened. I've been able to to move the lead in the past (pre-firmware upgrade) however, I'm trying to figure out why it happened. I've contacted TAC with little response back. I'm just putting this out there in case someone has experienced something like it. Thanks for replying.