APIC Upgrade Issues

[zurabtamar]

I am facing issues with upgrading my APICs from ACI version 5.2.3e to 6.0.8f. Below are the details and the steps I have performed so far:

APIC Configuration:

• APIC Model: L4
• BMC Version: 4.2(3e) → Updated to 4.3(4.242038)
• BIOS Version: C225M6.4.2.3e.0.0418231026 → Updated to C225M6.4.3.4b.0.0827240450

Issue:

• Initially, the APICs were running ACI version 5.2.3e.
• I attempted to upgrade to ACI version 6.0.8f via vKVM using an ISO file. During installation, the process became stuck and did not progress for hours.
• After rebooting, the APICs did not revert to the previous operating system and failed to boot up properly.
• Post this, I updated the CIMC firmware to ucs-c225m6-huu-4.3.4.242038, which updated the BMC and BIOS versions as mentioned above.

Current Behavior:
When trying to install various ACI versions, the installation process repeatedly gets stuck at random steps such as:

• "Starting firewalld - dynamic firewall daemon"
• "Enabling compressing RAM with zRAM"
• "Starting Login Service"

Additional Details:

• TPM Status: Enabled
• The same issue persists even after the firmware updates.

I would appreciate your guidance on resolving this issue, including any recommended troubleshooting steps.

Thank you.

[AshSe]

Hello @zurabtamar 

The issue you're facing with upgrading your APICs from ACI version 5.2.3e to 6.0.8f could be due to several factors, including compatibility issues, firmware mismatches, or configuration problems. Below are some troubleshooting steps and recommendations to help you resolve the issue:

---

1. Verify Compatibility

• Ensure that the APIC model (L4) and its hardware components (BIOS, BMC, TPM, etc.) are compatible with ACI version 6.0.8f. Refer to the Cisco ACI Compatibility Matrix to confirm that the firmware versions and hardware are supported for the target ACI version.
• Check if there are any specific prerequisites for upgrading from 5.2.3e to 6.0.8f. Some upgrades may require an intermediate version (e.g., 5.2.x → 5.3.x → 6.0.x).

---

2. Check Trusted Platform Module (TPM) Configuration

• Since TPM is enabled, ensure that the TPM version and configuration are compatible with ACI version 6.0.8f. Some ACI versions require TPM 2.0, and issues can arise if the TPM is misconfigured.
• If TPM is not required for your deployment, consider temporarily disabling it in the BIOS and retrying the installation.

---

3. Verify ISO File Integrity

• Ensure that the ISO file used for the upgrade is not corrupted. Verify the checksum (MD5 or SHA256) of the ISO file against the checksum provided by Cisco.
• If the ISO file is corrupted, download it again from the Cisco website.

---

4. Perform a Clean Installation

• If the upgrade process is failing repeatedly, consider performing a clean installation of ACI version 6.0.8f. This involves wiping the existing installation and starting fresh.
• Steps for a clean installation:
  • Boot into the vKVM console.
  • Use the ACI ISO file to perform a fresh installation.
  • During the installation process, ensure that the disk is properly formatted and any remnants of the previous installation are removed.

---

5. Check CIMC Settings

• Verify that the CIMC settings are correctly configured for the installation process. Ensure that the virtual media (vKVM) is properly mounted and accessible.
• Check the boot order in the BIOS to ensure that the APIC boots from the correct device (e.g., virtual CD/DVD drive for the ISO file).

---

6. Monitor Installation Logs

• During the installation process, monitor the logs to identify where the process is getting stuck. Use the vKVM console to view detailed logs.
• Common points of failure, such as "Starting firewalld" or "Enabling compressing RAM with zRAM," may indicate issues with the underlying hardware or software configuration.

---

7. Update Firmware to the Latest Version

• Ensure that all firmware components (BIOS, BMC, CIMC, etc.) are updated to the latest versions supported for your APIC model and ACI version.
• You mentioned that the BMC and BIOS were updated to 4.3(4.242038) and C225M6.4.3.4b.0.0827240450, respectively. Verify that these are the recommended versions for ACI version 6.0.8f.

---

8. Check Disk Health

• The issue could be related to disk health or storage configuration. Use the CIMC interface to check the status of the disks and RAID configuration.
• If the disk is failing or misconfigured, replace or reconfigure it before attempting the installation.

---

9. Try an Intermediate Upgrade

• If the direct upgrade to 6.0.8f continues to fail, try upgrading to an intermediate version (e.g., 5.2.3e → 5.3.x → 6.0.8f). This can help resolve compatibility issues or bugs in the upgrade process.

---

10. Contact Cisco TAC

• If the issue persists after performing the above steps, it is recommended to open a case with Cisco TAC. Provide them with the following details:
  • APIC model and hardware details.
  • Current firmware versions (BIOS, BMC, CIMC, etc.).
  • Logs from the installation process.
  • Steps you have already performed.

---

Additional Notes:

• Ensure that you have a backup of your ACI configuration before performing any upgrades or clean installations.
• If you are using a cluster of APICs, ensure that the other APICs are not affected by the upgrade process.

Hope This Helps!!!

AshSe

Forum Tips: 

1. Insert photos/images inline - don't attach.
2. Always mark helpful and correct answers, it helps others find what they need.
3. For a prompt reply, kindly tag @name. An email will be automatically sent to the member.

[Remi-Astruc]

Hi @zurabtamar ,

Are these APICs in cluster / connected to Leaf, or are these individual off-line APIC upgrades?

Did you run the Pre-upgrade validation script before upgrading? If yes, can you share the recorded output?

What gives CLI command acidiag avread ?

Regards

[YanL]

If you were trying to upgrade a fully-fit fabric you were using the wrong process. Fully-fit fabric are upgraded using the ACI GUI Cisco APIC Installation and ACI Upgrade and Downgrade Guide - Upgrading or Downgrading with APIC Release 5.1 or Later Using the GUI [Cisco Application Policy Infrastructure Controller (APIC)] - Cisco 

The procedure you used is typically when you need to join a new APIC to an existing fabric because an APIC failed or you're adding a second POD (multi-pod) and the APIC version needs to match the existing fabric. 

To recover your APIC you will need to remove the controller from the fabric, re-install the ACI version your fabric is currently running and re-join the fabric. With newer APICs there are additional steps from just using KVM mounted ISO that help with install speed and issues in general. You need to mount the ISO using CIMC, enable SoL and mount the ISO using CIMC CLI and not KVM, do not miss Step 7 From the SOL console

Cisco APIC Installation and ACI Upgrade and Downgrade Guide - Installing or Recovering Cisco APIC Images [Cisco Application Policy Infrastructure Controller (APIC)] - Cisco

[Perpetous_Dreamer_07]

---

@zurabtamar wrote:

• BIOS Version: C225M6.4.2.3e.0.0418231026 → Updated to C225M6.4.3.4b.0.0827240450

• Post this, I updated the CIMC firmware to ucs-c225m6-huu-4.3.4.242038, which updated the BMC and BIOS versions as mentioned above.

---

I am facing similar issues (getting mine onto 5.3(2b)) and resorted to upgrading BIOS/BMC in advance. But what the HUU did for me is just upgrade the CIMC version, BIOS and BMC seems to be running the old ones. What exact procedure did you follow in your case ? 
Thanks in advance.

[YanL]

For CIMC upgrades look into unattended HUU. Cisco Standalone C-Series HUU Utilities - Cisco Community much faster and don't have to baby the upgrade every 30 minutes although Cisco improved on HUU I don't use KVM method.