Upgrading Cisco ACI - APIC

[jameslopes]

Hi,

I have tried to upgrade the APICs of the ACI infrastructure  to the version 1.2(3c) but the upgrade wasn't sucessfull and when the APIC reboot, it try to boot in the new version, but there are not this version installed. So I need to select manually in the GRUB the correct version.

And when I try to upgrade the APIC to other version, I have this fault: 

"Controller 1 failed to upgrade to apic-1.0(4q). The upgrade has an upgrade status of Failed - . The failure occured at the Updating Grub/efiboot Boot Manager stage of the install."

The actual version is 1.0(2j).

Thanks,

James

[Tomas de Leon]

James,

Are you still experiencing issues?

You may be running into the following:

CSCut41444 - Free up loop devices on install failures..

Root Cause Analysis:

In order to upgrade its firmware, the APIC needs to mount the ISO image using a loop device. A loop device allows access to the files contained within the ISO, allowing the ISO to be treated as a block device. In 1.0(x) releases, if an APIC upgrade fails, the loop device used to mount the ISO aren't cleared.  Since a total of eight loop devices exist on an APIC by default, if all of these loop devices are in use, the APIC won’t be able to mount the ISO image and upgrade.

Workaround: 

1) Rebooting the APIC will force the loop devices to be cleared.

2) If the APIC cannot be rebooted, one can try to manually unmount the .img file and then the .iso to clear the loop devices using the following steps:

Running the df command as root we see:

root@ifav55-ifc1:~# df
Filesystem           1K-blocks      Used Available Use% Mounted on
<snip>
/firmware/fwrepos/fwrepo/aci-apic-dk9.1.1.1j-broken.iso
                       3982694   3982694         0 100% /tmp/tmpYTNfPF
/tmp/tmpYTNfPF/images/install.img
                        205952    205952         0 100% /tmp/tmp9en0R2
/firmware/fwrepos/fwrepo/aci-apic-dk9.1.1.1j-broken.iso
                       3982694   3982694         0 100% /tmp/tmpIFxfE7
/tmp/tmpIFxfE7/images/install.img
                        205952    205952         0 100% /tmp/tmpdEgIH0
/firmware/fwrepos/fwrepo/aci-apic-dk9.1.1.1j-broken.iso
                       3982694   3982694         0 100% /tmp/tmpw8HY3W
/tmp/tmpw8HY3W/images/install.img
                        205952    205952         0 100% /tmp/tmpfMD74k
/firmware/fwrepos/fwrepo/aci-apic-dk9.1.1.1j.bin
                       3982694   3982694         0 100% /tmp/tmpuddSrE
/tmp/tmpuddSrE/images/install.img
                        205952    205952         0 100% /tmp/tmpoliSrw
/dev/dm-11            41282880   5854708  33331124  15% /rfs1

 

Note: You must first unmount the .img file BEFORE the .bin (or .iso) file.

Run the following to unmount:

root@ifav55-ifc1:~# umount /tmp/tmpoliSrw
root@ifav55-ifc1:~# umount /tmp/tmpuddSrE

Fix Details:

In 1.1(x) releases, during the installation process we check for any loop devices in use and free them (umount) as needed.

If you still need assistance, please let us know

Thanks

T.

[#3]

Workaround: 

1) Rebooting the APIC will force the loop devices to be cleared.

2) If the APIC cannot be rebooted, one can try to manually unmount the .img file and then the .iso to clear the loop devices using the following steps:

Running the df command as root we see:

root@ifav55-ifc1:~# df
Filesystem           1K-blocks      Used Available Use% Mounted on
<snip>
/firmware/fwrepos/fwrepo/aci-apic-dk9.1.1.1j-broken.iso
                       3982694   3982694         0 100% /tmp/tmpYTNfPF
/tmp/tmpYTNfPF/images/install.img
                        205952    205952         0 100% /tmp/tmp9en0R2
/firmware/fwrepos/fwrepo/aci-apic-dk9.1.1.1j-broken.iso
                       3982694   3982694         0 100% /tmp/tmpIFxfE7
/tmp/tmpIFxfE7/images/install.img
                        205952    205952         0 100% /tmp/tmpdEgIH0
/firmware/fwrepos/fwrepo/aci-apic-dk9.1.1.1j-broken.iso
                       3982694   3982694         0 100% /tmp/tmpw8HY3W
/tmp/tmpw8HY3W/images/install.img
                        205952    205952         0 100% /tmp/tmpfMD74k
/firmware/fwrepos/fwrepo/aci-apic-dk9.1.1.1j.bin
                       3982694   3982694         0 100% /tmp/tmpuddSrE
/tmp/tmpuddSrE/images/install.img
                        205952    205952         0 100% /tmp/tmpoliSrw
/dev/dm-11            41282880   5854708  33331124  15% /rfs1

 

Note: You must first unmount the .img file BEFORE the .bin (or .iso) file.

Run the following to unmount:

root@ifav55-ifc1:~# umount /tmp/tmpoliSrw
root@ifav55-ifc1:~# umount /tmp/tmpuddSrE

Fix Details:

In 1.1(x) releases, during the installation process we check for any loop devices in use and free them (umount) as needed.

If you still need assistance, please let us know

Thanks

T.

[jameslopes]

Hi Thomas,

I have already tryed to reboot the apic, but the problem did not solve. When I try the command "df", this is the output:

admin@apic2:~> df

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/dm-1 37154368 9600340 25666696 28% /
tmpfs 4194304 52048 4142256 2% /dev/shm
tmpfs 32852268 636 32851632 1% /tmp
df: `/boot': No such file or directory
df: `/efiboot': No such file or directory
/dev/mapper/vg_ifc0_ssd-data
37154368 9600340 25666696 28% /data
df: `/logs': No such file or directory
df: `/dmecores': No such file or directory
df: `/techsupport': No such file or directory
/dev/mapper/vg_ifc0-firmware
37156416 4912612 30356368 14% /firmware
df: `/scratch': No such file or directory
df: `/data2': No such file or directory
df: `/data/log': No such file or directory
df: `/data/shell-data': No such file or directory
df: `/rfs2': No such file or directory
df: `/securedata': No such file or directory
df: `/var/run/bashroot/.aci': No such file or directory

And I don't have root access for this equipments.

Thaks

[Tomas de Leon]

James,

 Can you ssh to one of the APICs and get the output of the CLI command "firmware list" & "version"?

Did you put 1.0(4q) in the firmware repository? Did you try a Policy Upgrade? 

You may want to open a TAC case and they engineer will be able to help you out on this issue.

Thanks

T.