Issue when upgrading APIC

[compterds]

Hello everyone,

I recently started by upgrading my controller (APIC).

Our target was the 2.0.2f and there was initally the 1.1.3f on our hardware.

So as we've seen from cisco, we needed to go through the 1.2.3 before trying to reach the target. (2.0.2f)

Finally, it worked on 2 of 3 controllers, and we're encoutering an issue on the third one.

To be honest I don't know what to do anymore , I tried a lot of things as : 

Upgrading controller by CLI with http server

Upgrading controller by CLI with scp

GUI doent work as cisco recognized that it is impossible to upgrade by GUI on 1.2.3 (Bug)

And everytime the process crashed when extracting the iso.

That's why I tried many other version (even downgrade) and the output is still the same.

I've attached the output which is displayed in GUI

[Jason Williams]

Do you have other firmware images under the /firmware/ directory or firmware repository (GUI) that you no longer need?

If so, could you delete some of those images to free up some space? 

Did you verify the md5 of the image after it was uploaded to the APIC? 

CLI command on the APIC is md5sum <file name> 

[compterds]

There is only my 1.2.3 (current version) and when I'm trying to upgrade, I upload one iso and trying the upgrade. If it fails, I delete the file and try another.

I checked the md5, and it was the same as the iso downloaded on cisco.com

Yoann

[Tomas de Leon]

You could be running into a known issue where the installer log is stuck in a loop and is filling up the drive.  You will need "root" access to check and fix if this the issue.

This would require you opening a case with the Cisco ACI TAC.

Please run this on the APIC, "df -h /firmware" and "df -h | grep -E '[0-9]G' ". Please  provide the results.

Cheers!

T.

[compterds]

Hi there,

Thanks for your reply.

Here is the first one :

apic1# df -h /firmware
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_ifc0-firmware
40G 12G 26G 32% /firmware

and now the second one : 

apic1# df -h | grep -E '[0-9]G'
/dev/dm-1 40G 14G 25G 36% /
tmpfs 4.0G 72M 4.0G 2% /dev/shm
tmpfs 32G 304K 32G 1% /tmp
40G 14G 25G 36% /data
40G 12G 26G 32% /firmware
591G 375M 561G 1% /data2

When I do an "cat /var/log/messages"  (Don't know if the output has a link with our issue)

Apr 27 05:59:15 apic1 kernel: loop0: rw=0, want=600008, limit=27416
Apr 27 05:59:15 apic1 kernel: attempt to access beyond end of device
Apr 27 05:59:15 apic1 kernel: loop0: rw=0, want=600008, limit=27416
Apr 27 05:59:15 apic1 kernel: attempt to access beyond end of device
Apr 27 05:59:15 apic1 kernel: loop0: rw=0, want=600200, limit=27416
Apr 27 05:59:15 apic1 kernel: attempt to access beyond end of device
Apr 27 05:59:15 apic1 kernel: loop0: rw=0, want=600200, limit=27416
Apr 27 05:59:15 apic1 kernel: attempt to access beyond end of device
Apr 27 05:59:15 apic1 kernel: loop0: rw=0, want=599944, limit=27416
Apr 27 05:59:15 apic1 kernel: attempt to access beyond end of device
Apr 27 05:59:15 apic1 kernel: loop0: rw=0, want=599944, limit=27416
Apr 27 05:59:15 apic1 kernel: attempt to access beyond end of device
Apr 27 05:59:15 apic1 kernel: loop0: rw=0, want=599944, limit=27416
Apr 27 05:59:15 apic1 kernel: attempt to access beyond end of device
Apr 27 05:59:15 apic1 kernel: loop0: rw=0, want=599944, limit=27416
Apr 27 05:59:15 apic1 kernel: attempt to access beyond end of device
Apr 27 05:59:15 apic1 kernel: loop0: rw=0, want=599944, limit=27416
Apr 27 05:59:15 apic1 kernel: attempt to access beyond end of device
Apr 27 05:59:15 apic1 kernel: loop0: rw=0, want=599960, limit=27416
Apr 27 05:59:15 apic1 kernel: attempt to access beyond end of device
Apr 27 05:59:15 apic1 kernel: loop0: rw=0, want=599944, limit=27416
Apr 27 05:59:15 apic1 kernel: attempt to access beyond end of device
Apr 27 05:59:15 apic1 kernel: loop0: rw=0, want=599944, limit=27416
Apr 27 05:59:15 apic1 kernel: attempt to access beyond end of device
Apr 27 05:59:15 apic1 kernel: loop0: rw=0, want=599944, limit=27416

This morning I had an upgrade successful on this apic (from 1.2.3 to 2.0.2) after a hardreboot, but then I tried to upgrade from 2.0.2 to 2.1.2 and I've got the same issue than before...

Yoann

[YanL]

You could try to connect to the CIMC and mount the APIC firmware ISO in the KVM and boot it to upgrade the APIC. You probably will have to decommission the controller and re-register it.

Yan