ACI Spine upgrade stuck at "Approving Request"

RedNectar · ‎10-22-2020

I have a spine and leaf that don't want to complete and upgrade from v4 to v5.

A little history: This cluster had previously been upgraded to v5 OK, but was backgraded to 4 for another purpose. I want to move it back to 5.

But I upgraded again days ago -The APIC upgraded without problem, but for days, one leaf and one spine have been "queued for upgrade". But it never happens. (The other leaf is 1st gen, so can't be upgraded)

The CLI gives a little more information than the GUI:

apic1# show firmware upgrade scheduler-status switch-group Spine
Switch-Group-Name : Spine
Status            : running, upgraded 0, failed 0, inqueue 1
Details           : Node: 2101, Policy: Spine, Details: Node was already in
                   scheduler approved upgrade list, approving request

Can anyone tell me what "approving request" means - and where I can see the request, and ideally, kick it along?

RedNectar aka Chris Welsh.
Forum Tips: 1. Paste images inline - don't attach. 2. Always mark helpful and correct answers, it helps others find what they need.

Robert Burns · ‎10-23-2020

Hey Chris,

Can you share/attach outputs:

FROM THE APIC:
show version

acidiag fnvread
moquery -c firmwareCompRunning | grep -E "dn|expectedVer|version"
moquery -c faultInst -f 'fault.Inst.code=="F1582"'

cd /firmware/fwrepos/fwrepo
ls -lrt

If there are any Leaf Upgrade Groups started, see if you can stop/remove them. Next try to upgrade JUST the spine first. If you have Graceful Upgrade enabled - disable it. See if that kicks off, then follow through with your remaining supported Leafs. If not, I'll advise next steps after reviewing the requested outputs.

Robert

RedNectar · ‎10-23-2020

Hi @Robert Burns

Thanks for getting back to me.

I had actually completed your final steps anyway (remove all upgrade groups) and kicked the upgrade off again as shown below:

And I've ended up in exactly the same place, so here is the ouput you asked for:

apic1# show firmware upgrade scheduler-status switch-group Spine
Switch-Group-Name : Spine
Status            : running, upgraded 0, failed 0, inqueue 1
Details           : Node: 2101, Policy: Spine, Details: Node was already in
                   scheduler approved upgrade list, approving request
apic1# acidiag fnvread
      ID   Pod ID                 Name    Serial Number         IP Address    Role        State   LastUpdMsgId
--------------------------------------------------------------------------------------------------------------
    2101        1            Spine2101      FDO2329119D     10.2.200.65/32   spine         active   0
    2201        1             Leaf2201      FDO23340LX2     10.2.200.64/32    leaf         active   0
    2202        1             Leaf2202      SAL1914CNV4     10.2.200.66/32    leaf         active   0

Total 3 nodes

apic1# fnvread
        id               address  disabled    active  occupied permanent              model  nodeRole  nodeType  fabricId     podId
-----------------------------------------------------------------------------------------------------------------------------------------------
   2101(1)     10.2.200.65/32(1)     NO(0)    YES(1)    YES(1)    YES(1)      N9K-C9332C(1)      3(1)      0(1)      2(1)      1(1)
   2201(1)     10.2.200.64/32(1)     NO(0)    YES(1)    YES(1)    YES(1) N9K-C93180YC-FX(1)      2(1)      0(1)      2(1)      1(1)
   2202(1)     10.2.200.66/32(1)     NO(0)    YES(1)    YES(1)    YES(1)     N9K-C9396PX(1)      2(1)      0(1)      2(1)      1(1)


apic1# moquery -c firmwareCompRunning | grep -E "dn|expectedVer|version"
dn             : topology/pod-1/node-2101/sys/ch/supslot-1/sup/fpga-1/running
expectedVer    :
version        : 0x9
dn             : topology/pod-1/node-2101/sys/ch/supslot-1/sup/fpga-2/running
expectedVer    : 0x3
version        : 0x3
dn             : topology/pod-1/node-2201/sys/ch/supslot-1/sup/fpga-1/running
expectedVer    :
version        : 0x19
dn             : topology/pod-1/node-2201/sys/ch/supslot-1/sup/fpga-2/running
expectedVer    : 0x10
version        : 0x10
dn             : topology/pod-1/node-2202/sys/ch/supslot-1/sup/fpga-1/running
expectedVer    : 0x16
version        : 0x16
dn             : topology/pod-1/node-2202/sys/ch/supslot-1/sup/fpga-2/running
expectedVer    : 0x15
version        : 0x15
dn             : topology/pod-1/node-2202/sys/ch/supslot-1/sup/fpga-3/running
expectedVer    : 0x20
version        : 0x20
apic1#
apic1# cd /firmware/fwrepos/fwrepo
apic1# ls -lrth
total 16G
-rwxr-xr-x 1 root root 6.2G Aug 23 15:57 aci-apic-dk9.5.0.2h.bin
-rwxr-xr-x 1 root root 1.9G Aug 23 21:36 aci-n9000-dk9.14.2.4o.bin
-rwxr-xr-x 1 root root 5.7G Aug 23 21:47 aci-apic-dk9.4.2.4o.bin
-rwxr-xr-x 1 root root 501M Aug 23 22:18 aci-plugin-Cisco_NIBASE.bin
-rwxr-xr-x 1 root root 340M Aug 31 08:07 aci-plugin-Cisco_StateChangeChecker.bin
-rwxr-xr-x 1 root root 713M Aug 31 08:08 aci-plugin-Cisco_NIA.bin
-rwxr-xr-x 1 root root 2.4M Sep  8 19:18 aci-plugin-Cisco_PolicyViewer.bin
drwxr-xr-x 2 root root 4.0K Sep  8 19:18 md5sum
-rwxr-xr-x 1 root root 132K Oct  1 18:17 aci-catalog-dk9.70.8.2.bin
-rwxr-xr-x 1 root root 134K Oct 18 22:15 aci-catalog-dk9.70.8.4.bin
drwxr-xr-x 2 ifc  root 4.0K Oct 24 07:22 boot
apic1# ssh spine2101
<snip>
Spine2101# moquery -c firmwareRunning
Total Objects shown: 1

# firmware.Running
biosTs           : 2020-01-17T11:00:00.000+11:00
biosVer          : 05.40
childAction      :
descr            : version 14.2(4o) [build 14.2(4o)]
dn               : sys/fwstatuscont/running
internalLabel    : 6cee849b7a87ef3ce41ae728088a1c9f813ac3ac
ksFile           : bootflash:aci-n9000-dk9.14.2.4o.bin
modTs            : never
mode             : normal
peVer            : 4.2(4o)
rn               : running
srFwFlashRecTs   : never
srFwFlashRecVer  :
srFwFlashTs      : never
srFwFlashVer     :
srFwImageTs      : never
srFwImageVer     :
srFwRunningSrc   : unknown
srFwRunningVer   :
status           :
sysFile          : bootflash:///auto-s
ts               : 2020-07-03T22:40:55.000+11:00
type             : switch
version          : n9000-14.2(4o)

Spine2101# cat /mnt/pss/installer_detail.log | grep -E "Compatibility|New-Version|FPGA|Upgrade-Result|Success"
extract_fpga_imgs: Successfully extracted FPGA image
extract_bios_imgs: Successfully extracted BIOS images
update_tor_dev_versions IOFPGA: device id 220 and version 9
update_tor_dev_versions MIFPGA_0 device id 221 and version 3
update_tor_dev_versions MIFPGA_1 device id 221 and version 0
Compatibility check:
Module  Type           EPLD      Running-Version   New-Version  Upg-Required
     1   SUP  MI FPGA                    0.003       0.003          Yes
     1   SUP  IO FPGA                    0.009       0.017          Yes
Module 1 : MI FPGA [Programming] : 100.00% (     64 of      64 sectors)
Module 1 : IO FPGA [Programming] : 100.00% (     64 of      64 sectors)
Module    Type  Upgrade-Result
     1     SUP         Success
Module    Type  Upgrade-Result
     1     SUP         Success
Reseting Active SUP (Module 1) FPGAs. Please wait...
Spine2101# vsh -c "show version module 1 epld"

***********************************************
PFM Chassis_TYPE: 33
Active SUP Slot:1
PFM Mod info sw_card_id:355
PFM Mod info slot_num:0
PFM Mod info Mod_Type:0
***********************************************
update_tor_dev_versions IOFPGA: device id 220 and version 9
update_tor_dev_versions MIFPGA_0 device id 221 and version 3
update_tor_dev_versions MIFPGA_1 device id 221 and version 0
EPLD Device                     Version
---------------------------------------
MI FPGA                           0.003
IO FPGA                           0.009
MI FPGA2                          0.000
MI FPGA5                           -nan

RedNectar aka Chris Welsh.
Forum Tips: 1. Paste images inline - don't attach. 2. Always mark helpful and correct answers, it helps others find what they need.

Robert Burns · ‎10-23-2020

Your firmware repository looks like it's missing the aci-n9000-dk9.15.0.2h.bin switch image file. If this shows in the UI repository, can you try to delete and re-upload it.

Robert