NCS 5500 will not upgrade

Rauchster101 · ‎05-03-2022

We are continuously encountering this error while attempting to upgrade a NCS5500 from 7.0.2 to 7.3.2. The error presents as follows:

May 03 16:16:52 Install operation  95 by root:
 install add source /harddisk:/ NCS5500-iosxr-k9-7.3.2.tar 
May 03 16:16:53 Action 1: install add action started
May 03 16:16:58 Install operation will continue in the background
May 03 16:19:56 WARNING : the following files/packages were not added to the Software Repository due to unsupported file format
                README-NCS5500-iosxr-k9-7.3.2.txt
May 03 16:24:48 On Node 0/RP0 Sysadmin: Failed to scp with error: /misc/disk1/tftpboot/ncs5500-mini-x-7.3.2: No space left on device
May 03 16:24:49 ERROR!! failed while distributing packages to sysadmin

May 03 16:24:52 Install operation 95 aborted

We have verified space is available through the CLI show media command, going into the admin/run bash prompts and verifying files, as well as opening a FTP session to the device to verify the show media command is accurate.

RP/0/RP0/CPU0:NCS5501-LAB#show media
Mon May  2 20:44:40.547 UTC

Media Information for local node.

----------------------------------------------
Partition                    Size     Used  Percent    Avail
rootfs:                      3.9G     1.3G      34%     2.5G
apphost:                     3.7G     106M       3%     3.4G
/dev/sde                     969M     421M      47%     482M
harddisk:                    5.6G     2.1G      39%     3.3G
log:                         459M      75M      18%     350M
config:                      459M     2.0M       1%     423M
disk0:                       2.0G      40M       3%     1.8G
---------------------------------------------------
rootfs: = root file system (read-only)
log: = system log files (read-only)
config: = configuration storage (read-only)

Here are outputs of a few other commands that are relevant to the install:

RP/0/RP0/CPU0:NCS5501-LAB#cd harddisk:/
RP/0/RP0/CPU0:NCS5501-LAB#dir
Tue May  3 17:48:19.843 UTC

Directory of /misc/disk1
    13 drwxr-xr-x. 2       4096 May 24  2019 nvram
    28 -rwxr--r--. 1 2037104640 May  3 13:14 NCS5500-iosxr-k9-7.3.2.tar
129025 drwx------. 3       4096 May  3 15:00 ima
    12 drwxr-xr-x. 2       4096 May 24  2019 tftpboot
    17 drwxr-xr-x. 6       4096 May  3 15:05 cisco_support
   443 -rwx------. 1       8495 Apr 14 18:39 PRE_UPGRADE_running_config_04_14_2022.txt
    11 drwx------. 6      16384 May 24  2019 lost+found
   456 -rwx------. 1       8519 Apr 19 18:31 PRE_running_config_041922.txt
    14 drwxr-xr-x. 2       4096 May 24  2019 shutdown
    16 drwxr-xr-x. 2       4096 May 24  2019 ipodwdm_log
    15 -rw-r--r--. 1         44 May 24  2019 oor_aware_process
   370 -rw-r--r--. 1     260488 May  3 15:02 bcm_console_log_output.txt
   375 drwxr-xr-x. 2       4096 May  3 15:00 .sppdc
    21 drwxr-xr-x. 2       4096 May 24  2019 dumper
129029 drwxr-xr-x. 2       4096 Apr 21 14:48 showtech
   363 drwxr-xr-x. 6       4096 Nov 20  2019 xr
129027 drwxr-xr-x. 3       4096 Apr 21 14:48 pam

5831720 kbytes total (3506308 kbytes free)


RP/0/RP0/CPU0:NCS5501-LAB#show run | i fpd
Tue May  3 17:55:23.833 UTC
Building configuration...
fpd auto-upgrade enable

We had also previously run into a different installation error for no space left, but it presented differently, showing exactly how much space was needed, even though the available space was more than the number listed.

RP/0/RP0/CPU0:NCS5501-LAB#install add source harddisk:/ ncs5500-iosxr-k9-7.3.2.tar
Mon May  2 19:58:08.464 UTC
May 02 19:58:10 Install operation 89 started by root:
 install add source harddisk:/ ncs5500-iosxr-k9-7.3.2.tar 
May 02 19:58:16 Install operation will continue in the background
RP/0/RP0/CPU0:NCS5501-LAB#May 02 20:06:02 Install operation 89 aborted

Assistance in finding the reason why the device is saying that it doesn't have enough room, when it seems to have more than enough would be greatly appreciated.

tkarnani · ‎05-03-2022

I have attached the upgrade guide for 732, There is a known issue that requires bridge smu CSCvv35484

do you have it installed?

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvv35484

Thank you

hostasaurus · ‎08-31-2022

The bug ID seems to be missing affected releases, and while it says fixed, I'm having this same issue now trying to go to 7.5.2

smilstea · ‎08-31-2022

Can you try downloading the smu for 7.0.2 and installing it first then trying your upgrade?

https://software.cisco.com/download/home/279017029/type/280867577/release/7.0.2

Also I wanted to point out that the error message is complaining about sysadmin space (admin) not XR space, so you would need to check show media or show filesystem in admin mode.

Thanks,

Sam

hostasaurus · ‎08-31-2022

Unfortunately too late, I've already done it the old way by adding the iso first and then the rpm's, which allowed me to bypass the issue and complete the upgrade. I do have a df from in sysadmin->bash mode in case that's of any use:

[sysadmin-vm:0_RP0:~]$df -h
Filesystem                                  Size  Used Avail Use% Mounted on
/dev/loop0                                  2.4G  1.2G  1.1G  51% /
devfs                                        64K  4.0K   60K   7% /dev
tmpfs                                        64K     0   64K   0% /dev/cgroup
tmpfs                                        16G   28K   16G   1% /var/volatile
tmpfs                                        16G   72K   16G   1% /run
tmpfs                                        16G     0   16G   0% /media/ram
none                                         16G  1.2M   16G   1% /dev/shm
none                                         16G  1.2M   16G   1% /dev/shm
/dev/mapper/pci_disk1-ssd_disk1_calvados_1  5.6G  102M  5.2G   2% /misc/disk1
/dev/loop1                                  459M  180M  245M  43% /var/log
/dev/loop2                                  459M  3.7M  421M   1% /misc/config
/dev/loop3                                 1011M  8.0M  934M   1% /misc/scratch
none                                        512K     0  512K   0% /mnt
/dev/loop4                                  5.6G  1.2G  4.1G  22% /install_repo
tmpfs                                        10M  8.0K   10M   1% /media/install_tmpfs
/dev/ubi3_0                                  15M  632K   13M   5% /mnt/plog

For reference, which 7.0.2 SMU are you recommending I try? I have a few more of these yet to schedule, so I can try doing that on the next one when I go from 6 to 7.0.2 to 7.5.x

smilstea · ‎08-31-2022

You need the SMU for CSCvv35484 which Tarun mentioned, on the webpage I linked just search for that bug id and then download and install that smu first. Then your upgrade should be smoother

Sam

minas.balaskas · ‎06-28-2023

Hello smilstea,

I tried to install the SMUs but for .arm.rpm , i am taking the below skipping error

RP/0/RP0/CPU0:ZH_9218-78WUE-NCS1#sh install log reverse
Wed Jun 28 15:50:14.886 CEST
Jun 28 15:33:47 Install operation 30 started by root:
install add source harddisk:/ ncs5500-sysadmin-system-7.0.2.1-r702.CSCvv35484.arm.rpm ncs5500-sysadmin-system-7.0.2.1-r702.CSCvv35484.x86_64.rpm synchronous
Jun 28 15:33:49 Action 1: install add action started
Jun 28 15:33:53 Skipping package ncs5500-sysadmin-system-7.0.2.1-r702.CSCvv35484.arm.rpm of unsupported architecture type
Jun 28 15:34:00 Packages added:
Jun 28 15:34:00 ncs5500-sysadmin-system-7.0.2.1-r702.CSCvv35484.x86_64
Jun 28 15:34:01 Action 1: install add action completed successfully
Jun 28 15:34:03 Install operation 30 finished successfully

Can i proceed with the install activate without this arp.rpm ?

Thank you

smilstea · ‎06-28-2023

Some ncs5500 components use x86 and others use arm processors.

It looks like the x86 version installed correctly.

Sam

hostasaurus · ‎08-31-2022

Ah thanks; will give that a try.