IR809 Not enough disk space available after updating to IOx 1.9.0.5

[HarryDing0224]

Hello, I came to a problem after updating IR809 to IOx 1.9.0.5 from IOx 1.5 using Bundle Image ir800-universalk9_npe-bundle.SPA.159-3.M.bin.

After successfully updating the software, when I install an Docker application, ioxclient showed an error as below:

command Name:  application-install
Error. Server returned 500
{
	"description": "Not enough space in the disk, needed 217496576, got 198180864",
	"errorcode": -1003,
	"message": "Error during app installation"
}

Then I checked the diagnostic(disk type) on Cisco IOx Local Manager, and I found some information below:

--Top Disk Usage--
/*:
Error: 451M	/oldroot
38M	/home
11M	/lib64
7.4M	/boot
5.5M	/bin
3.6M	/lib
2.8M	/etc
645K	/usr
43K	/var
12K	/lost+found
5.0K	/sbin
2.0K	/ovfs-rw
2.0K	/ovfs
2.0K	/mnt
1.0K	/selinux
1.0K	/media
1.0K	/downloads
du: cannot access '/oldroot/proc/3910/task/3910/fdinfo/4': No such file or directory
du: cannot access '/oldroot/proc/3910/task/3910/fd/4': No such file or directory
du: cannot access '/oldroot/proc/3910/fdinfo/4': No such file or directory
du: cannot access '/oldroot/proc/3910/fd/4': No such file or directory
/software/*:
228M	/software/tmp
11M	/software/caf
16K	/software/lost+found
12K	/software/downloads
4.0K	/software/backup
4.0K	/software/apps

I have another original IR809 with IOx 1.6.0.5, the software of which is not updated yet. I checked the disk type diagnostic too and it showed:

--Top Disk Usage--
/*:
Error: 1.1G	/oldroot
35M	/home
11M	/lib64
7.4M	/boot
5.5M	/bin
3.6M	/lib
2.6M	/etc
612K	/usr
42K	/var
12K	/lost+found
5.0K	/sbin
2.0K	/ovfs-rw
2.0K	/ovfs
2.0K	/mnt
1.0K	/selinux
1.0K	/media
1.0K	/downloads
du: cannot access '/oldroot/proc/14812/task/14812/fdinfo/4': No such file or directory
du: cannot access '/oldroot/proc/14812/task/14812/fd/4': No such file or directory
du: cannot access '/oldroot/proc/14812/fdinfo/4': No such file or directory
du: cannot access '/oldroot/proc/14812/fd/4': No such file or directory
du: cannot access '/oldroot/proc/14736/fdinfo/46': No such file or directory
/software/*:
873M	/software/caf
64K	/software/downloads
16K	/software/lost+found
4.0K	/software/tmp
4.0K	/software/backup
4.0K	/software/apps

I am wondering why the disk space available get fewer after the bundle updating. Any ideas ?

[suressan]

Did you see automatic increase of the disk usage right after bundle install to newer version ?

Could you pls share before and after upgrade snapshot on the same device to get clear understanding ? 

For now try to clean up docker layers on local registry using below ioxclient command and then install again.

$ ioxclient layer delete

[HarryDing0224]

Thank you for your reply.

I don't know how to check if the disk usage increased after the upgrade while I notice there is fewer disk available for /software.

I did not create a snapshot just before the bundle installation but I have one in 2019-08-12 for the previous version. I uploaded both before and after snapshot of this IR809, and please check.

I tried to install the app after I deleted all the docker layers but it still failed, with error below:

currently active profile :  default
Command Name:  application-install
Saving current configuration
Could not complete your command :  Error. Server returned 500
{
	"description": "Invalid Archive file: [Errno 28] No space left on device",
	"errorcode": -1,
	"message": "Invalid Archive file"
} 

[HarryDing0224]

I am wondering that if it is because I did not delete the previous bundle image which occupied some disk space after I upgrade the new bundle image. If so,  is there any IOS command to check and delete them?

[HarryDing0224 · Accepted solution]

I found a document from cisco which just hit my issue.

The document is available on https://www.cisco.com/c/en/us/support/docs/cloud-systems-management/iox/212923-low-space-available-for-iox-package-depl.html