Re: ASR9001 - Problem with free space after deleting files

miodrg.jevtic@saga.rs · ‎06-08-2018

Hi all,

I have a problem with free space on disk0: on ASR9001. After deleting file, free space is the same:

RP/0/RSP0/CPU0:#dir disk0:
Sat Jun 2 21:36:41.353 CET

Directory of disk0:

15322 -r-- 978940 Thu Jan 1 01:06:46 1970 .bitmap
17 -r-- 565248 Thu Jan 1 01:06:46 1970 .inodes
18 -rw- 0 Thu Jan 1 01:06:46 1970 .boot
19 -rw- 0 Thu Jan 1 01:06:46 1970 .altboot
15327 drwx 4096 Fri Feb 5 15:10:00 2016 LOST.DIR
15328 -r-- 32768 Thu Jan 1 01:06:46 1970 .longfilenames
15329 drwx 4096 Fri Feb 5 15:10:52 2016 aaa
15330 drwx 4096 Fri Feb 5 15:28:42 2016 config
15331 drwx 4096 Fri Feb 5 15:10:55 2016 onepk_cli_rdsfs
15332 drwx 4096 Fri Feb 5 15:11:09 2016 sla
15333 drwx 4096 Wed Apr 27 03:19:56 2016 snmp
15386 drwx 4096 Fri Feb 5 15:12:30 2016 asr9k-scfclient-5.2.4
15335 drwx 4096 Sat Jun 2 21:13:03 2018 instdb
15392 drwx 4096 Fri Feb 5 15:12:31 2016 asr9k-diags-supp-5.2.4
15404 drwx 4096 Fri Feb 5 15:12:32 2016 asr9k-ce-5.2.4
15439 drwx 4096 Fri Feb 5 15:14:08 2016 asr9k-fwding-5.2.4
15339 drwx 4096 Fri Feb 5 15:12:40 2016 eem_rdsfs
551272 drwx 4096 Fri Feb 5 15:15:58 2016 asr9k-base-5.2.4
1196793 drwx 4096 Fri Feb 5 15:16:03 2016 asr9k-os-mbi-5.2.4
1196802 drwx 4096 Fri Feb 5 15:16:22 2016 asr9k-cpp-5.2.4
2107058 drwx 4096 Fri Feb 5 15:16:30 2016 iosxr-ce-5.2.4
2275410 drwx 4096 Fri Feb 5 15:16:31 2016 iosxr-diags-5.2.4
2275433 drwx 4096 Fri Feb 5 15:16:54 2016 iosxr-routing-5.2.4
2387572 drwx 4096 Fri Feb 5 15:19:48 2016 iosxr-fwding-5.2.4
3318937 drwx 4096 Fri Feb 5 15:23:01 2016 iosxr-infra-5.2.4
4307173 drwx 4096 Fri Feb 5 15:23:04 2016 asr9k-mini-px-5.2.4
15349 -rw- 5955 Sat Jun 2 21:13:05 2018 sam_certdb
15350 drwx 4096 Fri Feb 5 15:28:01 2016 license
15351 drwx 4096 Fri Feb 5 15:29:16 2016 sm_rdsfs
15352 drwx 4096 Fri Feb 5 15:29:15 2016 var
15353 drwx 4096 Fri Feb 5 15:29:16 2016 virtual-instance
15354 -rwx 1217 Sat Jun 2 21:01:54 2018 css-root.cer
4307193 drwx 4096 Thu Feb 4 23:43:41 2016 iosxr-infra-5.2.4.CSCut52232-1.0.0
4307203 drwx 4096 Thu Feb 4 23:43:42 2016 asr9k-px-5.2.4.CSCut52232-1.0.0
4307212 drwx 4096 Thu Feb 4 23:54:25 2016 iosxr-mgbl-5.2.4
4427650 drwx 4096 Thu Feb 4 23:54:28 2016 asr9k-mgbl-supp-5.2.4
4427659 drwx 4096 Thu Feb 4 23:54:31 2016 asr9k-mgbl-px-5.2.4
15360 -rw- 126 Sat Jun 2 21:13:05 2018 sam_crldb
4427666 drwx 4096 Thu Feb 4 23:54:46 2016 iosxr-mpls-5.2.4
4652725 drwx 4096 Thu Feb 4 23:54:49 2016 asr9k-mpls-px-5.2.4
4652730 drwx 4096 Thu Feb 4 23:56:13 2016 asr9k-fpd-5.2.4
4652849 drwx 4096 Thu Feb 4 23:56:16 2016 asr9k-fpd-px-5.2.4
4652854 drwx 4096 Thu Feb 4 23:57:24 2016 iosxr-mcast-5.2.4
7993201 drwx 4096 Thu Feb 4 23:57:28 2016 asr9k-mcast-supp-5.2.4
7993229 drwx 4096 Thu Feb 4 23:57:32 2016 asr9k-mcast-px-5.2.4
15368 drwx 4096 Fri Feb 5 00:22:18 2016 np
15369 drwx 4096 Wed Apr 27 03:41:49 2016 syslog
9453454 -rwx 327680 Sat Jun 2 21:12:07 2018 asr9k-px-5.3.0.CSCut52232.tar
9453438 drwx 4096 Sat Jun 2 21:12:57 2018 iosxr-infra-5.3.0.CSCut52232-1.0.0
9453429 -rwx 10943369 Thu May 17 13:47:29 2018 asr9k-k9sec-px.pie-5.2.4
9453475 drwx 4096 Sat Jun 2 21:13:01 2018 asr9k-px-5.3.0.CSCut52232-1.0.0
9453481 -rwx 1566924800 Sat Jun 2 21:30:46 2018 ASR9k-iosxr-px-k9-5.3.0.tar

4009738240 bytes total (1825811968 bytes free)

RP/0/RSP0/CPU0:#delete disk0:ASR9k-iosxr-px-k9-5.3.0.tar
Sat Jun 2 21:37:08.474 CET
Delete disk0:/ASR9k-iosxr-px-k9-5.3.0.tar[confirm]
RP/0/RSP0/CPU0:#

RP/0/RSP0/CPU0:#dir disk0:
Sat Jun 2 21:37:15.836 CET

Directory of disk0:

15322 -r-- 978940 Thu Jan 1 01:06:46 1970 .bitmap
17 -r-- 565248 Thu Jan 1 01:06:46 1970 .inodes
18 -rw- 0 Thu Jan 1 01:06:46 1970 .boot
19 -rw- 0 Thu Jan 1 01:06:46 1970 .altboot
15327 drwx 4096 Fri Feb 5 15:10:00 2016 LOST.DIR
15328 -r-- 32768 Thu Jan 1 01:06:46 1970 .longfilenames
15329 drwx 4096 Fri Feb 5 15:10:52 2016 aaa
15330 drwx 4096 Fri Feb 5 15:28:42 2016 config
15331 drwx 4096 Fri Feb 5 15:10:55 2016 onepk_cli_rdsfs
15332 drwx 4096 Fri Feb 5 15:11:09 2016 sla
15333 drwx 4096 Wed Apr 27 03:19:56 2016 snmp
15386 drwx 4096 Fri Feb 5 15:12:30 2016 asr9k-scfclient-5.2.4
15335 drwx 4096 Sat Jun 2 21:13:03 2018 instdb
15392 drwx 4096 Fri Feb 5 15:12:31 2016 asr9k-diags-supp-5.2.4
15404 drwx 4096 Fri Feb 5 15:12:32 2016 asr9k-ce-5.2.4
15439 drwx 4096 Fri Feb 5 15:14:08 2016 asr9k-fwding-5.2.4
15339 drwx 4096 Fri Feb 5 15:12:40 2016 eem_rdsfs
551272 drwx 4096 Fri Feb 5 15:15:58 2016 asr9k-base-5.2.4
1196793 drwx 4096 Fri Feb 5 15:16:03 2016 asr9k-os-mbi-5.2.4
1196802 drwx 4096 Fri Feb 5 15:16:22 2016 asr9k-cpp-5.2.4
2107058 drwx 4096 Fri Feb 5 15:16:30 2016 iosxr-ce-5.2.4
2275410 drwx 4096 Fri Feb 5 15:16:31 2016 iosxr-diags-5.2.4
2275433 drwx 4096 Fri Feb 5 15:16:54 2016 iosxr-routing-5.2.4
2387572 drwx 4096 Fri Feb 5 15:19:48 2016 iosxr-fwding-5.2.4
3318937 drwx 4096 Fri Feb 5 15:23:01 2016 iosxr-infra-5.2.4
4307173 drwx 4096 Fri Feb 5 15:23:04 2016 asr9k-mini-px-5.2.4
15349 -rw- 5955 Sat Jun 2 21:13:05 2018 sam_certdb
15350 drwx 4096 Fri Feb 5 15:28:01 2016 license
15351 drwx 4096 Fri Feb 5 15:29:16 2016 sm_rdsfs
15352 drwx 4096 Fri Feb 5 15:29:15 2016 var
15353 drwx 4096 Fri Feb 5 15:29:16 2016 virtual-instance
15354 -rwx 1217 Sat Jun 2 21:01:54 2018 css-root.cer
4307193 drwx 4096 Thu Feb 4 23:43:41 2016 iosxr-infra-5.2.4.CSCut52232-1.0.0
4307203 drwx 4096 Thu Feb 4 23:43:42 2016 asr9k-px-5.2.4.CSCut52232-1.0.0
4307212 drwx 4096 Thu Feb 4 23:54:25 2016 iosxr-mgbl-5.2.4
4427650 drwx 4096 Thu Feb 4 23:54:28 2016 asr9k-mgbl-supp-5.2.4
4427659 drwx 4096 Thu Feb 4 23:54:31 2016 asr9k-mgbl-px-5.2.4
15360 -rw- 126 Sat Jun 2 21:13:05 2018 sam_crldb
4427666 drwx 4096 Thu Feb 4 23:54:46 2016 iosxr-mpls-5.2.4
4652725 drwx 4096 Thu Feb 4 23:54:49 2016 asr9k-mpls-px-5.2.4
4652730 drwx 4096 Thu Feb 4 23:56:13 2016 asr9k-fpd-5.2.4
4652849 drwx 4096 Thu Feb 4 23:56:16 2016 asr9k-fpd-px-5.2.4
4652854 drwx 4096 Thu Feb 4 23:57:24 2016 iosxr-mcast-5.2.4
7993201 drwx 4096 Thu Feb 4 23:57:28 2016 asr9k-mcast-supp-5.2.4
7993229 drwx 4096 Thu Feb 4 23:57:32 2016 asr9k-mcast-px-5.2.4
15368 drwx 4096 Fri Feb 5 00:22:18 2016 np
15369 drwx 4096 Wed Apr 27 03:41:49 2016 syslog
9453454 -rwx 327680 Sat Jun 2 21:12:07 2018 asr9k-px-5.3.0.CSCut52232.tar
9453438 drwx 4096 Sat Jun 2 21:12:57 2018 iosxr-infra-5.3.0.CSCut52232-1.0.0
9453429 -rwx 10943369 Thu May 17 13:47:29 2018 asr9k-k9sec-px.pie-5.2.4
9453475 drwx 4096 Sat Jun 2 21:13:01 2018 asr9k-px-5.3.0.CSCut52232-1.0.0

4009738240 bytes total (1825811968 bytes free)

Any help?

Best regards,

Miodrag

xr-escalation · ‎06-08-2018

hi Miodrag,

there could be a delay in assessing the actual free/used space. Does it still show the same after some time? Can you cross-check that against shell commands:

run du -k /disk0:
run df -k /disk0:

Pozdrav,
/Aleksandar

miodrg.jevtic@saga.rs · ‎06-08-2018

Hi Aleksandar,

Yes, after 10 days used/free space is the same. I tried suggested commands:

RP/0/RSP0/CPU0:#run df -k /disk0:
Fri Jun 8 12:10:29.390 CET
/dev/eusb00t77 3915760 3317677 598082 85% /dev/disk0:/

Disk space is the same:

RP/0/RSP0/CPU0:ASR_CM#dir disk0:
Fri Jun 8 12:10:38.163 CET

Directory of disk0:

15322 -r-- 978940 Thu Jan 1 01:06:46 1970 .bitmap
17 -r-- 565248 Thu Jan 1 01:06:46 1970 .inodes
18 -rw- 0 Thu Jan 1 01:06:46 1970 .boot
19 -rw- 0 Thu Jan 1 01:06:46 1970 .altboot
15327 drwx 4096 Fri Feb 5 15:10:00 2016 LOST.DIR
15328 -r-- 32768 Thu Jan 1 01:06:46 1970 .longfilenames
15329 drwx 4096 Fri Feb 5 15:10:52 2016 aaa
15330 drwx 4096 Fri Feb 5 15:28:42 2016 config
15331 drwx 4096 Fri Feb 5 15:10:55 2016 onepk_cli_rdsfs
15332 drwx 4096 Fri Feb 5 15:11:09 2016 sla
15333 drwx 4096 Wed Apr 27 03:19:56 2016 snmp
15386 drwx 4096 Fri Feb 5 15:12:30 2016 asr9k-scfclient-5.2.4
15335 drwx 4096 Sat Jun 2 21:13:03 2018 instdb
15392 drwx 4096 Fri Feb 5 15:12:31 2016 asr9k-diags-supp-5.2.4
15404 drwx 4096 Fri Feb 5 15:12:32 2016 asr9k-ce-5.2.4
15439 drwx 4096 Fri Feb 5 15:14:08 2016 asr9k-fwding-5.2.4
15339 drwx 4096 Fri Feb 5 15:12:40 2016 eem_rdsfs
551272 drwx 4096 Fri Feb 5 15:15:58 2016 asr9k-base-5.2.4
1196793 drwx 4096 Fri Feb 5 15:16:03 2016 asr9k-os-mbi-5.2.4
1196802 drwx 4096 Fri Feb 5 15:16:22 2016 asr9k-cpp-5.2.4
2107058 drwx 4096 Fri Feb 5 15:16:30 2016 iosxr-ce-5.2.4
2275410 drwx 4096 Fri Feb 5 15:16:31 2016 iosxr-diags-5.2.4
2275433 drwx 4096 Fri Feb 5 15:16:54 2016 iosxr-routing-5.2.4
2387572 drwx 4096 Fri Feb 5 15:19:48 2016 iosxr-fwding-5.2.4
3318937 drwx 4096 Fri Feb 5 15:23:01 2016 iosxr-infra-5.2.4
4307173 drwx 4096 Fri Feb 5 15:23:04 2016 asr9k-mini-px-5.2.4
15349 -rw- 5955 Sat Jun 2 21:13:05 2018 sam_certdb
15350 drwx 4096 Fri Feb 5 15:28:01 2016 license
15351 drwx 4096 Fri Feb 5 15:29:16 2016 sm_rdsfs
15352 drwx 4096 Fri Feb 5 15:29:15 2016 var
15353 drwx 4096 Fri Feb 5 15:29:16 2016 virtual-instance
15354 -rwx 1217 Sat Jun 2 21:01:54 2018 css-root.cer
4307193 drwx 4096 Thu Feb 4 23:43:41 2016 iosxr-infra-5.2.4.CSCut52232-1.0.0
4307203 drwx 4096 Thu Feb 4 23:43:42 2016 asr9k-px-5.2.4.CSCut52232-1.0.0
4307212 drwx 4096 Thu Feb 4 23:54:25 2016 iosxr-mgbl-5.2.4
4427650 drwx 4096 Thu Feb 4 23:54:28 2016 asr9k-mgbl-supp-5.2.4
4427659 drwx 4096 Thu Feb 4 23:54:31 2016 asr9k-mgbl-px-5.2.4
15360 -rw- 126 Sat Jun 2 21:13:05 2018 sam_crldb
4427666 drwx 4096 Thu Feb 4 23:54:46 2016 iosxr-mpls-5.2.4
4652725 drwx 4096 Thu Feb 4 23:54:49 2016 asr9k-mpls-px-5.2.4
4652730 drwx 4096 Thu Feb 4 23:56:13 2016 asr9k-fpd-5.2.4
4652849 drwx 4096 Thu Feb 4 23:56:16 2016 asr9k-fpd-px-5.2.4
4652854 drwx 4096 Thu Feb 4 23:57:24 2016 iosxr-mcast-5.2.4
7993201 drwx 4096 Thu Feb 4 23:57:28 2016 asr9k-mcast-supp-5.2.4
7993229 drwx 4096 Thu Feb 4 23:57:32 2016 asr9k-mcast-px-5.2.4
15368 drwx 4096 Fri Feb 5 00:22:18 2016 np
15369 drwx 4096 Wed Apr 27 03:41:49 2016 syslog

4009738240 bytes total (612436480 bytes free)

Pozdrav,

Miodrag

xr-escalation · ‎06-08-2018

hi Miodrag,

hmm, the output shows that the free space was decreased even further

4009738240 bytes total (1825811968 bytes free)

4009738240 bytes total (612436480 bytes free)

Are you using disk0: to upload files etc? It's better to use harddisk: for that purpose because the IOS XR operating system uses disk0: for the installation, while the harddisk: is used for storing support files (showtechs, coredumps, etc.).

With "run du -k /disk0:" we'll be able to see what exactly is there on disk0: that uses up so much space. The output will be long, so better attach the output to your reply instead of not copy/paste the output into the message.

If this is a lab router, you can wipe out the disk0: using turboboot:
https://supportforums.cisco.com/t5/service-providers-documents/asr9000-xr-understanding-turboboot-and-initial-system-bring-up/ta-p/3146024

Pozdrav,
/Aleksandar

miodrg.jevtic@saga.rs · ‎06-08-2018

Hi Aleksandar,

Sorry for wrong output:

4009738240 bytes total (1825811968 bytes free) - is before copy IOS on disk0:

4009738240 bytes total (612436480 bytes free) - after copy IOS on disk0 and deleting.

Yes, I copied IOS on disk0. Mistake which I will try to correct in future.

This router is in production.

File RUN.txt attached.

Pozdrav,

Miodrag

xr-escalation · ‎06-08-2018

hi Miodrag,

it's not really a mistake, it's just a matter of best practices. :)

The "du -k" doesn't show anything in lost files and the total for "/disk0:" shows as 577 MB (591182 kB). This differs a lot from the output of "df -k" which was 2082 MB or 3239 MB in the outputs you collected earlier:

(4009738240 - 1825811968) / (1024 * 1024) = 2082 MB
(4009738240 - 612436480) / (1024 * 1024) = 3239 MB

There are two possible explanations:

1) the file was still in use by some process, although it was deleted. But in that case it should show up in lost files.
2) The "du" output doesn't include all files when directory depths is higher than 3 levels (I've seen this on some QNX forums). In latest XR releases, the output is valid even for eight levels of directories.

Are you planning an upgrade from 5.2.4 to a more recent EMR? A good choice would be 6.1.4 or 6.2.3. Very likely the issue you see will be resolved after the upgrade.

Pozdrav,
/Aleksandar

miodrg.jevtic@saga.rs · ‎06-08-2018

Hi Aleksandar,

problem with disk space appeard when I tried to upgrade on 6.0.2 (3DES IOS because this router doesn't have 3DES). Other routers in network have version 6.0.2 3DES. First, I tried with FTP and command admin install add tar ftp://x.x.x.x/asr9K-iosxr-px-k9-5.3.0.tar but I have problem. Then, tried with copy on disk0: and admin install add tar disko0:asr9K-iosxr-px-k9-5.3.0.tar but unsuccessfully, not enough disk space.

Only solution is reload the router :-).

Also, is it necessary to upgrade to 5.3.0 and then to 6.0.2. Currently version is 5.2.4. I'm asking because post-expiry SMU. I found that upgrading from Pre - 5.3.1 images to 6.0.2 must install post-expiry SMU.

Thanks in advanced.

Prozdrav,

Miodrag

xr-escalation · ‎06-08-2018

hi Miodrag ,

yes, post-expiry SMU is required.

When you install directly from a .tar, the OS will decompress the .tar to a temporary location on "harddisk:" before adding files to "disk0:". In older releases (before 6.x), the speed of access to harddisk was slow. Put it all together, the fastest way to upgrade is to add individual pies from the FTP server. You can simplify the admin task of selecting individual pies significantly if you make use of the CSM Server.

Download CSM:
https://software.cisco.com/download/release.html?mdfid=282423206&softwareid= 284777134&release=3.5&relind=AVAILABLE&rellifecycle=&reltype=latest

CSM Server Documentation:
https://supportforums.cisco.com/document/13154846/cisco-software-manager-33- overview-documentation

CSM Server Videos:
- Introduction to CSM Server: https://youtu.be/isxN08x-mr4
- Getting Started with CSM Server: https://www.youtube.com/watch?v=omdpr3uP_b4

Pozdrav,
/Aleksandar