IOS eXR rootfs: and harddisk:/misc/disk1 is above 80% utilization

[tritor]

Hi,
I've recently migrated an ASR9904 from cXR 6.6.3 to eXR 7.5.2 and the line cards seem to have not enough space on some mount points. It is not possible to install any new SMU, e.g. CSCwd36654:

Log:
LC/0/1/CPU0:Jun 7 07:10:29.348 CEST: resmon[240]: %HA-HA_WD-3-DISK_ALARM_ALERT : A monitored device / ( rootfs:/ ) is above 80% utilization. Current utilization = 85. Please remove unwanted user files and configuration rollback points.
LC/0/0/CPU0:Jun 7 07:10:34.952 CEST: resmon[174]: %HA-HA_WD-3-DISK_ALARM_ALERT : A monitored device / ( rootfs:/ ) is above 80% utilization. Current utilization = 85. Please remove unwanted user files and configuration rollback points.
LC/0/1/CPU0:Jun 7 07:15:29.348 CEST: resmon[240]: %HA-HA_WD-3-DISK_ALARM_ALERT : A monitored device /misc/disk1 ( harddisk: ) is above 80% utilization. Current utilization = 85. Please remove unwanted user files and configuration rollback points.
LC/0/0/CPU0:Jun 7 07:15:34.952 CEST: resmon[174]: %HA-HA_WD-3-DISK_ALARM_ALERT : A monitored device /misc/disk1 ( harddisk: ) is above 80% utilization. Current utilization = 85. Please remove unwanted user files and configuration rollback points.

#sh media location all
Media Info for Location: node0_0_CPU0
Partition Size Used Percent Avail
--------------------------------------------------------------------
rootfs: 3.8G 3.1G 85% 570M
log: 469M 64M 15% 371M
config: 469M 444K 1% 433M
disk0: 968M 2.3M 1% 900M

Media Info for Location: node0_1_CPU0
Partition Size Used Percent Avail
--------------------------------------------------------------------
rootfs: 3.8G 3.1G 85% 568M
log: 469M 81M 19% 354M
config: 469M 448K 1% 433M
disk0: 968M 3.1M 1% 899M

Media Info for Location: node0_RSP0_CPU0
Partition Size Used Percent Avail
--------------------------------------------------------------------
rootfs: 3.8G 2.8G 78% 814M
harddisk: 23G 114M 1% 21G
log: 1.9G 107M 6% 1.7G
config: 969M 2.3M 1% 901M
disk0: 3.8G 11M 1% 3.6G
harddiska: 3.4G 6.0M 1% 3.2G
apphost: 20G 251M 2% 18G

Media Info for Location: node0_RSP1_CPU0
Partition Size Used Percent Avail
--------------------------------------------------------------------
rootfs: 3.8G 2.8G 78% 815M
harddisk: 23G 116M 1% 21G
log: 1.9G 110M 7% 1.7G
config: 969M 2.1M 1% 901M
disk0: 3.8G 11M 1% 3.6G
harddiska: 3.4G 6.0M 1% 3.2G
apphost: 20G 161M 1% 18G

#sh platform
0/RSP0/CPU0 A9K-RSP880-LT-SE(Active) IOS XR RUN NSHUT
0/RSP1/CPU0 A9K-RSP880-LT-SE(Standby) IOS XR RUN NSHUT
0/0/CPU0 A99-8X100GE-SE IOS XR RUN NSHUT
0/1/CPU0 A9K-MOD400-SE IOS XR RUN NSHUT

#sh install active sum
Label : 7.5.2

Active Packages: 29
asr9k-xr-7.5.2 version=7.5.2 [Boot image]
asr9k-k9sec-x64-2.2.0.0-r752
asr9k-li-x64-1.1.0.0-r752
asr9k-9000v-nV-x64-1.0.0.0-r752
asr9k-mcast-x64-2.1.0.0-r752
asr9k-mpls-te-rsvp-x64-2.1.0.0-r752
asr9k-ospf-x64-1.0.0.0-r752
asr9k-services-x64-1.0.0.0-r752
asr9k-bng-supp-x64-1.0.0.0-r752
asr9k-mgbl-x64-2.0.0.0-r752
asr9k-optic-x64-1.0.0.0-r752
asr9k-mpls-x64-2.0.0.0-r752
asr9k-bng-x64-1.0.0.0-r752
asr9k-diags-x64-1.0.0.1-r752.CSCwb66960
asr9k-common-pd-fib-64-1.0.0.1-r752.CSCwd17716
asr9k-parser-x64-2.0.0.1-r752.CSCwc76982
asr9k-ospf-x64-1.0.0.2-r752.CSCwd32773
asr9k-iosxr-routing-64-1.0.0.2-r752.CSCwa34480
asr9k-iosxr-os-64-1.0.0.1-r752.CSCwb87810
asr9k-iosxr-fwding-64-2.0.0.9-r752.CSCwb43385
asr9k-iosxr-infra-64-2.0.0.10-r752.CSCwd53232
asr9k-mpls-te-rsvp-x64-2.1.0.1-r752.CSCwc29059
asr9k-spirit-boot-64-1.0.0.2-r752.CSCwd65553
asr9k-os-supp-64-4.0.0.5-r752.CSCwe11993
asr9k-gcp-fwding-64-1.0.0.7-r752.CSCwe29150
asr9k-base-64-5.0.0.7-r752.CSCwd53232
asr9k-fwding-64-4.0.0.14-r752.CSCwe80631
asr9k-bgp-x64-2.0.0.4-r752.CSCwe26697
asr9k-bng-x64-1.0.0.2-r752.CSCwf04617

#sh install ina sum
1 inactive package(s) found:
asr9k-spirit-boot-64-1.0.0.0-r752

#sh install super sum
No fully superseded package(s) found on the system

#rollback configuration ?
last Rollback last <n> commits made
to Rollback up to (and including) a specific commit
to-exclude Rollback up to (and excluding) a specific commit
1000000041 Commit ID

Any hints how to increase the mentioned file system? Or what I've done wrong?

Thanks a lot,
Torsten

[tkarnani · Accepted solution]

there are two bugs that i know of

https://bst.cisco.com/bugsearch/bug/CSCwc75529 or

can you try the work around listed here:

https://bst.cisco.com/bugsearch/bug/CSCwc75522

<B>Conditions:</B>
tmpdir_cleanup.sh script is responsible for cleaning /tmp when it reaches 90% capacity (max is 63 Mb). However before this changed it stored archive on /misc/disk1 no matter if that partition is mounted (e.g. harddisk on RP) or not (LC) thus on LC it consumed rootfs space. This optimization will cause script to delete archived /tmp files on LC to preserve rootfs space.

<B>Workaround:</B>
This issue can be temperarily cleared by removing below files (if any). 

1. Disable attestation agent. This will stop creating those ima files.
 
    config-t)# attestation runtime-integrity cache job disable
    config-t)# commit

2. Remove cache and phu files. these are safe to remove.
   run ssh lc<Y>_xr <== Here "Y" is slot number. For LC 0/2/cpu0, Y=2
   cd /misc/disk1/ima/ima_cache_node0-x-CPU0-xxxxxxxxx
   chattr -i ima-cache-x
   rm ima-cache-x

3. Delete phu.log files from below paths;

  cd /tmp
  ls -l phu.log 
  rm phu.log

  cd /misc/disk1/tmp_archive/tmp 
  ls -l phu.log
  rm phu.log

<B>Further Problem Description:</B>

[tritor]

Hi,
thanks for your advice.
I've tried the workaround and it makes it a bit better:

#sh media location all
Media Info for Location: node0_0_CPU0
Partition Size Used Percent Avail
--------------------------------------------------------------------
rootfs: 3.8G 3.0G 82% 687M
log: 469M 60M 14% 374M
config: 469M 444K 1% 433M
disk0: 968M 2.4M 1% 899M

Media Info for Location: node0_1_CPU0
Partition Size Used Percent Avail
--------------------------------------------------------------------
rootfs: 3.8G 3.0G 82% 685M
log: 469M 77M 18% 357M
config: 469M 448K 1% 433M
disk0: 968M 3.1M 1% 899M

I was able to install the wanted SMUs, but I'm still concerned about future SMUs..

Thanks, Torsten