Buy or Renew
Buy or Renew
Cisco + Splunk: It’s a new day for your data. Learn more.
 
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
1555
Views
5
Helpful
3
Replies

Cisco vXR 7.3.1 freezes - "High disk usage alert /misc/disk1"

davehouser1
Level 1
Level 1

‎08-09-2022 05:45 PM - edited ‎08-09-2022 05:51 PM

Hi,

I am running two Cisco vXR 7.3.1 in GNS3 as PE systems. 
The problem is both of them randomly will just freeze, and I cant SSH to them anymore. Checking the console shows the following:

vXR 01

 

 

 

0/RP0/ADMIN0:Aug 10 00:00:23.884 UTC: mediasvr[2841]: %MEDIASVR-MEDIASVR-4-PARTITION_USAGE_ALERT : High disk usage alert : /misc/disk1 exceeded 94%  
0/RP0/ADMIN0:Aug 10 00:04:53.807 UTC: wd_diskmon[11939]: %INFRA-WD_DISKMON_SYSADMIN-2-DISK_CRIT : Sysadmin: Device: /misc/disk1 Usage %: 95, State: CRITICAL, CRITICAL Threshold %: 95  
0/RP0/ADMIN0:Aug 10 00:04:53.894 UTC: mediasvr[2841]: %MEDIASVR-MEDIASVR-4-PARTITION_USAGE_ALERT : High disk usage alert : host /misc/scratch exceeded 89%  
0/RP0/ADMIN0:Aug 10 00:10:53.812 UTC: wd_diskmon[11939]: %INFRA-WD_DISKMON_SYSADMIN-2-DISK_CRIT : Sysadmin: Device: /misc/disk1 Usage %: 95, State: CRITICAL, CRITICAL Threshold %: 95  
0/RP0/ADMIN0:Aug 10 00:15:23.910 UTC: mediasvr[2841]: %MEDIASVR-MEDIASVR-4-PARTITION_USAGE_ALERT : High disk usage alert : /misc/disk1 exceeded 94%  
0/RP0/ADMIN0:Aug 10 00:16:53.816 UTC: wd_diskmon[11939]: %INFRA-WD_DISKMON_SYSADMIN-2-DISK_CRIT : Sysadmin: Device: /misc/disk1 Usage %: 95, State: CRITICAL, CRITICAL Threshold %: 95  
0/RP0/ADMIN0:Aug 10 00:19:53.917 UTC: mediasvr[2841]: %MEDIASVR-MEDIASVR-4-PARTITION_USAGE_ALERT : High disk usage alert : host /misc/scratch exceeded 89%  
0/RP0/ADMIN0:Aug 10 00:23:53.820 UTC: wd_diskmon[11939]: %INFRA-WD_DISKMON_SYSADMIN-2-DISK_CRIT : Sysadmin: Device: /misc/disk1 Usage %: 95, State: CRITICAL, CRITICAL Threshold %: 95

 

 

 

vXR02

 

 

 

0/RP0/ADMIN0:Aug  9 23:27:46.708 UTC: mediasvr[2840]: %MEDIASVR-MEDIASVR-4-PARTITION_USAGE_ALERT : High disk usage alert : host /misc/scratch exceeded 89%  
0/RP0/ADMIN0:Aug  9 23:42:46.726 UTC: mediasvr[2840]: %MEDIASVR-MEDIASVR-4-PARTITION_USAGE_ALERT : High disk usage alert : host /misc/scratch exceeded 89%  
0/RP0/ADMIN0:Aug  9 23:57:46.745 UTC: mediasvr[2840]: %MEDIASVR-MEDIASVR-4-PARTITION_USAGE_ALERT : High disk usage alert : host /misc/scratch exceeded 89%  
0/RP0/ADMIN0:Aug 10 00:12:46.764 UTC: mediasvr[2840]: %MEDIASVR-MEDIASVR-4-PARTITION_USAGE_ALERT : High disk usage alert : host /misc/scratch exceeded 89%

 

 

 

If I hit ENTER a couple times form console, after about 1-10 min, the system will come back to life and I can log in, then SSH will work again. 

I ran the following to see what is going on

 

 

 

P/0/RP0/CPU0:xr-pe-01#admin
Wed Aug 10 00:33:49.148 UTC

mr admin connected from 192.0.0.4 using ssh on sysadmin-vm:0_RP0
sysadmin-vm:0_RP0# run
Wed Aug  10 00:33:51.254 UTC+00:00

[sysadmin-vm:0_RP0:~]$df -H
Filesystem                                             Size  Used Avail Use% Mounted on
/dev/loop0                                             3.2G  543M  2.4G  19% /
devfs                                                  1.1M  4.1k  1.1M   1% /dev
tmpfs                                                   66k     0   66k   0% /dev/cgroup
tmpfs                                                  8.7G     0  8.7G   0% /var/volatile
tmpfs                                                  8.7G   95k  8.7G   1% /run
tmpfs                                                  8.7G     0  8.7G   0% /media/ram
none                                                   8.7G  803k  8.7G   1% /dev/shm
tmpfs                                                   68M   29k   68M   1% /tmp
none                                                   8.7G  803k  8.7G   1% /dev/shm
tmpfs                                                   68M   29k   68M   1% /tmp
/dev/mapper/panini_vol_grp-ssd_disk1_calvados_1        4.1G  3.7G  205M  95% /misc/disk1
/dev/mapper/panini_vol_grp-ssd_disk1_calvados_swtam_1   49M  537k   45M   2% /misc/swtam
/dev/mapper/sysadmin-vm_encrypted_log                  498M   80M  388M  17% /var/log
/dev/mapper/sysadmin-vm_encrypted_config               498M  6.2M  461M   2% /misc/config
/dev/mapper/sysadmin-vm_encrypted_scratch              1.1G  5.0M  969M   1% /misc/scratch
none                                                   525k     0  525k   0% /mnt
/dev/loop4                                             4.1G  899M  3.0G  24% /install_repo
[sysadmin-vm:0_RP0:~]$0/RP0/ADMIN0:Aug 10 00:34:10.389 UTC: mediasvr[2841]: %MEDIASVR-MEDIASVR-4-PARTITION_USAGE_ALERT : High disk usage alert : /misc/disk1 exceeded 94%

 

 

 

I can see that `/misc/disk1` partition is getting filled up, I ran the following to see what are the biggest files

 

 

sysadmin-vm:0_RP0:~]$ls -lS |head
total 3533964
-rw-r--r--. 1 root root 584252536 Jul  6 19:41 default-sdr--2.20220706-193925.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root 582290649 Jul  6 18:44 default-sdr--2.20220706-184037.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root 556521170 Jul  6 20:22 default-sdr--2.20220706-201836.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root 496746496 Jul  6 18:43 default-sdr--1.20220706-183708.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root 490942464 Jul  6 19:41 default-sdr--1.20220706-193602.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root 132510221 Jul 14 03:23 default-sdr--2.20220714-032053.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root 132488940 Jul 10 06:59 default-sdr--2.20220710-065756.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root 132439909 Jul 12 13:54 default-sdr--2.20220712-135204.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root 132423725 Jul 10 06:39 default-sdr--2.20220710-063540.core.RP.lxcdump.tar.lz4

 

 

Here is a list of all files in the folder:

 

 

[sysadmin-vm:0_RP0:~]$ls -lS
total 3533964
-rw-r--r--. 1 root root 584252536 Jul  6 19:41 default-sdr--2.20220706-193925.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root 582290649 Jul  6 18:44 default-sdr--2.20220706-184037.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root 556521170 Jul  6 20:22 default-sdr--2.20220706-201836.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root 496746496 Jul  6 18:43 default-sdr--1.20220706-183708.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root 490942464 Jul  6 19:41 default-sdr--1.20220706-193602.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root 132510221 Jul 14 03:23 default-sdr--2.20220714-032053.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root 132488940 Jul 10 06:59 default-sdr--2.20220710-065756.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root 132439909 Jul 12 13:54 default-sdr--2.20220712-135204.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root 132423725 Jul 10 06:39 default-sdr--2.20220710-063540.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root 132419795 Jul 10 06:49 default-sdr--2.20220710-064647.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root 130028153 Jul  6 21:22 default-sdr--2.20220706-211934.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root 103329792 Jul  7 22:11 default-sdr--2.20220707-220852.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root   1778404 Aug  7 04:17 cm_2866.by.6.20220807-041603.sysadmin-vm:0_RP0.fb4cb.core.gz
-rw-r--r--. 1 root root   1007315 Jul 29 15:31 cm_23610.by.6.20220729-152952.sysadmin-vm:0_RP0.fb4cb.core.gz
-rw-r--r--. 1 root root    544224 Jul 23 02:55 debug_agent_20003.by.6.20220723-025516.sysadmin-vm:0_RP0.d5ddb.core.gz
-rw-r--r--. 1 root root    527633 Aug  9 20:25 vm_manager_2883.by.6.20220809-202528.sysadmin-vm:0_RP0.2d758.core.gz
-rw-r--r--. 1 root root    525895 Jul 19 15:40 vm_manager_2859.by.6.20220719-154050.sysadmin-vm:0_RP0.2d758.core.gz
-rw-r--r--. 1 root root    408299 Aug  9 18:28 syslogd_helper_2820.by.6.20220809-182828.sysadmin-vm:0_RP0.22ca3.core.gz
-rw-r--r--. 1 root root    402783 Aug  4 04:12 syslogd_relay_2909.by.6.20220804-041239.sysadmin-vm:0_RP0.1c974.core.gz
-rw-r--r--. 1 root root    402315 Jul 20 05:42 ds_18986.by.6.20220720-054154.sysadmin-vm:0_RP0.6bb46.core.gz
-rw-r--r--. 1 root root    401739 Jul  7 09:49 syslogd_helper_2860.by.6.20220707-094926.sysadmin-vm:0_RP0.22ca3.core.gz
-rw-r--r--. 1 root root    392045 Jul 22 13:58 ds_31601.by.6.20220722-135809.sysadmin-vm:0_RP0.6bb46.core.gz
-rw-r--r--. 1 root root    390556 Jul 24 00:34 wd_diskmon_12775.by.6.20220724-003348.sysadmin-vm:0_RP0.65123.core.gz
-rw-r--r--. 1 root root    371739 Jul 21 09:52 timezone_config_18972.by.6.20220721-095231.sysadmin-vm:0_RP0.d6c5e.core.gz
-rw-r--r--. 1 root root    369418 Aug  9 20:25 timezone_config_2881.by.6.20220809-202518.sysadmin-vm:0_RP0.d6c5e.core.gz
-rw-r--r--. 1 root root    356126 Jul 19 22:06 ship_server_20702.by.6.20220719-220613.sysadmin-vm:0_RP0.dca7e.core.gz
-rw-r--r--. 1 root root    355334 Jul 19 11:06 ship_server_385.by.6.20220719-110522.sysadmin-vm:0_RP0.dca7e.core.gz
-rw-r--r--. 1 root root    332698 Aug  7 04:17 rvm_mgr_2882.by.6.20220807-041559.sysadmin-vm:0_RP0.61d7d.core.gz
-rw-r--r--. 1 root root    321192 Jul  7 09:49 shelf_mgr_2853.by.6.20220707-094927.sysadmin-vm:0_RP0.e9e54.core.gz
-rw-r--r--. 1 root root    312396 Aug  8 13:56 wd_memmon_20066.by.6.20220808-135653.sysadmin-vm:0_RP0.1d704.core.gz
-rw-r--r--. 1 root root    246702 Jul 20 23:04 wd_memmon_31259.by.6.20220720-230435.sysadmin-vm:0_RP0.1d704.core.gz
-rw-r--r--. 1 root root    233161 Jul 20 15:05 wd_memmon_20130.by.6.20220720-150508.sysadmin-vm:0_RP0.1d704.core.gz
-rw-r--r--. 1 root root    229546 Jul 24 11:14 wd_memmon_28713.by.6.20220724-111407.sysadmin-vm:0_RP0.1d704.core.gz
-rw-r--r--. 1 root root    225917 Jul 18 15:42 wd_memmon_4843.by.6.20220718-154205.sysadmin-vm:0_RP0.1d704.core.gz
-rw-r--r--. 1 root root    218125 Jul 25 07:51 dumper_28687.by.6.20220725-075149.sysadmin-vm:0_RP0.77cbb.core.gz
-rw-r--r--. 1 root root    196039 Jul 21 09:53 ssh_key_server_17769.by.6.20220721-095240.sysadmin-vm:0_RP0.af3f0.core.gz
-rw-r--r--. 1 root root    179876 Aug  9 20:25 ssh_key_server_3858.by.6.20220809-202514.sysadmin-vm:0_RP0.af3f0.core.gz
-rw-r--r--. 1 root root    159009 Jul 22 13:58 debug_client_22603.by.6.20220722-135807.sysadmin-vm:0_RP0.1a164.core.gz
-rw-r--r--. 1 root root    157211 Jul 24 00:34 debug_client_15177.by.6.20220724-003347.sysadmin-vm:0_RP0.1a164.core.gz
-rw-r--r--. 1 root root    156273 Aug  7 04:17 debug_client_2869.by.6.20220807-041559.sysadmin-vm:0_RP0.1a164.core.gz
-rw-r--r--. 1 root root     98989 Aug  9 18:28 lsof_log.log
-rw-r--r--. 1 root root     66229 Jul  7 09:49 shelf_mgr_2853.by.6.20220707-094927.sysadmin-vm:0_RP0.e9e54.core.txt
-rw-r--r--. 1 root root     41001 Jul 19 15:40 vm_manager_2859.by.6.20220719-154050.sysadmin-vm:0_RP0.2d758.core.txt
-rw-r--r--. 1 root root     40991 Aug  9 20:25 vm_manager_2883.by.6.20220809-202528.sysadmin-vm:0_RP0.2d758.core.txt
-rw-r--r--. 1 root root     36260 Jul 29 15:31 cm_23610.by.6.20220729-152952.sysadmin-vm:0_RP0.fb4cb.core.txt
-rw-r--r--. 1 root root     35519 Jul 23 00:18 debug_client_18002.by.6.20220723-001738.sysadmin-vm:0_RP0.1a164.core.gz
-rw-r--r--. 1 root root     35490 Aug  7 04:17 cm_2866.by.6.20220807-041603.sysadmin-vm:0_RP0.fb4cb.core.txt
-rw-r--r--. 1 root root     32441 Aug  9 18:28 syslogd_helper_2820.by.6.20220809-182828.sysadmin-vm:0_RP0.22ca3.core.txt
-rw-r--r--. 1 root root     32143 Jul  7 09:49 syslogd_helper_2860.by.6.20220707-094926.sysadmin-vm:0_RP0.22ca3.core.txt
-rw-r--r--. 1 root root     31655 Aug  4 04:12 syslogd_relay_2909.by.6.20220804-041239.sysadmin-vm:0_RP0.1c974.core.txt
-rw-r--r--. 1 root root     27260 Jul 24 00:34 wd_diskmon_12775.by.6.20220724-003348.sysadmin-vm:0_RP0.65123.core.txt
-rw-r--r--. 1 root root     26711 Aug  7 04:17 rvm_mgr_2882.by.6.20220807-041559.sysadmin-vm:0_RP0.61d7d.core.txt
-rw-r--r--. 1 root root     26575 Jul 19 11:06 ship_server_385.by.6.20220719-110522.sysadmin-vm:0_RP0.dca7e.core.txt
-rw-r--r--. 1 root root     25487 Jul 21 09:52 timezone_config_18972.by.6.20220721-095231.sysadmin-vm:0_RP0.d6c5e.core.txt
-rw-r--r--. 1 root root     24168 Aug  9 20:25 timezone_config_2881.by.6.20220809-202518.sysadmin-vm:0_RP0.d6c5e.core.txt
-rw-r--r--. 1 root root     24015 Jul 19 22:06 ship_server_20702.by.6.20220719-220613.sysadmin-vm:0_RP0.dca7e.core.txt
-rw-r--r--. 1 root root     23724 Jul 25 07:51 dumper_28687.by.6.20220725-075149.sysadmin-vm:0_RP0.77cbb.core.txt
-rw-r--r--. 1 root root     22100 Aug  8 13:56 wd_memmon_20066.by.6.20220808-135653.sysadmin-vm:0_RP0.1d704.core.txt
-rw-r--r--. 1 root root     22061 Jul 22 13:58 ds_31601.by.6.20220722-135809.sysadmin-vm:0_RP0.6bb46.core.txt
-rw-r--r--. 1 root root     22014 Jul 24 11:14 wd_memmon_28713.by.6.20220724-111407.sysadmin-vm:0_RP0.1d704.core.txt
-rw-r--r--. 1 root root     22001 Jul 18 15:42 wd_memmon_4843.by.6.20220718-154205.sysadmin-vm:0_RP0.1d704.core.txt
-rw-r--r--. 1 root root     21928 Jul 20 23:04 wd_memmon_31259.by.6.20220720-230435.sysadmin-vm:0_RP0.1d704.core.txt
-rw-r--r--. 1 root root     21729 Jul 20 05:42 ds_18986.by.6.20220720-054154.sysadmin-vm:0_RP0.6bb46.core.txt
-rw-r--r--. 1 root root     20979 Jul 20 15:05 wd_memmon_20130.by.6.20220720-150508.sysadmin-vm:0_RP0.1d704.core.txt
-rw-r--r--. 1 root root     19486 Jul 23 00:18 debug_client_18002.by.6.20220723-001738.sysadmin-vm:0_RP0.1a164.core.txt
-rw-r--r--. 1 root root     19380 Jul 21 09:53 ssh_key_server_17769.by.6.20220721-095240.sysadmin-vm:0_RP0.af3f0.core.txt
-rw-r--r--. 1 root root     19058 Aug  9 20:25 ssh_key_server_3858.by.6.20220809-202514.sysadmin-vm:0_RP0.af3f0.core.txt
-rw-r--r--. 1 root root     19007 Aug  7 04:17 debug_client_2869.by.6.20220807-041559.sysadmin-vm:0_RP0.1a164.core.txt
-rw-r--r--. 1 root root     18904 Jul 28 03:18 set_hostname_2778.by.6.20220728-031800.sysadmin-vm:0_RP0.0c6d9.core.txt
-rw-r--r--. 1 root root     18878 Jul 23 02:55 debug_agent_20003.by.6.20220723-025516.sysadmin-vm:0_RP0.d5ddb.core.txt
-rw-r--r--. 1 root root     17524 Jul 22 13:58 debug_client_22603.by.6.20220722-135807.sysadmin-vm:0_RP0.1a164.core.txt
-rw-r--r--. 1 root root     17172 Jul 24 00:34 debug_client_15177.by.6.20220724-003347.sysadmin-vm:0_RP0.1a164.core.txt
-rw-r--r--. 1 root root     14270 Jul 28 03:18 set_hostname_2778.by.6.20220728-031800.sysadmin-vm:0_RP0.0c6d9.core.gz
-rw-r--r--. 1 root root      6045 Jul 20 05:42 debug_client_18959.by.6.20220720-054156.sysadmin-vm:0_RP0.1a164.core.gz
-rw-r--r--. 1 root root      5332 Jul 29 15:31 ssh_key_server_3878.by.6.20220729-152952.sysadmin-vm:0_RP0.af3f0.core.lz4
drwxr-xr-x. 4 root root      4096 Aug  9 20:26 cisco_support
drwxr-xr-x. 2 root root      4096 Jun 17 12:31 dumper
drwx------. 2 root root      4096 Jun 17 12:22 lost+found
drwxr-xr-x. 2 root root      4096 Jun 17 12:22 tftpboot
-rw-r--r--. 1 root root      3975 Jul 20 05:42 debug_client_18959.by.6.20220720-054156.sysadmin-vm:0_RP0.1a164.core.txt
-rw-r--r--. 1 root root      1892 Jul 29 15:31 ssh_key_server_3878.by.6.20220729-152952.sysadmin-vm:0_RP0.af3f0.core.txt
-rw-r--r--. 1 root root        37 Jul  7 03:45 default-sdr--2.20220707-034057.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root        37 Jul  7 19:33 default-sdr--2.20220707-192843.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root        37 Jul  9 14:21 default-sdr--2.20220709-141857.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root        37 Jul  9 15:52 default-sdr--2.20220709-154947.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root        37 Jul 10 06:26 default-sdr--2.20220710-062127.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root        37 Jul 14 01:50 default-sdr--2.20220714-014732.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root        37 Jul 14 03:02 default-sdr--2.20220714-025741.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root        37 Jul 14 03:13 default-sdr--2.20220714-030915.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  6 22:09 default-sdr--1.20220706-220519.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  6 22:54 default-sdr--1.20220706-224959.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  6 23:15 default-sdr--1.20220706-230830.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  7 00:25 default-sdr--1.20220707-002011.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  7 01:25 default-sdr--1.20220707-012148.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  7 03:06 default-sdr--1.20220707-030249.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  7 03:21 default-sdr--1.20220707-032005.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  7 04:15 default-sdr--1.20220707-041048.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  7 06:55 default-sdr--1.20220707-064959.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  7 07:16 default-sdr--1.20220707-071413.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  7 08:11 default-sdr--1.20220707-080747.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  7 14:28 default-sdr--1.20220707-142333.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  7 16:21 default-sdr--1.20220707-161837.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  7 18:27 default-sdr--1.20220707-182157.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  7 19:32 default-sdr--1.20220707-192708.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  7 20:23 default-sdr--1.20220707-202211.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  7 22:11 default-sdr--1.20220707-220718.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  7 23:21 default-sdr--1.20220707-232020.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  8 00:12 default-sdr--1.20220708-000646.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  8 03:44 default-sdr--1.20220708-034243.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  8 05:54 default-sdr--1.20220708-055325.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  8 08:26 default-sdr--1.20220708-082141.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  8 10:09 default-sdr--1.20220708-100525.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  8 16:27 default-sdr--1.20220708-162149.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  8 18:18 default-sdr--1.20220708-181347.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  9 05:10 default-sdr--1.20220709-050836.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  9 06:24 default-sdr--1.20220709-061850.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  9 23:27 default-sdr--1.20220709-232211.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul 12 01:47 default-sdr--1.20220712-014416.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul 12 17:23 default-sdr--1.20220712-172152.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul 12 20:12 default-sdr--1.20220712-201138.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul 13 17:57 default-sdr--1.20220713-175231.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul 14 02:01 default-sdr--1.20220714-015439.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul 18 09:30 default-sdr--1.20220718-092606.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul 19 19:56 default-sdr--1.20220719-195320.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul 29 18:38 default-sdr--1.20220729-183354.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Aug  1 23:13 default-sdr--1.20220801-231231.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Aug  2 23:24 default-sdr--1.20220802-231919.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Aug  3 22:34 default-sdr--1.20220803-223220.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Aug  7 16:51 default-sdr--1.20220807-164830.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Aug  9 03:56 default-sdr--1.20220809-035239.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Aug  9 21:42 default-sdr--1.20220809-214040.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  6 21:12 default-sdr--2.20220706-210701.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  6 22:09 default-sdr--2.20220706-220717.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  6 22:54 default-sdr--2.20220706-225201.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  6 23:15 default-sdr--2.20220706-231027.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  7 00:25 default-sdr--2.20220707-002243.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  7 01:26 default-sdr--2.20220707-012358.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  7 03:06 default-sdr--2.20220707-030451.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  7 03:29 default-sdr--2.20220707-032202.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  7 03:34 default-sdr--2.20220707-033000.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  7 04:15 default-sdr--2.20220707-041226.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  7 06:56 default-sdr--2.20220707-065132.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  7 07:16 default-sdr--2.20220707-071541.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  7 08:11 default-sdr--2.20220707-080917.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  7 09:57 default-sdr--2.20220707-095255.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  7 10:57 default-sdr--2.20220707-105456.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  7 12:27 default-sdr--2.20220707-122628.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  7 14:28 default-sdr--2.20220707-142522.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  7 16:22 default-sdr--2.20220707-162013.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  7 18:27 default-sdr--2.20220707-182328.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  7 20:28 default-sdr--2.20220707-202351.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  7 23:27 default-sdr--2.20220707-232157.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  8 00:12 default-sdr--2.20220708-000821.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  8 03:49 default-sdr--2.20220708-034419.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  8 05:59 default-sdr--2.20220708-055503.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  8 08:27 default-sdr--2.20220708-082328.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  8 12:24 default-sdr--2.20220708-122129.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  8 16:27 default-sdr--2.20220708-162404.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  8 18:18 default-sdr--2.20220708-181528.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  9 05:15 default-sdr--2.20220709-051019.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  9 06:24 default-sdr--2.20220709-062023.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  9 14:11 default-sdr--2.20220709-140706.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  9 15:12 default-sdr--2.20220709-150744.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  9 15:42 default-sdr--2.20220709-153901.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul  9 23:27 default-sdr--2.20220709-232340.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul 10 06:10 default-sdr--2.20220710-060948.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul 12 01:47 default-sdr--2.20220712-014458.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul 12 13:44 default-sdr--2.20220712-134056.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul 12 17:23 default-sdr--2.20220712-172239.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul 12 20:12 default-sdr--2.20220712-201016.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul 13 17:57 default-sdr--2.20220713-175359.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul 13 18:32 default-sdr--2.20220713-183047.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul 14 01:40 default-sdr--2.20220714-013429.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul 14 02:47 default-sdr--2.20220714-024503.core.RP.lxcdump.tar.lz4
-rw-r--r--. 1 root root         0 Jul 18 09:30 default-sdr--2.20220718-092702.core.RP.lxcdump.tar.lz4

 

 

What is odd, sometimes when a vXR freezes, then I unfreeze it, admin, run, and check `df -H` the partition is no where near what the terminal is complaining about. So I assume there is some mechanism that is cleaning up after itself, and some times it cant?
Some times I even see these error messages, and at this point I just need to reload (reboot) the vXR.

 

 

[22048.168007] BUG: soft lockup - CPU#1 stuck for 22s! [ixdb_gc:8898]
[22048.185007] BUG: soft lockup - CPU#2 stuck for 22s! [vpe_main:8156]

 

Its not clear to me what is filling up the partition or if its related to why the vXR freezes. I can only assume its a log that is filling up from my poor configurations. 
How can I figure out what is filling up? I COULD build a cronjob to clean up whatever fill is filling up, but what is the correct way to fix this? 
Anyone have any ideas

EDIT: 
The workaround I found was to delete the default-sdr--2.20220706-184037.core.RP.lxcdump.tar.lz4 file first. It was huge.
Then I noticed many more `default-dr--*.lz4` files were being created. So I deleted the glob, now `df -H` is back to normal. I THINK these files are made by the XR to pass off to support when there is a problem, I see a message about them after I reboot the system. Is there a way to stop the system from creating these files? 

Labels:
0 Helpful
3 Replies 3

balaji.bandi
Hall of Fame
Hall of Fame

‎08-09-2022 11:48 PM

XRV is very high resource required image for the Lab, what kind of compute you have ?

if you looking only do some labs, I go back to 6.X image and they work  as expected (until you see any specific reason to use 7.3 code)

https://community.cisco.com/t5/service-providers-knowledge-base/disk-utilization-characteristic-data-for-ios-xr-image/ta-p/3162293

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

0 Helpful

davehouser1
Level 1
Level 1
In response to balaji.bandi

‎08-10-2022 04:27 AM

I don't understand what I am supposed to be looking at from the link you shared. Is it resource requirements? If so its not clear on what the requirements are for a vXR 7.3.1 are from that link.

I am currently running GNS as a vmware appliance. I have given the appliance a lot of resources (30 CPU cores, and 128 GB of memory) and I can give more as the guest is running on very large hosts. However these are shares and the vm never gets close to using all the resources. Memory and CPU stay at around 30-40% for the GNS3 appliance. Re: Disk space the gns3 guest has 500GB to use which is not being close to being used. The GNS3 configuration for each vXR is using 1vCPU and 17000MB of ram. These settings are the default and what were recommended to be used by guides on the internet. 

Therefore I don't believe this is a CPU or RAM issue. Its obvious the partition is filling up from what I shared above, and its getting filled up by those .lz4 files. The vXR's work perfectly fine with no problems, until that partition gets filled.

Finally, I must use vXR 7.3.1 image, its a requirement for the project I am on. Cannot go down to a earlier version.


Again, what are those files for? And how can I stop the system from building them or clean them up?

0 Helpful

tkarnani
Cisco Employee
Cisco Employee
In response to davehouser1

‎08-11-2022 04:12 AM - edited ‎08-11-2022 04:18 AM

 

BUG: soft lockup - CPU#1 stuck for

the error above indicates that the watchdog/nmi process is waiting for the CPU to respond. either the cpu is not responding or the kernel locked up, when this happens the device will write a core file to the disk and reload.

Those core files indicate a crash, you can delete them via the following command

admin

delete harddisk:*.core*

delete harddisk:*.lz4

 

the minimum requirements for xrv9k are listed here

https://www.cisco.com/c/en/us/td/docs/routers/virtual-routers/xrv9k-73x/release/notes/b-release-notes-xrv9000-r731.html

 

 

Customers Also Viewed These Support Documents