topic Re: ISE 2.1.0.474 backup issue hangs at 20% in Network Access Control

ISE 2.1.0.474 backup issue hangs at 20%

rikerrobert — Fri, 21 Feb 2020 18:45:36 GMT

I am backing up ISE via ise-config backup of the PAN. I tried doing the backup from the web and then moved to the CLI to get more info. I attempted an operational backup and that worked without issue to an FTP repository. I re-attempted the configuration backup with the same repository and it failed again. No logs to really speak that I could find. Being an R&S guy, I checked debugs for backup-restore backup and was presented with the below when I kicked off the config backup again.

wilisep1/admin# backup BACKUP repository UPGRADE ise-config encryption-key plain P@ssw0rd
6 [15324]:[info] backup-restore:backup: br_history.c[521] [admin]: ISE backup/restore initiated by CLI as ise.br.status is not 'in-progress' in /tmp/ise-cfg-br-flags
7 [15324]:[debug] backup-restore:backup: br_backup.c[585] [admin]: initiating backup BACKUP to repos UPGRADE
7 [15324]:[debug] backup-restore:backup: br_backup.c[629] [admin]: no staging url defined, using local space
7 [15324]:[debug] backup-restore:backup: br_backup.c[54] [admin]: flushing the staging area
7 [15324]:[debug] backup-restore:backup: br_backup.c[658] [admin]: creating /opt/backup/backup-BACKUP-1518626716
7 [15324]:[debug] backup-restore:backup: br_backup.c[662] [admin]: creating /opt/backup/backup-BACKUP-1518626716/backup/cars
% Internal CA Store is not included in this backup. It is recommended to export it using "application configure ise" CLI command
% Creating backup with timestamped filename: BACKUP-CFG10-180214-1045.tar.gpg
% backup in progress: Starting Backup...10% completed
7 [15324]:[debug] backup-restore:backup: br_backup.c[726] [admin]: creating /opt/backup/backup-BACKUP-1518626716/backup/ise
7 [15324]:[debug] backup-restore:backup: br_backup.c[752] [admin]: calling script /opt/CSCOcpm/bin/isecfgbackup.sh
% backup in progress: Validating ISE Node Role...15% completed
%Error: Aborting ISE configuration backup as it is not supported on this node type.
sed: couldn't flush /tmp/sed5CRQRo: No space left on device
% Application backup error

The issue I am running into is the last few lines. This is an ISE appliance and not a VM. Any insight is helpful.

Thanks,

Rob

Re: ISE 2.1.0.474 backup issue hangs at 20%

Mark Elsen — Thu, 15 Feb 2018 06:06:14 GMT

- Check available disk space on your ISE server using : show disks

Re: ISE 2.1.0.474 backup issue hangs at 20%

rikerrobert — Thu, 15 Feb 2018 16:09:02 GMT

wilisep1/admin# show disks

Internal filesystems:
/ : 43% used ( 6009820 of 14987616)
/dev : 0% used ( 0 of 8110824)
/dev/shm : 0% used ( 0 of 8121068)
/run : 10% used ( 795620 of 8121068)
/sys/fs/cgroup : 0% used ( 0 of 8121068)
/storedconfig : 2% used ( 1585 of 95054)
/boot : 23% used ( 104933 of 487634)
/tmp : 100% used ( 1966668 of 1983056)
/opt : 15% used ( 75886760 of 549840808)
/run/user/440 : 0% used ( 0 of 1624216)
/run/user/301 : 0% used ( 0 of 1624216)
/run/user/0 : 0% used ( 0 of 1624216)
/run/user/303 : 0% used ( 0 of 1624216)
/run/user/321 : 0% used ( 0 of 1624216)
warning - /tmp is 100% used (1966668 of 1983056)

As you can see the /tmp file is full. I haven't found a way to clear it.

Re: ISE 2.1.0.474 backup issue hangs at 20%

Mark Elsen — Thu, 15 Feb 2018 17:30:40 GMT

- Yeah, I always find it poor from Cisco, to use linux (open source) and not give customers the benefit of it (root access and management would solve this in 2-secs). It's agains open-source philosophy! Anyway following the link below ; there is a Rootpatch for ISE, unfortunately you will have to get it from TAC :

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvc95735/?referring_site=bugquickviewredir

Re: ISE 2.1.0.474 backup issue hangs at 20%

ibrahim.sadiq — Wed, 21 Feb 2018 11:05:01 GMT

Does this bug applicable only for the physical appliances (as it is mentioned for Cisco ISE 3300) or even for the Virtual Appliances?

Cos, we have a similar issue in a Virtual Appliance.

Re: ISE 2.1.0.474 backup issue hangs at 20%

Jatin Katyal — Wed, 21 Feb 2018 11:40:27 GMT

Yes, it is applicable for ISE running on VM's too.

"Show disks" on the ISE CLI will tell you more how the disk looks like.

Look for this line in the output.

/tmp : 100% used

Once you confirm that, open a TAC case to get this cleared.

Re: ISE 2.1.0.474 backup issue hangs at 20%

ibrahim.sadiq — Wed, 21 Feb 2018 13:44:34 GMT

Looks we have sufficient free space available

ise/admin# show disks

Internal filesystems:
/ : 18% used ( 2509964 of 14987616)
/dev : 0% used ( 0 of 8124524)
/dev/shm : 0% used ( 0 of 8134196)
/run : 1% used ( 25384 of 8134196)
/sys/fs/cgroup : 0% used ( 0 of 8134196)
/boot : 24% used ( 107955 of 487634)
/storedconfig : 2% used ( 1582 of 95054)
/tmp : 1% used ( 6500 of 1983056)
/opt : 23% used ( 129154740 of 593333176)
/run/user/440 : 0% used ( 0 of 1626840)
/run/user/301 : 0% used ( 0 of 1626840)
/run/user/0 : 0% used ( 0 of 1626840)
/run/user/304 : 0% used ( 0 of 1626840)
/run/user/303 : 0% used ( 0 of 1626840)
/run/user/322 : 0% used ( 0 of 1626840)
/run/user/321 : 0% used ( 0 of 1626840)
all internal filesystems have sufficient free space

We get the following error in the backup status;

ise/admin# show backup status
%% Configuration backup status
%% ----------------------------
% backup name: ISE_Config_Backup
% repository: SFTP_Repository
% start date: Wed Feb 21 10:47:18 CET 2018
% scheduled: no
% triggered from: Admin web UI
% host: XYZ.ISE-INTERNAL.NET
% status: DB BACKUP FAILED : . Backup aborted

Re: ISE 2.1.0.474 backup issue hangs at 20%

Jatin Katyal — Wed, 21 Feb 2018 14:48:57 GMT

then your problem is different...what code ( including patch) you have running on ISE.

Check the condition of the defect and see if the symptoms of your problem matching it.

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvc79739/?reffering_site=dumpcr