ise backup stuck at 20%

[muhsi_2015]

Hi,

/admin# sh backup status
%% Configuration backup status
%% ----------------------------
% backup name: repo
% repository: repo
% start date: Thu Feb 16 16:37:09 AST 2017
% scheduled: no
% triggered from: Admin web UI
% host: ise01.test.local
% status: Backup is in progress...
% progress %: 20
% progress message: Backing up ISE Configuration Data

Cisco Application Deployment Engine OS Release: 2.4
ADE-OS Build Version: 2.4.0.147
ADE-OS System Architecture: x86_64

Copyright (c) 2005-2014 by Cisco Systems, Inc.
All rights reserved.
Hostname: ise01

Version information of installed applications
---------------------------------------------

Cisco Identity Services Engine
---------------------------------------------
Version : 2.0.1.130
Build Date : Thu Mar 3 05:38:48 2016
Install Date : Tue May 17 19:15:41 2016

Please help

[Gagandeep Singh]

Hi,

The option is to reload the server once, if it got hung. I would suggest to do that in off hours.

There is another process of taking root access and kill the specific process associated with backup.

That requires TAC case to be open.

Long story short, would suggest to reload it once, that should resolve this issue.

Regards

Gagan

PS: rate if it helps!!!!!

[muhsi_2015]

Hi,

It was related with  sftp server , when I change sftp server it did work properly .

But the thing it was not throwing any error 

Thanks

[jack samuel]

Dear Gagandeep

My backup fails in 75% below are the logs from CLI by sh logging command.

2017-02-14T15:00:32.255654+04:00 ISE01 ADE-SERVICE[2017]: [6441]:[info] transfer: cars_xfer.c[142] [system]: sftp copy out of /opt/backup/backup-Backup_14feb-1487069877/
Backup_14feb-CFG10-170214-1457.tar.gpg requested
2017-02-14T15:00:32.255678+04:00 ISE01 ADE-SERVICE[2017]: [6441]:[info] transfer: cars_xfer_util.c[515] [system]: resolved server to 172.24.4.249
2017-02-14T15:00:32.279175+04:00 ISE01 ADE-SERVICE[2017]: [6441]:[info] transfer: sftp_handler.c[437] [system]: DEBUG: local user UID: 300 sftp_run_parent FD: 692 remote host
: 172.24.4.249 remote user: abc command: put /opt/backup/backup-Backup_14feb-1487069877/Backup_14feb-CFG10-170214-1457.tar.gpg /./Backup_14feb-CFG10-170214-1457.ta
r.gpg
2017-02-14T15:01:01.550278+04:00 ISE01 root: info:[application:operation:isehourlycron.sh] Waiting up to 20 seconds for lock: APP_BACKUP to complete
2017-02-14T15:01:19.115669+04:00 ISE01 ADE-SERVICE[2017]: [2791]:[error] cdp:infra: cdp-parse.c[824] [system]: parse_cdp_packet(): cdp checksum error, dropping packet
2017-02-14T15:01:21.552824+04:00 ISE01 root: info:[application:operation:isehourlycron.sh] Database is still locked by lock: APP_BACKUP. Aborting. Please try it later
2017-02-14T15:01:21.554491+04:00 ISE01 root: info:[application:operation:isehourlycron.sh] % Error: Another ISE DB process (APP_BACKUP) is in progress, cannot perform cleanup
at this time
2017-02-14T15:01:32.976293+04:00 ISE01 ADE-SERVICE[2017]: [6441]:[error] transfer: sftp_handler.c[232] [system]: sftp_select Error: timeout!
2017-02-14T15:01:33.185669+04:00 ISE01 ADE-SERVICE[2017]: [6441]:[error] backup-restore:backup: br_backup.c[860] [system]: Backup failed: copy Backup_14feb-CFG10-170214-
1457 out to repository SFTP_BACKUP failed
2017-02-14T15:01:33.185787+04:00 ISE01 ADEOSJAVAAPI[6441]: ADEAUDIT 2012, type=BACKUP, name=BACKUP FAILED, username=admin, cause=A backup has failed, adminipaddress=10.241.4.
249, interface=GUI, detail=Backup failed: copy Backup_14feb-CFG10-170214-1457 out to repository SFTP_BACKUP failed
2017-02-14T15:01:33.256509+04:00 ISE01 ADEOSJAVAAPI[6441]: ADEAUDIT 3013, type=BACKUP, name=BACKUP FAILED, username=system, cause=Error during backup, adminipaddress=127.0.0.
1, interface=CLI, detail=Backup failed: copy Backup_14feb-CFG10-170214-1457 out to repository SFTP_BACKUP failed
2017-02-14T15:01:33.256624+04:00 ISE01 ADE-SERVICE[2017]: [6441]:[info] backup-restore:backup: br_history.c[454] [system]: updating /tmp/ise-cfg-br-flags with status: complet
e and message: Backup failed: copy to repository failed

[jack samuel]

which sftp server you are using

[muhsi_2015]

solarwind 

[jack samuel]

i am also using the same, any specific settings you will suggest for sftp server if you have done,

[Gagandeep Singh]

There are few things needs to be check in case of SFTP:

1. Does user has read/write privilege on SFTP server?

2. Does it support DH 14?

3. Is SCP enabled on the SFTP?

4. Does your password contains more than 16 characters? If yes, please make it
small as ISE has a bug filed with this issue.

5. Are you able to SSH to this SFTP from ISE?

If all above sanity checks are correct, then there is nothing in terms of configuration change required on ISE or SFTP.

Would request you to reload the ISE server just to release the hung backup or there are some files needs to be removed from ROOT.

like

/tmp/adeos-backup.semafile
/tmp/cars-snmapcfg.lock

Which I wouldn't recommend for now.

Regards

Gagan

PS: rate if it helps!!!!!

[jack samuel]

Dear gagan

thanks for the reply, as i was away on holidays couldn't reply you.

• yes the user has privileges
• how can verify DH14 in solar winds sftp
• yes scp is enabled
• NO my password is very small ( 4 digits)
• yes i m able to do ssh to the sftp from ise

How i can remove those temp file can you guide me the path to remove those. and how we can identify that these are the only temp files that has to be cleared,i am asking you becz i want to learn how troubleshooting can be done to the deep level

/tmp/adeos-backup.semafile
/tmp/cars-snmapcfg.lock

thanks

[Gagandeep Singh]

For these files to remove, you have to open TAC case for root access.

Did you try reload once on the Server.

Regards

Gagan

[jack samuel]

Dear Gagandeep,

yes i try to reload and the backup worked for me luckily, how i can verify the below

• how can verify DH14 in solar winds sftp

thanks

[Gagandeep Singh]

Not sure but it should be in some settings on SFTP where it says FIPS compliant with DH 14 support.

Regards

Gagan

 rate helpful posts!!!