ISE 3.2 patch 4 Configuration Backup via GUI failed

ajc · ‎11-21-2023

Configuration backup from ISE 2.7 p10 always worked fine via GUI for FTP. Now I am running 3.2 p4 and having issues with such simple process using FTP.

If I do from CLI : show repository bckREPO, I get all the content of the FTP server directory with no issues.

Next the logs from the ISE Node, I will give a try via CLI. Hoping this is not ANOTHER bug.

2023-11-21T11:00:38.228094-05:00 ISENODE ADE-SERVICE[1691]: [6059]:[info] backup-restore:backup: br_backup.c[1129] [system]: Backup password provided by user
2023-11-21T11:02:33.584390-05:00 ISENODE root: [backup.sh] backup file ISE3-2-P4-CFG10-231121-1053.tar.gpg successfully created
2023-11-21T11:02:33.595425-05:00 ISENODE ADE-SERVICE[1691]: [6059]:[info] backup-restore:backup: br_backup.c[278] [system]: No post-backup entry in the manifest file for ise
2023-11-21T11:02:33.595507-05:00 ISENODE ADE-SERVICE[1691]: [6059]:[info] backup-restore:backup-logs: br_backup.c[141] [system]: backup in progress:Moving Backup file to the repository...75▒ompleted
2023-11-21T11:02:33.645355-05:00 ISENODE ADE-SERVICE[1691]: [6059]:[info] transfer: cars_xfer.c[236] [system]: ftp copy out of /opt/backup/backup-ISE3-2-P4-1700582018/ISE3-2-P4-CFG10-231121-1053.tar.gpg requested
2023-11-21T11:07:40.701113-05:00 ISENODE CARSSetup[6059]: ADEAUDIT 2012, type=BACKUP, name=BACKUP FAILED, username=admin, cause=A backup has failed, adminipaddress=172.28.64.26, interface=GUI, detail=Backup failed: copy ISE3-2-P4-CFG10-231121-1053 out to repository bckREPO failed
2023-11-21T11:07:40.701191-05:00 ISENODE CARSSetup[6059]: ADEAUDIT 3013, type=BACKUP, name=BACKUP FAILED, username=system, cause=Error during backup, adminipaddress=127.0.0.1, interface=CLI, detail=Backup failed: copy ISE3-2-P4-CFG10-231121-1053 out to repository bckREPO failed
2023-11-21T11:07:40.701566-05:00 ISENODE ADE-SERVICE[1691]: [6059]:[info] locks:file: lock.c[359] [system]: deleted progress file /tmp/adeos-backuprestore-inprogress
2023-11-21T11:07:40.701664-05:00 ISENODE ADE-SERVICE[1691]: [6059]:[info] locks:file: lock.c[496] [system]: deleted backup-restore progress file: /tmp/adeos-backuprestore-inprogress
2023-11-21T11:07:40.701880-05:00 ISENODE ADE-SERVICE[1691]: [6059]:[info] backup-restore:history: br_history.c[295] [system]: Current Date : Tue Nov 21 11:07:40 EST 2023
2023-11-21T11:07:40.702297-05:00 ISENODE ADE-SERVICE[1691]: [6059]:[error] backup-restore:backup: br_backup.c[1220] [system]: Backup failed: copy ISE3-2-P4-CFG10-231121-1053 out to repository bckREPO failed

ajc · ‎11-21-2023

UPDATE: It does not even work from CLI,

ISE NODE/admin#backup ISE3-2P4 repository bckREPO ise-config encryption-key plain ISE1234
Warning: Do not use CTRL+C or close this terminal window until the backup is completed.
% Internal CA Store is not included in this backup. It is recommended to export it using "application configure ise" CLI command
% Creating backup with timestamped filename: ISE3-2P4-CFG10-231121-1155.tar.gpg
% backup in progress: Starting Backup...10% completed
% backup in progress: Validating ISE Node Role...15% completed
% backup in progress: Backing up ISE Configuration Data...20% completed
% backup in progress: Backing up ISE Indexing Engine Data...45% completed
% backup in progress: Backing up ISE Logs...50% completed
% backup in progress: Completing ISE Backup Staging...55% completed
% backup in progress: Backing up ADEOS configuration...55% completed
% backup in progress: Moving Backup file to the repository...75% completed
File transfer error
ISENODE/admin#

davidgfriedman · ‎11-21-2023

With these two lines indicating you were FTP'ing for 5 minutes 7 seconds, my first thought (since you can read the repository) would be to think you ran out of disk space on your target FTP server's filesystem. Did you double check your target repo filesystem's free disk space?

2023-11-21T11:02:33.645355-05:00 ISENODE ADE-SERVICE[1691]: [6059]:[info] transfer: cars_xfer.c[236] [system]: ftp copy out of /opt/backup/backup-ISE3-2-P4-1700582018/ISE3-2-P4-CFG10-231121-1053.tar.gpg requested
2023-11-21T11:07:40.701113-05:00 ISENODE CARSSetup[6059]: ADEAUDIT 2012, type=BACKUP, name=BACKUP FAILED, username=admin, cause=A backup has failed, adminipaddress=172.28.64.26, interface=GUI, detail=Backup failed: copy ISE3-2-P4-CFG10-231121-1053 out to repository bckREPO failed

Damien Miller · ‎11-21-2023

Does "show repo bckREPO" display the remote folder contents?

If you're using SFTP, you need to make sure you have imported the server keys before you can use the repo. The other thing to check is to make sure the user configured has write access to the FTP server folder.

ajc · ‎11-21-2023

Hi Damien,

Answer is yes, see the output from show repository bckREPO.

ISENODE/admin#show repository bckREPO
asdm-7122.bin
CERTIFICATES-2.7DEPLOYMENT
ISE-29-SIZE-BACKUP.png
Meraki AP power capture 1.pcapng
Meraki AP power capture 2.pcapng
ISENODE/admin#

I have an ISE 2.7 deployment and I can do backup anytime with no issues. This particular ISE 3.2 p4 node was built using a restored configuration from our ISE 2.7 p10 deployment. I tested this 3.2 p4 Node for EAP-TLS, MAB, PEAP, Guest Portal and ALL of them worked fine. It is just this simple FTP configuration backup process. I will have to open a TAC Case.

I am using an FTP server app on my computer as repository with enough space in the disk (246 GB). It works fine for ISE 2.7 as I mentioned above.

This is not even production implementation, only running basic tests BEFORE moving into production. I will give a try to SFTP.

ajc · ‎11-23-2023

I suspect there is a BUG because the REPOSITORIES from my ISE 2.7 p10 configuration ARE NOT replicated into the actual configuration of the original ISE 3.2 p4 server when you check for that information via CLI. I mean, all my repositories from ISE 2.7 p10 were NOT present when I checked for that via CLI / show running in the ISE 3.2 p4 with the restored config. ONLY the "localdisk" repository was there.

I reconfigured back those repositories via GUI and confirmed they were present via CLI but still FTP failed.

Interesting to notice that I created a fresh installed ISE 3.2 p4 VM and created a config backup via FTP to the same repository and it worked fine. That tells me the restore ISE 2.7 p10 configuration into the original ISE 3.2 p4 VM caused some issues with the FTP repository operation. I have not tried SFTP because I always want to know the reason because something that worked before now fails.

ajc · ‎11-23-2023

UPDATE: As expected there is an issue with the ISE 2.7 p10 config backup restore into 3.2 p4. I finally found the reason because FTP does not work after restoring my 2.7 p10 config backup into ISE 3.2 p4. THE FTP REPOSITORY configuration is messed up and it does not work anymore via GUI for ISE 3.2 p4.

The config backup restore process only puts back the localdisk repository and anything else is missed in the CLI show running output even though it is displayed via GUI. Removing those GUI FTP repositories and adding them back via does not solve the issue, adding new ones via GUI does not work as well.

I repeated the FTP config backup process after adding another repository via CLI (that would not be added to the GUI list as per following output) and it worked.

ISENODE/admin(config-repository-testing1)#url ftp: //172.22.80.143/
% Warning: Repositories configured from CLI cannot be used from the ISE web
UI and are not replicated to other ISE nodes. If this repository is not
created in the ISE web UI, it will be deleted when ISE services restart.
ISENODE/admin(config-repository-testing1)#user testing password ?
Description: Configure repository password for access
Possible completions:
hash Specifies an ENCRYPTED (hashed) password will follow
plain Specifies an UNENCRYPTED plain text password will follow
ISENODE/admin(config-repository-testing1)#user testing password plain testing ?
Possible completions:
<cr>
ISENODE/admin(config-repository-testing1)#user testing password plain testing
ISENODE/admin(config-repository-testing1)#end
ISENODE/admin#show run
interface GigabitEthernet 0
ip address 10.10.10.12 255.255.255.0
ipv6 enable
ipv6 address autoconfig
!
!
repository localdisk
url disk: /
!
repository meraki (THIS REPOSITORY WAS ADDED VIA GUI AND THE FTP CONFIG BACKUP PROCESS FAILED)
url ftp: //10.10.10.143/
user meraki password hash **********
!
repository testing1 (THIS REPOSITORY WAS ADDED VIA CLI AS INDICATED AT THE BEGINNING OF THIS OUTPUT - WORKED)
url ftp: //10.10.10.143/
user testing password hash **********
!
ISENODE/admin#show repository testing1
6 [3706637]:[info] transfer: cars_xfer.c[329] [system]: ftp dir of repository testing1 requested
7 [3706637]:[debug] transfer: cars_xfer_util.c[2315] [system]: ftp get dir for repos testing1
7 [3706637]:[debug] transfer: cars_xfer_util.c[2328] [system]: initializing curl
7 [3706637]:[debug] transfer: cars_xfer_util.c[2340] [system]: full url is ftp://172.22.80.143/
7 [3706637]:[debug] transfer: cars_xfer_util.c[2214] [system]: initializing curl
7 [3706637]:[debug] transfer: cars_xfer_util.c[2228] [system]: full url is ftp://172.22.80.143/Defaultselfsignedservercerti.pem
7 [3706637]:[debug] transfer: cars_xfer_util.c[2248] [system]: res: 78
7 [3706637]:[debug] transfer: cars_xfer_util.c[2214] [system]: initializing curl
7 [3706637]:[debug] transfer: cars_xfer_util.c[2228] [system]: full url is ftp://172.22.80.143/Defaultselfsignedservercerti.pvk
7 [3706637]:[debug] transfer: cars_xfer_util.c[2248] [system]: res: 78
7 [3706637]:[debug] transfer: cars_xfer_util.c[2214] [system]: initializing curl
7 [3706637]:[debug] transfer: cars_xfer_util.c[2228] [system]: full url is ftp://172.22.80.143/Defaultselfsignedservercerti.zip
7 [3706637]:[debug] transfer: cars_xfer_util.c[2248] [system]: res: 78
Defaultselfsignedservercerti.pem
Defaultselfsignedservercerti.pvk
Defaultselfsignedservercerti.zip
7 [3706637]:[debug] transfer: cars_xfer.c[378] [system]: freed file list
ISENODE/admin#
ISENODE/admin#
ISENODE/admin#backup test10 repository testing1 ise-config encryption-key plain ABCD123admin
Warning: Do not use CTRL+C or close this terminal window until the backup is completed.
% Internal CA Store is not included in this backup. It is recommended to export it using "application configure ise" CLI command
% Creating backup with timestamped filename: test10-CFG10-231123-1300.tar.gpg
% backup in progress: Starting Backup...10% completed
% backup in progress: Validating ISE Node Role...15% completed
% backup in progress: Backing up ISE Configuration Data...20% completed
% backup in progress: Backing up ISE Indexing Engine Data...45% completed
% backup in progress: Backing up ISE Logs...50% completed
% backup in progress: Completing ISE Backup Staging...55% completed
% backup in progress: Backing up ADEOS configuration...55% completed
% backup in progress: Moving Backup file to the repository...75% completed
% backup in progress: Completing Backup...100% completed
ISENODE/admin#
ISENODE/admin#show repository testing1
Defaultselfsignedservercerti.pem
Defaultselfsignedservercerti.pvk
Defaultselfsignedservercerti.zip
test10-CFG10-231123-1300.tar.gpg (BACKUP SUCCESSFULLY TRANSFERRED TO THE FTP REMOTE REPOSITORY)
ISENODE/admin#

CLI Configuration backup process using an FTP repository created also via CLI is displayed in the GUI even though that repository does not exists there.

GUI listed repository does not show the CLI created FTP repository.