topic Re: Cisco ISE Backup Fails -Repository validation: confd implementatio in Network Access Control

Cisco ISE Backup Fails -Repository validation: confd implementations

rocteam — Mon, 04 Nov 2024 07:58:32 GMT

Hello,

The ISE is set up to Backup automatically:
Configuration Data Backup - Daily
Operational Data Backup - Weekly
Repository - NFS

The Daily works fine, but the Weekly Operational Backup fails witht the message:
Repository (Rep_Name) validation failed due to error: confd implementatons

They both use the same Repo, that is why I do not know why it happens.

Thanks in advance for help.

Re: Cisco ISE Backup Fails -Repository validation: confd implementatio

Mark Elsen — Mon, 04 Nov 2024 10:31:58 GMT

- What happens if you use a different Repo for both ,

Re: Cisco ISE Backup Fails -Repository validation: confd implementatio

Aref Alsouqi — Mon, 04 Nov 2024 10:48:58 GMT

Are you using the same folder for both backups? if not, maybe it is a permission issue on the operational backup folder?

Re: Cisco ISE Backup Fails -Repository validation: confd implementatio

rocteam — Mon, 04 Nov 2024 11:17:46 GMT

Thanks for the reply.

We are using the same repository and folder for both backups.

Re: Cisco ISE Backup Fails -Repository validation: confd implementatio

Aref Alsouqi — Mon, 04 Nov 2024 11:53:17 GMT

Interesting. What happens if you try to trigger the operational backup manually? would it still fail? if not, I would try to delete the operational backup schedule and recreate it.

Re: Cisco ISE Backup Fails -Repository validation: confd implementatio

rocteam — Mon, 04 Nov 2024 13:00:27 GMT

Manually via GUI fails witht error: copy to repository failed.
Via CLI ends with success, but this outcome:

Warning: Do not use CTRL+C or close this terminal window until the backup is completed.

% Creating backup with timestamped filename: Operational_Data_Backup_260924-OPS10-240926-1525.tar.gpg

% backup in progress: Starting Backup...10% completed

% backup in progress: starting dbbackup using expdp.......20% completed

ttBulkCp: Error received while connecting to data store -- 08S01: [TimesTen][TimesTen 18.1.4.15.0 ODBC Driver][TimesTen]TT0799: Unable to connect to daemon; check daemon status -- file "dbConnect.c", lineno 2537, procedure "sbDbConnect" (TimesTen error code = 799).

ttSchema: Error received while connecting to data store -- 08S01: [TimesTen][TimesTen 18.1.4.15.0 ODBC Driver][TimesTen]TT0799: Unable to connect to daemon; check daemon status -- file "dbConnect.c", lineno 2537, procedure "sbDbConnect" (TimesTen error code = 799).

% backup in progress: starting cars logic.......50% completed

% backup in progress: Moving Backup file to the repository...75% completed

% backup in progress: Completing Backup...100% completed

Re: Cisco ISE Backup Fails -Repository validation: confd implementatio

Mark Elsen — Mon, 04 Nov 2024 13:34:37 GMT

- Check if this can help : https://community.cisco.com/t5/network-access-control/operational-backup-failing-ise-2-7/m-p/4812262/highlight/true#M581095

Re: Cisco ISE Backup Fails -Repository validation: confd implementatio

rocteam — Mon, 04 Nov 2024 13:54:51 GMT

thanks for the reply.
It can be the cause. I also tried your first suggestions, did a backup with different repos, the result was the same.
For the restart, we will need a window from customer. I will let you know the outcome once done. May take some time 🙂

Re: Cisco ISE Backup Fails -Repository validation: confd implementatio

Aref Alsouqi — Mon, 04 Nov 2024 14:41:10 GMT

I would try to delete and recreate the scheduled backup and see if that helps.

Re: Cisco ISE Backup Fails -Repository validation: confd implementatio

rocteam — Tue, 05 Nov 2024 08:13:53 GMT

I can not delete the scheduled backup, tried to deactivate and activate again - same error.

Re: Cisco ISE Backup Fails -Repository validation: confd implementatio

Mark Elsen — Tue, 05 Nov 2024 09:07:24 GMT

- FYI : https://www.cisco.com/c/en/us/support/docs/security/identity-services-engine/212594-debugs-to-troubleshoot-on-ise.html
When collecting all log files you may also have confd.log or related name ; check it's content

Re: Cisco ISE Backup Fails -Repository validation: confd implementatio

Aref Alsouqi — Tue, 05 Nov 2024 09:19:19 GMT

Does it return any specific error when you try to delete it?

Re: Cisco ISE Backup Fails -Repository validation: confd implementatio

rocteam — Fri, 08 Nov 2024 07:21:46 GMT

Sorry for late reply.
I just do not see any option in GUI to delete a scheduled Backup. I can only deactivate it.

Re: Cisco ISE Backup Fails -Repository validation: confd implementatio

Aref Alsouqi — Fri, 08 Nov 2024 09:55:50 GMT

You're right and that was my bad, I got confused myself with this, sorry about that. At this point probably I would try to deactivate the backup and try to delete the repository and recreate it. Not sure though if this would help but I think it does worth the try.

Re: Cisco ISE Backup Fails -Repository validation: confd implementatio

rocteam — Tue, 12 Nov 2024 08:56:23 GMT

No problem 🙂
I tried to deactivate schedule backup - delete repo and add new - same result

Re: Cisco ISE Backup Fails -Repository validation: confd implementatio

Aref Alsouqi — Tue, 12 Nov 2024 09:08:42 GMT

I would suggest to get TAC involved before you restart the node as that would allow them to see the issue as it's happening and trying to find out the root cause.

Re: Cisco ISE Backup Fails -Repository validation: confd implementatio

rocteam — Tue, 12 Nov 2024 09:11:09 GMT

Yes, I already opened a TAC Case and uploaded logs, dumps, etc.
Will let you know if the problem is solved.

Re: Cisco ISE Backup Fails -Repository validation: confd implementatio

rocteam — Wed, 27 Nov 2024 06:57:50 GMT

I found the solution together with Cisco TAC!
Root cause was the firewall blocking connections between repositories and monitoring nodes.
In ISE when using GUI or scheduled backups:
- The config backups are done from admin nodes
- The operational backups are done from monitoring nodes
The firewall was not set up to allow connections from monitoring nodes - ONLY admin nodes.
That is why when performing operational backup from admin node using CLI, it was successfull - but not from GUI

After allowing the connections from all admin and monitoring nodes, all backups are working fine.

Thank you for trying to find a solution.

Re: Cisco ISE Backup Fails -Repository validation: confd implementatio

Aref Alsouqi — Wed, 27 Nov 2024 11:05:59 GMT

You're welcome, and thanks for sharing the root cause and the solution.