ISE 2.4 P6 - Backup failed - ISE INDEXING ENGINE BACKUP FAILED

[holdentom218]

Hi,

I'm running a fresh install of ISE 2.4 patch 6 in a VM (single server not cluster). I've installed the OVA on an ESX server. I'm using the trial license so I don't have TAC support. I am unable to do a config backup as it fails due to an "Application backup error". I wasn't able to run a config backup before patching either. I thought that it may have been an issue in the 2.4 unpatched. I can run an operational backup without issue. I patched from the same repository so I know that connection is fine. I've restarted several times.

ISENodeA/admin# backup Config_Backup_2.4_Patch6_Initial repository Repository_FTP ise-config encryption-key plain xxxxx

% Internal CA Store is not included in this backup. It is recommended to export it using "application configure ise" CLI command
% Creating backup with timestamped filename: Config_Backup_2.4_Patch6_Initial-CFG10-190305-1943.tar.gpg
% backup in progress: Starting Backup...10% completed
% backup in progress: Validating ISE Node Role...15% completed
% backup in progress: Backing up ISE Configuration Data...20% completed
% backup in progress: Backing up ISE Indexing Engine Data...45% completed
% Application backup error

If I check the backup history it says "ISE INDEXING ENGINE BACKUP FAILED"

ISENodeA/admin# show backup history

Tue Mar 5 19:45:54 AEST 2019: backup Config_Backup_2.4_Patch6_Initial-CFG10-190305-1943.tar.gpg to repository Repository_FTP: failed - ISE INDEXING ENGINE BACKUP FAILED : Errors in ADE.log. Backup aborted

I can see that the indexing engine isn't running on the ISE node but everything else is. I've seen a few posts of people with issues when the indexing engine isn't running but nothing relates exactly to my issue.

ISENodeA/admin# show application status ise | inc Index
ISE Indexing Engine not running

If I look in ADE.log then I see the below "Failed connect to localhost:9200; Connection refused" and the indexing error.

ISENodeA/admin# show logging system ade/ADE.log
2019-03-05T19:45:54.490619+10:00 ISENodeA admin: error:[backup-restore:backup:isecfgbackup.sh] ISE Indexing Engine Backup Failed: Details in ADE.log
2019-03-05T19:45:54.492365+10:00 ISENodeA admin: error:[backup-restore:backup:isecfgbackup.sh] Following errors occurred when running command curl -m 600 -sS -k -XGET 'https://localhost:9200/ise/nad/_search?scroll=5m&pretty' -d ' {"size": 10000, "query": {"match_all": {} }}'
2019-03-05T19:45:54.556661+10:00 ISENodeA admin: info:[backup-restore:backup:isecfgbackup.sh] curl: (7) Failed connect to localhost:9200; Connection refused
2019-03-05T19:45:54.563574+10:00 ISENodeA ADE-SERVICE[911]: [12370]:[error] backup-restore:backup: br_backup.c[796] [admin]: ISE INDEXING ENGINE BACKUP FAILED : Errors in ADE.log. Backup aborted
2019-03-05T19:45:54.565864+10:00 ISENodeA ADEOSShell[12370]: ADEAUDIT 2012, type=BACKUP, name=BACKUP FAILED, username=admin, cause=A backup has failed, adminipaddress=127.0.0.1, interface=CLI, detail=ISE INDEXING ENGINE BACKUP FAILED : Errors in ADE.log. Backup aborted
2019-03-05T19:45:54.568426+10:00 ISENodeA ADEOSShell[12370]: ADEAUDIT 3013, type=BACKUP, name=BACKUP FAILED, username=admin, cause=Error during backup, adminipaddress=127.0.0.1, interface=CLI, detail=ISE INDEXING ENGINE BACKUP FAILED : Errors in ADE.log. Backup aborted
2019-03-05T19:45:54.570580+10:00 ISENodeA ADE-SERVICE[911]: [12370]:[info] backup-restore:backup: br_history.c[487] [admin]: updating /tmp/ise-cfg-br-flags with status: complete and message: (null)
2019-03-05T19:45:54.570583+10:00 ISENodeA ADE-SERVICE[911]: [12370]:[info] backup-restore:backup: br_cli.c[1128] [admin]: error message: Application backup error
2019-03-05T19:46:39.349320+10:00 ISENodeA ADE-SERVICE[911]: [18123]:[info] utils: vsh_pipe.c[49] [admin]: Invoked pipe handler

I've looked around for potential issues. I have plenty of space on all disk partitions, I have forward and reverse DNS entries (check from the NODE CLI works fine), NTP is running and sync'd, my Admin cert if still valid etc. I can see in the ise-psc.log:

ISENodeA/admin# show logging application ise-psc.log

2019-03-05 19:45:54,716 INFO [AdeSyslogProcessor][] cpm.infrastructure.systemconfig.syslogproc.AdeSyslogContent -::::- SCH: InfraTCPMessage: msg: InfraUdpMessage: <130>Mar 5 19:45:5
4 ISENodeA ADEOSShell[12370]: ADEAUDIT 2012, type=BACKUP, name=BACKUP FAILED, username=admin, cause=A backup has failed, adminipaddress=127.0.0.1, interface=CLI, detail=ISE INDEXING ENGINE BACKUP FAILED : Errors in ADE.log. Backup aborted
2019-03-05 19:45:54,716 INFO [AdeSyslogProcessor][] cpm.infrastructure.systemconfig.syslogproc.AdeSyslogContent -::::- SCH: messageCode is : 2012
2019-03-05 19:45:54,716 INFO [AdeSyslogProcessor][] cpm.infrastructure.systemconfig.syslogproc.AdeSyslogContent -::::- SCH: InfraTCPMessage: msg: InfraUdpMessage: <130>Mar 5 19:45:5
4 ISENodeA ADEOSShell[12370]: ADEAUDIT 3013, type=BACKUP, name=BACKUP FAILED, username=admin, cause=Error during backup, adminipaddress=127.0.0.1, interface=CLI, detail=ISE INDEXING ENGINE BACKUP FAILED : Errors in ADE.log. Backup aborted
2019-03-05 19:45:54,716 INFO [AdeSyslogProcessor][] cpm.infrastructure.systemconfig.syslogproc.AdeSyslogContent -::::- SCH: messageCode is : 3013
2019-03-05 19:45:54,716 INFO [AdeSyslogProcessor][] cpm.infrastructure.systemconfig.syslogproc.AdeSyslogContent -::::- AUDITMSG : unsupported cars audit message code :3013ate is installed on Administration->System->Certificates->Trusted Certificates. If not, install them and restart application services. Exception: None of the configured nodes are available:

Anyone have any ideas?

Cheers,
Tom

[howon]

Indexing service should be running. Have you tried to reissue the admin certificate and restart? If you are using different certificate try using self-signed one for test.

[holdentom218]

Hi Howon, thanks but tried that already. Created another self signed cert (I was using the default), assigned Admin and EAP to the new self signed cert, ISE restarted the Application Server automatically but I restarted the server manually as well, but same issue, ISE Indexing Engine not running and backup fails.

Regards - Tom

[Benjamin TARAUD]

Hi!

Did you find a workaround for your issue ? I have EXACTLY the same error on 3 differents deployments. 

I have a Webex monday with TEC Engineer to find an issue.

Have a nice day,

Ben

[hslai]

Please work with TAC on this issue. Yours might have a different reason that this service not running.

[EdoukouMelaine81749]

Hello @Benjamin TARAUD, did you get any update on the subject? thank for sharing

[gbrevern]

This worked for me - We have a setup with PAN & SAN (among other personas) so I had to run it on both at CLIs:

application configure ise

From the menu, these were the ones of interest:

[20]Reset Context Visibility
[21]Synchronize Context Visibility With Database

Tried [21] on the PAN first - which failed in this case.

Then tried [20] on the PAN - You then get instructions to run it on the SAN as well & instructions on the order in which to do things.

The backup that was previously failing now works.

Hope this helps.

[erik.hammervold]

This also helped me. Exact same issue. Thanks

[Marcelo Morais]

Hi @erik.hammervold ,

 for future reference of anyone with the same issue ... take a look at: ISE 2.3 Rest/Sync Context Visibility.

Hope this helps !!!

[Walker]

Good morning gbrevern,

When you perform option 20, does it remove all information from context visibility database? I am having this exact same issue.

[Marcelo Morais]

Hi @Walker ,

 yes, when you choose 20:

"This will remove all data from Context Visibility. Do you want to proceed [y/n]:"

Hope this helps !!!

[Walker]

@Marcelo Morais 

I just had a misunderstanding. After researching, I understand that these actions do no affect the Oracle database that store the active endpoint information. I performed these actions tonight and it resolved my Context Visibility issues as well as the backup issues. Thank you!

[darrenkmiyamoto]

This worked for me on ISE 3.1.0.518 patch 3.  Backups now work.  Thanks!

application configure ise
[20]Reset Context Visibility