ISE INDEXING ENGINE BACKUP FAILED : Errors in ADE.log

[yastop]

Our Cisco ISE environment has recently encountered issues when attempting to create configuration backups.
When we try to create a backup of the configuration, we receive error messages (Backup Failed: Server=SL-PAN01; Message=ISE INDEXING ENGINE BACKUP FAILED: Errors in ADE.log. Backup terminated), and the
backup process is not completed successfully. We have verified that all settings are configured correctly, but the problem persists. Local repository backup on disk is also not functioning.

Our support contract has expired, and we are unable to open a TAC case. Does anyone have any ideas on how to resolve this?
Primary PAN
Version: 2.7.0.356
Patch Information: 6,7

Secondary PAN
Version: 2.7.0.356
Patch Information: 6,7

Internal filesystems:
/ : 21% used ( 2848400 of 14987616)
/dev : 0% used ( 0 of 32892636)
/dev/shm : 0% used ( 0 of 32904016)
/run : 1% used ( 1360 of 32904016)
/sys/fs/cgroup : 0% used ( 0 of 32904016)
/storedconfig : 2% used ( 1588 of 95054)
/boot : 28% used ( 126763 of 487634)
/tmp : 1% used ( 7108 of 1983056)
/opt : 46% used ( 253956476 of 593333176)
/run/user/440 : 0% used ( 0 of 6580804)
/opt/docker/runtime/overlay/6b375ac9c0352d383129571e4a6ec41f2d785b3e44316fa6f6b0 d20dd9a2c616/merged : 46% used ( 253956476 of 593333176)
/run/user/301 : 0% used ( 0 of 6580804)
/run/user/321 : 0% used ( 0 of 6580804)
/run/user/0 : 0% used ( 0 of 6580804)
/run/user/304 : 0% used ( 0 of 6580804)
all internal filesystems have sufficient free space

ADElog

2023-11-03T13:00:01.991759+00:00 SLISE-PAN01 root: info:[application:operation:p latformproperties.sh] Old Memory Size : 65808032
2023-11-03T13:00:10.735677+00:00 SLISE-PAN01 root: Security Protocol list Start
2023-11-03T13:00:10.912305+00:00 SLISE-PAN01 root: Inside Session facade init
[K2023-11-03T13:00:10.925083+00:00 SLISE-PAN01 root: Old Memory Size : 65808032
2023-11-03T13:00:11.528744+00:00 SLISE-PAN01 root: Time taken for NSFAdminServic eFactory to load625
2023-11-03T13:00:11.684114+00:00 SLISE-PAN01 root: ERROR StatusLogger Log4j2 cou ld not find a logging implementation. Please add log4j-core to the classpath. Us ing SimpleLogger to log to the console...
2023-11-03T13:00:24.871918+00:00 SLISE-PAN01 root: AuditMessage: 34151: AdminAud it.AcsInstance=SLISE-PAN02ise.we.nl, AdminAudit.OperationMessageText=Certific ate Validation failed for host: SLISE-PAN02.ise.we.nl, AdminAudit.AdminName=Un known
2023-11-03T13:00:25.320485+00:00 SLISE-PAN01 root: AuditMessage: 34151: AdminAud it.AcsInstance=SLISE-PAN02.ise.we.nl, AdminAudit.OperationMessageText=Certific ate Validation failed for host: SLISE-PAN02.ise.we.nl, AdminAudit.AdminName=Un known
2023-11-03T13:00:29.107621+00:00 SLISE-PAN01 root: Old Memory Size : 65808032
2023-11-03T13:00:35.113937+00:00 SLISE-PAN01 root: Old Memory Size : 65808032
2023-11-03T13:00:35.118050+00:00 SLISE-PAN01 root: Old Memory Size : 65808032
2023-11-03T13:00:35.123343+00:00 SLISE-PAN01 root: Old Memory Size : 65808032
2023-11-03T13:00:35.125617+00:00 SLISE-PAN01 root: Old Memory Size : 65808032
2023-11-03T13:00:35.146424+00:00 SLISE-PAN01 root: Old Memory Size : 65808032
2023-11-03T13:00:35.832931+00:00 SLISE-PAN01 root: java.io.EOFException: End of input
2023-11-03T13:00:35.832944+00:00 SLISE-PAN01 root: #011at com.google.gson.stream [K.JsonReader.nextNonWhitespace(JsonReader.java:887)
2023-11-03T13:00:35.832947+00:00 SLISE-PAN01 root: #011at com.google.gson.stream .JsonReader.nextValue(JsonReader.java:787)
2023-11-03T13:00:35.832950+00:00 SLISE-PAN01 root: #011at com.google.gson.stream .JsonReader.quickPeek(JsonReader.java:365)
2023-11-03T13:00:35.832953+00:00 SLISE-PAN01 root: #011at com.google.gson.stream .JsonReader.expect(JsonReader.java:321)
2023-11-03T13:00:35.832956+00:00 SLISE-PAN01 root: #011at com.google.gson.stream .JsonReader.beginObject(JsonReader.java:306)
2023-11-03T13:00:35.832959+00:00 SLISE-PAN01 root: #011at com.cisco.ise.vcs.cli. es.util.VaAggregationUtil.populateVaAggregrationData(VaAggregationUtil.java:305)
2023-11-03T13:00:35.832961+00:00 SLISE-PAN01 root: #011at com.cisco.ise.vcs.cli. es.util.VaAggregationUtil.createAggregationIndex(VaAggregationUtil.java:123)
2023-11-03T13:00:35.832964+00:00 SLISE-PAN01 root: #011at com.cisco.ise.vcs.cli. es.util.ESCliUtil.processRequest(ESCliUtil.java:174)
2023-11-03T13:00:35.832966+00:00 SLISE-PAN01 root: #011at com.cisco.ise.vcs.cli. es.util.ESCliUtil.main(ESCliUtil.java:87)
2023-11-03T13:01:01.536531+00:00 SLISE-PAN01 root: info:[application:operation:i sehourlycron.sh] clearing the vsh files from /tmp
2023-11-03T13:01:01.572682+00:00 SLISE-PAN01 root: notice:[application:operation :isehourlycron.sh] number of process 1
2023-11-03T13:01:01.574072+00:00 SLISE-PAN01 root: notice:[application:operation :isehourlycron.sh] removing vsh files
[K2023-11-03T13:01:01.609001+00:00 SLISE-PAN01 root: info:[application:operation:i sehourlycron.sh] Retaining latest core file: core.SLISE-PAN01.elasticsearch[_.29 12.gz
2023-11-03T13:01:02.306155+00:00 SLISE-PAN01 root: info:[application:operation:i sehourlycron.sh] preparing the cleanup script for the unused tokens
2023-11-03T13:01:02.308950+00:00 SLISE-PAN01 root: info:[application:operation:i sehourlycron.sh] cleaning the unused tokens...
2023-11-03T13:01:02.356439+00:00 SLISE-PAN01 root:
2023-11-03T13:01:02.356449+00:00 SLISE-PAN01 root: SQL*Plus: Release 12.1.0.2.0 Production on Fri Nov 3 13:01:02 2023
2023-11-03T13:01:02.356452+00:00 SLISE-PAN01 root:
2023-11-03T13:01:02.356454+00:00 SLISE-PAN01 root: Copyright (c) 1982, 2014, Ora cle. All rights reserved.
2023-11-03T13:01:02.356457+00:00 SLISE-PAN01 root:
2023-11-03T13:01:02.408474+00:00 SLISE-PAN01 root:
2023-11-03T13:01:02.408483+00:00 SLISE-PAN01 root: Connected to:
2023-11-03T13:01:02.408486+00:00 SLISE-PAN01 root: Oracle Database 12c Enterpris e Edition Release 12.1.0.2.0 - 64bit Production
2023-11-03T13:01:02.408489+00:00 SLISE-PAN01 root: With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
2023-11-03T13:01:02.408491+00:00 SLISE-PAN01 root:
2023-11-03T13:01:02.409365+00:00 SLISE-PAN01 root:
2023-11-03T13:01:02.409372+00:00 SLISE-PAN01 root: PL/SQL procedure successfully [K completed.
2023-11-03T13:01:02.409374+00:00 SLISE-PAN01 root:
2023-11-03T13:01:02.410081+00:00 SLISE-PAN01 root: Disconnected from Oracle Data base 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
2023-11-03T13:01:02.410087+00:00 SLISE-PAN01 root: With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
2023-11-03T13:01:02.419823+00:00 SLISE-PAN01 root: info:[application:operation:i sehourlycron.sh] The ISE data filesystem is 46 percent full. No ISE DB cleanup r equired for disk space.
2023-11-03T13:01:02.425164+00:00 SLISE-PAN01 root: info:[application:operation:i sehourlycron.sh] The ISE database transaction log dir is only 1 MB. No ISE DB cl eanup required for transaction log size.
2023-11-03T13:01:02.427139+00:00 SLISE-PAN01 root: info:[application:operation:i sehourlycron.sh] Running DB Monitoring and Self Healing Scripts....
2023-11-03T13:01:04.417545+00:00 SLISE-PAN01 root: Following are Database Warnin g alerts:
2023-11-03T13:01:04.421717+00:00 SLISE-PAN01 root: WARNING: Stale objects found
2023-11-03T13:01:04.421908+00:00 SLISE-PAN01 root: Following are Database Critic al alerts:
2023-11-03T13:01:04.429541+00:00 SLISE-PAN01 root: CRITICAL: Stale objects found
2023-11-03T13:01:04.429776+00:00 SLISE-PAN01 root:
2023-11-03T13:01:04.443527+00:00 SLISE-PAN01 root: info:[application:operation:i sehourlycron.sh] Completed running DB Monitoring and Self Healing Scripts....
[K2023-11-03T13:01:04.449279+00:00 SLISE-PAN01 root: info:[application:operation:i sehourlycron.sh] Flushing Elasticsearch translogs
2023-11-03T13:01:04.476595+00:00 SLISE-PAN01 root: % Total % Received % Xfe rd Average Speed Time Time Time Current
2023-11-03T13:01:04.476609+00:00 SLISE-PAN01 root:  Dload Upload Total Spent Left Speed
2023-11-03T13:01:04.631533+00:00 SLISE-PAN01 root: #015 0 0 0 0 0  0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (35) Encountered end of file
2023-11-03T13:01:04.772840+00:00 SLISE-PAN01 root: info:[application:operation:i sehourlycron.sh] Checking m&t tablespace size...:33
2023-11-03T13:01:04.888553+00:00 SLISE-PAN01 root: info:[application:operation:i sehourlycron.sh] start pinning MNT tables to memory
2023-11-03T13:01:04.939375+00:00 SLISE-PAN01 root:
2023-11-03T13:01:04.939396+00:00 SLISE-PAN01 root: SQL*Plus: Release 12.1.0.2.0 Production on Fri Nov 3 13:01:04 2023
2023-11-03T13:01:04.939415+00:00 SLISE-PAN01 root:
2023-11-03T13:01:04.939421+00:00 SLISE-PAN01 root: Copyright (c) 1982, 2014, Ora cle. All rights reserved.
2023-11-03T13:01:04.939425+00:00 SLISE-PAN01 root:
2023-11-03T13:01:04.988422+00:00 SLISE-PAN01 root: Last Successful login time: F ri Nov 03 2023 13:01:04 +00:00
2023-11-03T13:01:04.988432+00:00 SLISE-PAN01 root:
[K2023-11-03T13:01:04.988435+00:00 SLISE-PAN01 root: Connected to:
2023-11-03T13:01:04.988438+00:00 SLISE-PAN01 root: Oracle Database 12c Enterpris e Edition Release 12.1.0.2.0 - 64bit Production
2023-11-03T13:01:04.988441+00:00 SLISE-PAN01 root: With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
2023-11-03T13:01:04.988444+00:00 SLISE-PAN01 root:
2023-11-03T13:01:04.995007+00:00 SLISE-PAN01 root:
2023-11-03T13:01:04.995020+00:00 SLISE-PAN01 root: PL/SQL procedure successfully  completed.
2023-11-03T13:01:04.995025+00:00 SLISE-PAN01 root:
2023-11-03T13:01:04.995796+00:00 SLISE-PAN01 root: Disconnected from Oracle Data base 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
2023-11-03T13:01:04.995802+00:00 SLISE-PAN01 root: With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
2023-11-03T13:01:05.000777+00:00 SLISE-PAN01 root: info:[application:operation:i sehourlycron.sh] Completed pinning MNT tables to memory
2023-11-03T13:01:05.002528+00:00 SLISE-PAN01 root: info:[application:operation:i sehourlycron.sh] Checking if Upgrade is running
2023-11-03T13:01:05.034912+00:00 SLISE-PAN01 root: info:[application:operation:i sehourlycron.sh] Upgrade is not running
2023-11-03T13:01:05.042633+00:00 SLISE-PAN01 root: info:[application:operation:i sehourlycron.sh] preparing the cleanup script for the alarmcleanup
2023-11-03T13:01:05.045235+00:00 SLISE-PAN01 root: info:[application:operation:i [Ksehourlycron.sh] cleaning the outdated Alarms...
2023-11-03T13:01:05.089229+00:00 SLISE-PAN01 root:
2023-11-03T13:01:05.089245+00:00 SLISE-PAN01 root: SQL*Plus: Release 12.1.0.2.0 Production on Fri Nov 3 13:01:05 2023
2023-11-03T13:01:05.089248+00:00 SLISE-PAN01 root:
2023-11-03T13:01:05.089250+00:00 SLISE-PAN01 root: Copyright (c) 1982, 2014, Ora cle. All rights reserved.
2023-11-03T13:01:05.089253+00:00 SLISE-PAN01 root:
2023-11-03T13:01:05.141955+00:00 SLISE-PAN01 root:
2023-11-03T13:01:05.141962+00:00 SLISE-PAN01 root: Connected to:
2023-11-03T13:01:05.141965+00:00 SLISE-PAN01 root: Oracle Database 12c Enterpris e Edition Release 12.1.0.2.0 - 64bit Production
2023-11-03T13:01:05.141968+00:00 SLISE-PAN01 root: With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
2023-11-03T13:01:05.141970+00:00 SLISE-PAN01 root:
2023-11-03T13:01:05.219793+00:00 SLISE-PAN01 root:
2023-11-03T13:01:05.219804+00:00 SLISE-PAN01 root: PL/SQL procedure successfully  completed.
2023-11-03T13:01:05.219807+00:00 SLISE-PAN01 root:
2023-11-03T13:01:05.221635+00:00 SLISE-PAN01 root: Disconnected from Oracle Data base 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
2023-11-03T13:01:05.221640+00:00 SLISE-PAN01 root: With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
[K2023-11-03T13:01:05.230480+00:00 SLISE-PAN01 root: info:[application:operation:i sehourlycron.sh] Running KPM metrics...
2023-11-03T13:01:05.276000+00:00 SLISE-PAN01 root:
2023-11-03T13:01:05.276013+00:00 SLISE-PAN01 root: SQL*Plus: Release 12.1.0.2.0 Production on Fri Nov 3 13:01:05 2023
2023-11-03T13:01:05.276017+00:00 SLISE-PAN01 root:
2023-11-03T13:01:05.276022+00:00 SLISE-PAN01 root: Copyright (c) 1982, 2014, Ora cle. All rights reserved.
2023-11-03T13:01:05.276026+00:00 SLISE-PAN01 root:
2023-11-03T13:01:05.325082+00:00 SLISE-PAN01 root: Last Successful login time: F ri Nov 03 2023 13:01:05 +00:00
2023-11-03T13:01:05.325091+00:00 SLISE-PAN01 root:
2023-11-03T13:01:05.325094+00:00 SLISE-PAN01 root: Connected to:
2023-11-03T13:01:05.325096+00:00 SLISE-PAN01 root: Oracle Database 12c Enterpris e Edition Release 12.1.0.2.0 - 64bit Production
2023-11-03T13:01:05.325099+00:00 SLISE-PAN01 root: With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
2023-11-03T13:01:05.325101+00:00 SLISE-PAN01 root:
2023-11-03T13:01:07.492671+00:00 SLISE-PAN01 root:
2023-11-03T13:01:07.492680+00:00 SLISE-PAN01 root: 4 rows created.
2023-11-03T13:01:07.492683+00:00 SLISE-PAN01 root:
2023-11-03T13:01:07.492685+00:00 SLISE-PAN01 root:
2023-11-03T13:01:07.492687+00:00 SLISE-PAN01 root: Commit complete.
[K2023-11-03T13:01:07.492689+00:00 SLISE-PAN01 root:
2023-11-03T13:01:07.492692+00:00 SLISE-PAN01 root: Disconnected from Oracle Data base 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
2023-11-03T13:01:07.492695+00:00 SLISE-PAN01 root: With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
2023-11-03T13:01:07.496452+00:00 SLISE-PAN01 root: info:[application:operation:i sehourlycron.sh] Completed running KPM metrics
2023-11-03T13:01:07.514350+00:00 SLISE-PAN01 root: info:[application:operation:i sehourlycron.sh] The Availble Memory ratio 55.45
2023-11-03T13:01:07.518250+00:00 SLISE-PAN01 root: info:[application:operation:i sehourlycron.sh] No trigger drop_cache as Available Memory 55.45%
root: info:[application:operation:p latformproperties.sh] Old Memory Size : 65808032
root: info:[application:operation:d ocker_daemon_control.sh] Done checking status of Docker Daemon status:running pi d:19188
root: info:[application:operation:s p-hub-control.sh] Container run status

[marce1000 · Accepted solution]

 - Try this procedure : https://www.cisco.com/c/en/us/support/docs/security/identity-services-engine-23/213610-ise-2-3-rest-sync-context-visibility.html

   Also look into : https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwb62192 
                                    and or try with P8

 M.

[yastop]

thank you very much