@Quintin.Mayo I presume you might have fixed it since it has been a while, but I was able to get to the bottom of this with a TAC case:
2023-10-14 18:36:21 :INFO: Results of executing [source /opt/cisco/uccx/bin/uccx_ids.env ; ontape -v -s -L 0 -t STDIO > /common/drf/ccx_comps/uccx_db/ontape_uccx_backup.gz], returnCode :[256], Output :[Using the backup and restore filter /bin/gzip -c.
Archive failed - ISAM error: An error has occurred during archive back up.
In short our UCCX backup schedule the backup was running at a time when there is a process that updates Statistics in the UCCX DB, this creates a DB problem that can't be caught with normal commands (utils dbreplication runtimestate - utils uccx dbreplication status), but the back up does get affected, to fix it do this on the PUB and the issue will be solved:
utils service restart Cisco Unified CCX Database
utils service restart Cisco DRF Master
utils service restart Cisco DRF Local
Long Story: Our backups runs at time daily (2:50AM) while a a UCCX process named Update Database Statistics runs at 3AM during the weekends, in the UCCX operations guide it says not have a backup overlap when this process is happening, in older UCCX versions this process used to happen daily at 2AM and that is why we had the backup at 2:50 to avoid it, we thought we were smart outmaneuvering the issue by choosing this time but the timing of the backup changed.
I suspect this could happen also even if your backup is running at a different time, for example VM issues (CPU-RAM-HDD) while this process happens could trigger it and CCX can't recover.
Another Giveway to know that this is the issue in question is in the log:
2023-10-14 18:36:21 :INFO: Results of executing [source /opt/cisco/uccx/bin/uccx_ids.env ; ontape -v -s -L 0 -t STDIO > /common/drf/ccx_comps/uccx_db/ontape_uccx_backup.gz], returnCode :[256], Output :[Using the backup and restore filter /bin/gzip -c.
Archive failed - ISAM error: An error has occurred during archive back up.
The return code 256 from the IBM documentation, points to a DB without transaction logs:
-256 Transaction not available. You cannot begin a transaction in this database because it does not have a transaction log. In order to support transactions, you must start a transaction log. Refer to your Administrator's Guide for information on how to start a transaction log.
https://www.ibm.com/docs/en/informix-servers/12.10?topic=informix-error-messages
Below the documentation about the Update Statistics process
In older versions 10 it used to run daily at 2AM
https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cust_contact/contact_center/crs/express_10_6/maintain_and_operate/guide/UCCX_BK_UAC60444_00_uccx-operations-guide-106.pdf
Hope this helps all other engineers who run into this error.
