Old Thread but finally found the solution:
2023-10-14 18:36:21 :INFO: Results of executing [source /opt/cisco/uccx/bin/uccx_ids.env ; ontape -v -s -L 0 -t STDIO > /common/drf/ccx_comps/uccx_db/ontape_uccx_backup.gz], returnCode :[256], Output :[Using the backup and restore filter /bin/gzip -c.
Archive failed - ISAM error: An error has occurred during archive back up.
In short, our UCCX backup schedule was running at a time when there is a process that updates Statistics in the UCCX DB, this creates a DB problem that can't be caught with normal commands (utils dbreplication runtimestate - utils uccx dbreplication status), in fact your OS replication and UCCX replication will look good, but the back up does get affected, to fix it do this on the PUB and the issue will be solved:
utils service restart Cisco Unified CCX Database
utils service restart Cisco DRF Master
utils service restart Cisco DRF Local
Long Story: Our backups runs at 2:50AM daily and it takes a long while (close to 15min), a UCCX process named Update Database Statistics runs at 3AM during the weekends, in the UCCX operations guide it says not have a backup overlap when this process is happening, in older UCCX versions this process used to happen daily at 2AM and that is why we had the backup at 2:50 to avoid it, we thought we were smart outmaneuvering the issue by choosing this time but the timing of the backup changed.
I suspect this issue could happen even if your backup is running at a different time, as long as your VM is affected around 3AM during the weekend while this Update statistics process happens, a VM issue related to (CPU-RAM-HDD) could be a trigger also.
Another indication to know that this is the issue in question is in the log:
2023-10-14 18:36:21 :INFO: Results of executing [source /opt/cisco/uccx/bin/uccx_ids.env ; ontape -v -s -L 0 -t STDIO > /common/drf/ccx_comps/uccx_db/ontape_uccx_backup.gz], returnCode :[256], Output :[Using the backup and restore filter /bin/gzip -c.
Archive failed - ISAM error: An error has occurred during archive back up.
The return code 256 from the IBM documentation, points to a DB without transaction logs:
-256 Transaction not available. You cannot begin a transaction in this database because it does not have a transaction log. In order to support transactions, you must start a transaction log. Refer to your Administrator's Guide for information on how to start a transaction log.
https://www.ibm.com/docs/en/informix-servers/12.10?topic=informix-error-messages
Below the documentation about the Update Statistics process
In older versions 10 it used to run daily at 2AM
https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cust_contact/contact_center/crs/express_10_6/maintain_and_operate/guide/UCCX_BK_UAC60444_00_uccx-operations-guide-106.pdf
Hope this helps all other engineers who run into this error.
