Version 2.3.5.6-70143
Backups are failing with the following error in RED:
```
2024-09-12 03:31:53 4bb6b3ea-21b5-4240-8955-74521395c18f BACKUP.PREPARE_BACKUP SUCCESS
2024-09-12 03:32:30 4bb6b3ea-21b5-4240-8955-74521395c18f BACKUP.maglev-system:credentialmanager SUCCESS
2024-09-12 03:32:43 4bb6b3ea-21b5-4240-8955-74521395c18f BACKUP.fusion:postgres SUCCESS
2024-09-12 03:32:43 4bb6b3ea-21b5-4240-8955-74521395c18f BACKUP.maglev-system:glusterfs-server SUCCESS
2024-09-12 10:38:08 4bb6b3ea-21b5-4240-8955-74521395c18f BACKUP.FINALIZE_BACKUP FAILURE
2024-09-12 10:38:08 4bb6b3ea-21b5-4240-8955-74521395c18f BACKUP.POST_BACKUP.fusion:maintenance-service FAILURE
2024-09-12 10:38:08 4bb6b3ea-21b5-4240-8955-74521395c18f BACKUP.POST_BACKUP.ndp:pipelineadmin FAILURE
2024-09-12 10:38:08 4bb6b3ea-21b5-4240-8955-74521395c18f BACKUP.maglev-system:mongodb FAILURE
2024-09-12 10:38:08 4bb6b3ea-21b5-4240-8955-74521395c18f BACKUP.ndp:elasticsearch FAILURE "DB record did not update for '24038.83337211609' seconds, exceeded '24000' seconds timeout."
Assurance data is 3.5TB – full backups of both automation and assurance are being done. The backup runs for 6 hours and then fails with these errors. The NFS share is on an Isilon cluster, and backups have been working since DNAC was first deployed 5 years ago. The share is mounted directly on DNAC from the server without cascading.
- Remounted NFS
- Tried backup
- Created a new directory
- Remounted NFS
- Tried backup again
drwxrwxrwx. 3 nfsnobody nfsnobody 21 Sep 12 13:32 dnac_assurance
TAC won't support it because it's a non-standard setup. Has anyone seen this issue before, or is there anything else I can try?
