Cisco ISE - Container Storage Space Issue

[netmart2]

Hello,

Our Cisco ISE HA does suffer from some performance issue and by checking the logs, it became obvious that were are dealing with a space issue:

Sending SNMPv2 trap for partition /opt/podman/containers/storage/overlay/...merged which reaches the free space threshold limit

Apparently, the ISE nodes do hit bug id CSCwf39443 0-ENH: Create a mechanism to free space from disk occupied by /opt/podman/containers/storage/overlay/

Symptom: - Full disk due to partitions starting with path /opt/podman/containers/storage/overlay/ are allocating too much disk space - Full disk condition prevent ISE to generate backups

Conditions: Running ISE 2.7 or later

Workaround: In most of the times reload should be enough to restart the container storage, if not reimage of the node

Further Problem Description: The container might belong to different ISE components (Rabbit MQ, Context Visibility, Elastic Search, TC-NAC), Investigation is required to find the affected component.

Could someone please advise how to restart the container storage as suggested above.

Thanks.

[balaji.bandi]

In most of the times reload   - suggesting to reload the tin which has the issue and check.

[netmart2]

Before, we are reloading the box, we wanted to run a backup, but we cannot because of the diskspace issue.

% backup in progress: Starting Backup...10% completed
% Error: The data filesystem is 72 percent full, which is above threshold of 70. Exiting the backup
% Application backup error

To me it is hard to believe that it is not possible to cleanup the disk.

Any suggestion? 

[Aref Alsouqi]

I would highly recommend raising this with TAC, they can access to ISE root shell and see what's going on and trying to clean up the unnecessary files from the root shell which is one thing we can't do ourselves. 

[balaji.bandi]

First i would suggest to try offload the backup to out of ISE so you have safe backup.

as suggested also contact TAC for assistance, so they may clear some files on the backend to get some free space if required,.