TACACS: Reached TACACS+ maximum client limit

[The Arrow]

Hi All,

We could see TACACS: Reached TACACS+ maximum client limit error on one of the CISCO ISE PSN nodes so we tried restarting the service after the restarting the service could see AD is not getting joined due to below error.

Status: Join Operation Failed: Cannot connect to AD Connector. It may be unavailable due to limited disk space Error Description: Cannot Connect To AD Connector. It May Be Unavailable Due To Limited Disk Space

Support Details...
Error Name: ERROR_FILE_NOT_FOUND
Error Code: 2

We could see there is available disk space but getting the same error when we join the AD.

Internal filesystems:
/dev : 0% used ( 0 of 32827872)
/dev/shm : 1% used ( 84 of 32844304)
/run : 1% used ( 1212 of 32844304)
/sys/fs/cgroup : 0% used ( 0 of 32844304)
/ : 39% used ( 5977996 of 16499528)
/storedconfig : 3% used ( 2137 of 95054)
/boot : 18% used ( 158628 of 991512)
/opt : 12% used ( 62101564 of 591316016)
/tmp : 1% used ( 6524 of 1983056)
/run/user/440 : 0% used ( 0 of 6568860)
/opt/podman/containers/storage/overlay-containers/c6cc517e695700cf5952c11febd3781d59f85ea611d152218eb513b60c1f797a/userdata/shm : 0% used ( 0 of 64000)
/opt/podman/containers/storage/overlay/520bc4378528348b4efc717bfbc086d00f8b634fef33e4c25672c9a73da88c03/merged : 12% used ( 62101564 of 591316016)
/opt/podman/containers/storage/overlay-containers/4c79333ffcb5609bebc7c4156e06382bcaf7706eff5feb81a7ddb4d1aa2dd188/userdata/shm : 1% used ( 8 of 64000)
/opt/podman/containers/storage/overlay/17e2865270a10c00b4681abab4208c1b322172fc1000c97eec86b195a700165f/merged : 12% used ( 62101564 of 591316016)
/run/user/301 : 0% used ( 0 of 6568860)
/opt/podman/containers/storage/overlay-containers/6758d959bdacbc3d96122ff3e9a5e83d84006d7156229e7594a9bc6cd67d1c5c/userdata/shm : 0% used ( 0 of 64000)
/opt/podman/containers/storage/overlay/c5b75531c8c6182a8d3f0aa1cdfab87e9e721376bf23f75736c697f28a801b70/merged : 12% used ( 62101564 of 591316016)
/run/user/321 : 0% used ( 0 of 6568860)
/run/user/0 : 0% used ( 0 of 6568860)
/run/user/300 : 0% used ( 0 of 6568860)
/run/user/302 : 0% used ( 0 of 6568860)
  all internal filesystems have sufficient free space

ISE version is running on 3.1 P4

Is there any command to delete the disk space over the CLI ? Any other suggestions or help would be appreciated.  

[MHM Cisco World]

https://community.cisco.com/t5/network-access-control/cisco-ise-2-2-tacacs-max-connections-limit-5001-has-reached/td-p/3911081

[The Arrow]

Hi @MHM Cisco World 

Try to do restart the service but when i try application stop ise it got hung and i see below errors in logs continuously also the DB is locked.

Sending SNMPv2 trap for partition / which reaches the free space threshold limit.

Now unable to start the ise too says below error.

Waiting up to 300 seconds for lock: APP_STOP to complete
Database is still locked by lock: APP_STOP. Aborting. Please try it later
Error: Another ISE DB process (APP_STOP) is in progress, cannot perform Application Start at this time

Reload the node will help ?

[MHM Cisco World]

if you check the bug workaround the recommend is reboot PSN 

[The Arrow]

Hi @MHM Cisco World 

We tried reload gets the below message.

Trying to stop processes gracefully. Reload might take approximately 3 mins

Broadcast message from admin@XXXXXX (Sun May 21 10:13:59 2023):

Trying to stop processes gracefully. Reload might take approximately 3 mins

[The Arrow]

@MHM Cisco World Seems to be the process got stuck doesn't aloow to reboot the node.

[hslai]

@The Arrow If it still stuck, just power cycle the ISE node.

[The Arrow · Accepted solution]

The issue has been fixed last day by shut down the VM and power ON.

[The Arrow]

The issue has been fixed last day by shut down the VM and powering ON.

Thank you.

[MHM Cisco World]

You are so welcome