Cisco ISE Disk space problem

faizan.khan · ‎09-27-2017

We have ISE V2.1.0.474 Patch 3. I guess we’re having problem with replication. When we look at VM, it is shown that primary node is used 280+ GB and Secondary (new) is used 72+ GB? It's a big difference, the primary is taking much more space than secondary, we need to clean the Primary. My question is how to clean the disk on ISE that had problems.
sh disk (On primary)
Internal filesystems:
/ : 16% used ( 2213420 of 14987616)
/dev : 0% used ( 0 of 8124672)
/dev/shm : 0% used ( 0 of 8134360)
/run : 1% used ( 50036 of 8134360)
/sys/fs/cgroup : 0% used ( 0 of 8134360)
/tmp : 2% used ( 18704 of 1983056)
/boot : 24% used ( 108016 of 487634)
/opt : 30% used ( 50227372 of 180485852)
/storedconfig : 2% used ( 1585 of 95054)
/run/user/0 : 0% used ( 0 of 1626876)
/run/user/440 : 0% used ( 0 of 1626876)
/run/user/321 : 0% used ( 0 of 1626876)
all internal filesystems have sufficient free space
sh disk (on Secondary)
Internal filesystems:
/ : 16% used ( 2228260 of 14985756)
/dev : 0% used ( 0 of 8124700)
/dev/shm : 0% used ( 0 of 8134360)
/run : 1% used ( 41928 of 8134360)
/sys/fs/cgroup : 0% used ( 0 of 8134360)
/boot : 24% used ( 107914 of 481746)
/storedconfig : 2% used ( 1600 of 89231)
/tmp : 1% used ( 6452 of 1975372)
/opt : 23% used ( 60732172 of 283687164)
/run/user/440 : 0% used ( 0 of 1626876)
/run/user/0 : 0% used ( 0 of 1626876)
/run/user/301 : 0% used ( 0 of 1626876)
/run/user/321 : 0% used ( 0 of 1626876)
/run/user/322 : 0% used ( 0 of 1626876)
/run/user/304 : 0% used ( 0 of 1626876)
/run/user/303 : 0% used ( 0 of 1626876)
all internal filesystems have sufficient free space

bryantsteve · ‎09-27-2017

Hi Faizan

Also had problems with ISE replication believe these may be related cisco bugs associated with v 2.1 patch 3 disk space issues

CSCvd56372

CSCvd56328

CSCvc79739

I opened a TAC case, was provided root kit to access SQL for a workaround fix for v 2.1 patch but I think the v 2.2 may address these bugs

Austin Sabio · ‎09-27-2017

Indeed. Internal files look good nothing is exceeding %30. make sure secondary node is reachable and not out of sync. However, look in the alarm section for any replication issue something smiliar to

Slow Replication Error	Slow or a stuck replication is detected .	Please verify that the node is reachable and part of the deployment.
Slow Replication Info	Slow or a stuck replication is detected .	Please verify that the node is reachable and part of the deployment.
Slow Replication Warning	Slow or a stuck replication is detected .	Please verify that the node is reachable and part of the deployment.

Replication Failed

The secondary node failed to consume the replicated message.

Login to the Cisco ISE GUI and perform a manual syncup from the deployment page. De-register and register back the affected Cisco ISE node.

Replication Stopped

ISE node could not replicate configuration data from the PAN.

Login to the Cisco ISE GUI to perform a manual syncup from the deployment page or de-register and register back the affected ISE node with required field.

Source: https://www.cisco.com/c/en/us/td/docs/security/ise/2-0/admin_guide/b_ise_admin_guide_20/b_ise_admin_guide_20_chapter_011000.html#concept_BA50BA2E31CD411585B5D28869B4F564

also you can check replication using show backup status and show backup history

show backup history	Displays the backup history of the system.
show backup status	Displays the status of the backup operation.

+ checking system logs such as replication.log and tracking.log not sure about the specific path I think troubleshooting and monitoring section or something similar to 'debug log' sub-tab.

Possible related bugs via https://bst.cloudapps.cisco.com/bugsearch/:

CSCvb43993:ISE-Replication Status does not update after restore

CSCup05062:Slow replication in ISE 1.2 if primary DNS server is unavailable

CSCvc79381: ISE 2.1 Replication failed: Error synchronizing object

I hope this helps. Good luck!