ISE Mnt node out of sync

[samy.ccnp]

Hello Guys,

We have distributed ISE deployment with couple of PSNs and Mnts however we have recently observed that both primary and secondary Mnt nodes are out of sync and we are not able to see any further new logs.

Further to our troubleshooting we have found that disk directory is 100% full so i am not sure whether this is related with this if yes then please suggest how to optimize/clear this /opt directory.

Mntprimary/admin# show disks

Internal filesystems:
/ : 17% used ( 2297928 of 14987616)
/dev : 0% used ( 0 of 32895128)
/dev/shm : 0% used ( 0 of 32905040)
/run : 1% used ( 25356 of 32905040)
/sys/fs/cgroup : 0% used ( 0 of 32905040)
/storedconfig : 2% used ( 1587 of 95054)
/tmp : 1% used ( 6200 of 1983056)
/opt : 100% used ( 171294636 of 180485852)
/boot : 23% used ( 103678 of 487634)
/run/user/440 : 0% used ( 0 of 6581008)
/run/user/301 : 0% used ( 0 of 6581008)
/run/user/0 : 0% used ( 0 of 6581008)
/run/user/304 : 0% used ( 0 of 6581008)
/run/user/303 : 0% used ( 0 of 6581008)
/run/user/321 : 0% used ( 0 of 6581008)
warning - /opt is 100% used (171294636 of 180485852)

ISE version:- 

Version information of installed applications
---------------------------------------------

Cisco Identity Services Engine
---------------------------------------------
Version : 2.1.0.474
Build Date : Wed May 25 04:34:43 2016
Install Date : Thu Nov 9 13:32:42 2017

Thanks in advance.

[Aravind Ravichandran]

Am not sure whether that out of sync is related with this or not, CSCve13949 is a bug, which fills the /opt & won't clear the old files automatically & is fixed in 2.1 patch-4. Please contact TAC & troubleshoot on this issue.

[packetplumber9]

I have seen this bug in versions past the listed fixed release on 2.2 as well.  As was mentioned, TAC support is needed to clean up the files.  It's helpful to get the issue addressed with TAC before /opt hits 100% because then there is no room to copy over the root patch and booting the server with a recovery disk is needed.  

[hslai]

/opt : 100% used ( 171294636 of 180485852)

---

This looks like a VM with ~ 200-GB HD and too small for MnT. I would suggest to install new ISE nodes with at least 600-GB HD on each. See Disk Space Requirements