ISE disk space

[csosup]

Hi

Our Cisco ISE appliance is running low on space which causes the backups to fail.

Looking into the disk space it looks like we have lots of old application logs (appserver.catalina.out) and also the we have old system log from 2015/2016 in a location called mntbackup? (mntbackup/FULL_TS_BACKUP_LOG_<DATE>.log)

I cant seem to delete files in either of these locations (i can see them by running the commands 'sh logging system' or 'sho logging application' but cant delete) after googling it looks like at least one of these is down to a Cisco Bug: CSCva95303.

I have inherited this appliance, i know we dont have support, the disks were sized properly upon initial setup and also it needs an update running version 2.0.0.306, but i dont want to do an update until this is clear as it hasn't been backing up recently

Is there anything i can do to clear these files out?

Thanks

[Arne Bier]

Have you looked around on the CLI (show disk) to see whether there are any files you can remove? 
I think it might ultimately require the root patch to gain Linux shell access. That means TAC. 
if this were a VM then you could shut the system down and mount the ISE vmdk on a Linux host and clean up the files therein. 

[csosup]

Hi Arne

I have tried to delete files in the past but these files i am talking about arent listed in the show disk or DIR commands see below

csoise01/admin# dir

Directory of disk:/

82 May 22 2018 13:18:33 1
16384 Jun 09 2015 14:30:28 lost+found/

Usage for disk: filesystem
38416384 bytes total used
14520705024 bytes free
15345455104 bytes available
csoise01/admin# sh disk

disk repository: 1% used (37516 of 14985796)

Internal filesystems:
/ : 70% used ( 119601720 of 180476204)
/dev/shm : 0% used ( 0 of 8167076)
/boot : 7% used ( 29623 of 481764)
/storedconfig : 2% used ( 1583 of 89231)
/tmp : 1% used ( 3260 of 1975372)
all internal filesystems have sufficient free space

I know TAC is the option but we dont have suppose (and probably wont get it any time soon unfortunately)

This is a virtual appliance so i could give the Linux host a go, do you have any more in on that route? 

[Arne Bier · Accepted solution]

Hi

This requires a bit of Linux knowledge - see how you go. And WARNING - if you mess this up then you can stop your ISE from working. I use this procedure in my lab VMs only - I need to make space because my labs run on SSDs )

For some history, see my earlier article on this here.

Procedure:

You must shut down the ISE VM.

Create/Use a Linux VM (e.g. Ubuntu or CentOS) and add an additional disk to it - instead of creating a new disk, use existing disk (the .vmdk from ISE).

Boot up the Linux VM. If this is the second disk, then the device will most likely be /dev/sdb  (check with the dmesg command)

As root user, make two mount points

mkdir /mnt/part2
mkdir /mnt/part7

mount two file systems

mount /dev/sdb2 /mnt/part2
mount /dev/sdb7 /mnt/part7

Inspect these partitions - delete only the obvious stuff like big core files, ancient files (upgrade/patch related, etc.) - make sure you know what you're deleting!

Once done, unmount the partitions

umount /dev/sdb2
umount /dev/sdb7

That's it.

[csosup]

Thanks for this, i had heard of some sort of linux hack but couldnt find it until your reply.

That did the trick and i have managed to trim down a load of old logs to give us enough breathing space for now... My plan is to sort our new ISE servers to replace our current ones as they are running an old version and only have 200 GB drives