ISE disk usage following a restore

[cameronreeves]

I have a production system running ISE 2.1 Patch 6. Total disk used is ~500gb.

I have built an ISE 2.1 P6 test system (for assessing the 2.1 to 2.2 upgrade process and duration) and restored the config and operational backups from the production system to my test system. Total disk used is ~200gb.

Questions:

- Does ISE do an automatic purge at some point of the backup or restore?

- Do I need to complete a 'backup-logs' from the CLI to backup/restore system logs in addition to the operational backup?

As a separate challenge, I wish there was an easy way to determine the total amount of space used by logging files/databases within ISE. Perhaps the inclusion of overall disk usage (and a subset for logging usage) as part of the System Summary on the Home Page. How do other people manage disk usage within ISE - CLI, SNMP, other?

[Arne Bier]

I don't fully grasp it either but ISE 2.3 does try to display the disk usage of the MnT nodes at least.  You can decide how many days' worth of Radius and TACACS logs you want to keep

I think the right-most, thin blue bar represents the TACACS usage (but trying to hover over that is tricky - I think I was hovering over it but it didn't say it was for TACACS).  The other one on the left that I highlighted in red appears to represent 13GB worth of radius logs so far.  I could probably stretch my retention period to 1 year at this level of low activity.

If you want to monitor the disk usage on each node then use the show disk command.

There is an SNMP trap that I have also set to alert if the entire disk system usage is above 80% (i.e. 20% free) - I have tested this and it seems to work.  You cannot trap at a volume/mount point level - it applies to entire disk.  If you search this forum you will find a discussion I wrote on this topic.

When I do a config backup in my prod system it keeps growing and it's currently 400MB.  That worries me because that is supposed to be a config backup (what is it backing up???? No way that I have been configuring 400MB worth of stuff).  I don't have more than 200 endpoints in my DB either.  Who knows what it's doing ...

I don't make operational backups - this is where I would expect logs to be kept.

[Craig Hyps · Accepted solution]

Yes, you can purge operational data which is the MnT logging data for running reports.  You may be seeing issues with ISE 2.2/2.3 related to larger config backups due to issue with Context Visibility and current patches are resolving the retention of excessive transactional data.

As pointed out, the logging page will show utilization.  Yes, it is a combination of RADIUS + TACACS and a thin bar at 80% which represents a purge threshold.  If not running T+ via scripted admin, then that line will be negligible.

ISE does not purge backup files from your repository.

Local and system logs are not part of config or operational backup.  Under Operations > Troubleshooting, you can capture a support bundle and include local and system logs.

[cameronreeves]

Thanks for the replies, I appreciate your time.

Arne: Unfortunately we are running ISE 2.1 (reasonably solid and reliable), so dont get the database utilisation currently. Maybe thats a reason to push ahead to 2.3! Could you check the output from show disk for me so that I can compare what it reports compared to the GUI?

I also have similar concerns about config backups being large (ours are ~700mb), but they work so I am relucant to question it. Ironically, I think that the operational backup and backup-logs files should be much larger.

Chyps: We have the operational data set to purge after 90 days, but your comments about issues with retention of excessive transaction data are interesting - I will keep a close eye on this.

I think your comment about local and system logs not being part of the config/operational backups is where I struggle the most to understand the ISE backups and are in reality the probable reason for the disk utilisation discrepancy.

At a high level, I am still in a pickle with regards to the disk utilisation, testing the 2.1 -> 2.2 upgrade process, and issues with the backup-logs not restoring (error about 'invalid backup file version'). In our environment the config backup is 762mb, operational is 23.5gb, and backup-logs is 730mb. It's feasible that compression could get the backup-logs down to a very small size, but without a working restore of the backup-logs its impossible to tell.

Ultimately what is required is an accurate measure of how disk is used between radius logs, tacacs logs, system logs, local logs, and anything else which consumes space (operating system, temp files, dump files, etc). I understand Cisco's stance on controlling access to the full CLI, but without controls built into the product to report on these (CLI or GUI), it makes administration and understanding the system when testing upgrades feel like I am flying blind.

[Craig Hyps]

The transaction log issue would not apply to ISE 2.1.

Yes, backups are compressed.

Yes, we added a GUI in ISE 2.2 to show running disk utilization per previous comments.

I recommend opening TAC case regarding "issues with the backup-logs not restoring (error about 'invalid backup file version')"

It may be a misleading error, but make sure target system has sufficient disk (equal or greater than source system). 

[Arne Bier]

Hi Cameron

Below is the show disk of the MnT that was shown previously in GUI format.

sco8834ise050/admin# show disk

disk repository: 5% used (1420416 of 30106488)

Internal filesystems:
/ : 21% used ( 2922336 of 14987616)
/dev : 0% used ( 0 of 32895200)
/dev/shm : 0% used ( 0 of 32905156)
/run : 1% used ( 776 of 32905156)
/sys/fs/cgroup : 0% used ( 0 of 32905156)
/storedconfig : 2% used ( 1583 of 95054)
/boot : 23% used ( 104092 of 487634)
/tmp : 1% used ( 6612 of 1983056)
/opt : 6% used ( 64115636 of 1212603984)
/run/user/440 : 0% used ( 0 of 6581032)
/run/user/301 : 0% used ( 0 of 6581032)
/run/user/321 : 0% used ( 0 of 6581032)
/run/user/0 : 0% used ( 0 of 6581032)
/run/user/304 : 0% used ( 0 of 6581032)
/run/user/303 : 0% used ( 0 of 6581032)
  all internal filesystems have sufficient free space

As far as the SNMP trap config in ADE-OS is concerned, below is the line of config that alerts when 80% space is used (i.e. 20% available)

snmp-server trap dskThresholdLimit "20"

You can easily analyse what's inside an ISE backup file.  I copied my latest 410MB backup to a Linux server and unencrypted the file with gpg.  e.g.    gpg -d  mybackup.gpg > mybackup.tar

and then untar the bundle with    tar -xvf mybackup.tar

If you run the du -k command over the resultant directory called 'backup' contained in the tar bundle, you will get a nice summary of the type of data that is backed up.  Sadly (and weirdly) the logs are in there.  And they are causing the main problem.  In my case the logs take up 8GB of uncompressed space - if I remove the logs, then I am left with 149MB of uncompressed data.

Put another way, the logs make up 98% of the backup file in my case.

Just to be clear - this is NOT a support bundle and this is NOT an operational backup.  What are the logs doing in this file?

[cameronreeves]

Thanks for the feedback Arne, I'll continue the investigation as to where our data has gone and the ensuing upgrade steps.

[Nadav]

Any idea if this is still the case for ISE 2.4 P4?

If logs are still taking up the vast majority of the backups, that's unfortunate.

Is there a way to truncate these logs so that they are only X days back?

[Jason Kunst]

please see log retention in cisco live : BRKSEC-3699

https://community.cisco.com/t5/security-documents/ise-training/ta-p/3619944

If you're having problems please work through TAC on bugs

[Nadav]

Hi Jason,

My question was regarding whether or not the system logs (not operational data) are still unnecessarily enlargening configuration data backups.

This is not addressed in the Cisco Live you linked, that has to do more with log retention of operational data.

I don't think this is a question for TAC yet, it's more a matter of whether or not the same trend persists in modern versions of ISE. If it does, then I may open a case.

[Jason Kunst]

I will be reaching out to an SME on this to get back with us.

[Nadav]

Awesome, thanks!

[Nidhi · Accepted solution]

There has not been any change with respect to the this ask. 

system logs are part of configuration backup and these are needed in case of system failure or upgrade failures to restore it to the original state. 

Hope this helps. 

Thanks,

Nidhi

[Nidhi]

additionally system logs are needed if you need to restore on a different setup . 

[mcloudops]

I have an issue with ISE 2.4 patch 9, where /opt folder is keep increasing. Anyone know whats the reason for this? Due to this am unable to to access GUI of ISE.