How to clean up ISE Database to reduce Config Backup size

[Arne Bier]

Hello

I am sitting with a 24GB config backup file and there are only 110,000 endpoints in the database. I don't believe that 24 GB should be the expected size of the database with this many endpoints, am I right?  If ISE supports 2 million endpoints, then I shudder to think what size the backup would be (20 * 24 GB = 480GB)

The gpg file is full of noise, but the significant contributer to the size is the Oracle "dump" ?

-rw-r----- oracle/oinstall 24403230720 2021-07-07 00:14 backup/ise/db/EXP_FULL_DB_BKP_2021_07_07_003018.dmp

Is a .dmp file expected? 

The backup was done on ISE 2.0.1 and imported/upgraded into ISE 2.4 - the ISE 2.4 config backup was also 24 GB.

I am wondering what steps an end user can take to clean up / optimise the Oracle database. Because this doesn't seem to relate to the GUI "Purge All Operational Data", or "Delete Local Logs Now" - I did both of those.

[Marcelo Morais]

Hi @Arne Bier ,

 please try the following: 

ise/admin# application configure ise
Selection configuration option
...
[3]Purge M&T Operational Data
...

then execute the URT (ise-urtbundle-2.4.0.357-1.0.0.SPA.x86_64.tar.gz), for:

...
Running schema upgrade on cloned database
- Running db sanity to check and fix if any index corruption
...

Hope this helps !!!

[Marcelo Morais]

Hi @Arne Bier ,

 it's worth a shot to try the following:

ise/admin# application configure ise
Selection configuration option
...
[3]Purge M&T Operational Data
...

and execute the URT (ISE-URTBundle-2.4.0.357-1.0.0.SPA.x86_64.tar.gz), to check the following:

...
Running schema upgrade on cloned database
- Running db sanity to check and fix if any index corruption
...

Hope this helps !!!

[Arne Bier]

Hi @Marcelo Morais - good suggestions - I tried that along with a few other options in that same menu and nothing is cleaning the database. It's always stuck on 24GB. I didn't expect the MnT to have anything to do with this, because we didn't restore an operational database backup.  Even the 2.7 URT didn't clean up anything

This ISE config backup contains a lot of history (ISE logs going back as far as 2014) - I would imagine the database has seen a lot of action. Analysing the Endpoints database, I believe we only need to keep 20% or less and the 2.0 config has been migrated over nicely into ISE 2.4. 

Probably needs a TAC case to look under the hood. 

[douglasdmoe]

Hi Arne - 

Did you ever find a solution to this issue?   I am running into a similar problem with the config backup growing way too large.   I am running v3.0 patch 4 and have yet to find a solution.

Thanks.

[Marcelo Morais]

Hi @douglasdmoe ,

 worth the shot to try the following ...

 If you de-register a PSN, then it will become a Standalone Node with all the  info in Policy, Posture, Profiling, ..., but without any info in Context Visibility, Report, Live Log, ... 

Note 1: in a LAB the Config Backup from this Standalone Node was 50% smaller.

Note 2: if you recreate the Cluster with this "Standalone Node" as a PPAN, then you will start the Cluster with a "clean the database"

Hope this helps !!!

[Arne Bier · Accepted solution]

Hi @douglasdmoe  - the 24GB backup in my customer's case was cleared out with a single SQL command by TAC. They were able to ascertain that there was a lot of Oracle garbage that was being dragged around for many years. Next backup was in the order of a few hundred MBs.

One SQL command can do a lot of good :-). But still - in the day to day running of ISE, there are logs files and all sorts of nonsense that is included in a config backup. If anyone really needed that stuff then ISE has the option of creating a support bundle. If you look at other products in the market, their config backups are tiny and run to completion in a few minutes. And you can often download the results via a browser. I guess the philosophy used in ISE is to trust no one (not even the ISE admin) to look under the hood, or to have an easy time in dong routine backups. ISE doesn't even clean up after itself - I have to figure out my own plans to keep the repo free and available.  On the other hand, ISE will let me easily export the private keys of certificates. I like that, but I could imagine people who would not like that.

[douglasdmoe]

Thanks for that info Arne.    I have been working with TAC for a few weeks on this issue and it has not yet been resolved.   Is there a TAC case number that I could refer the engineer to so they could review how your issue was resolved?  

Thanks!

[Arne Bier]

Hello @douglasdmoe 

I don't have the TAC case number any longer. You can tell the TAC that the SR was opened in 2021, customer based in NZ and that I was an assigned contact. They should be able to filter on that.

Only TAC could figure this out - in my case it was one SQL table that was full of junk and was truncated (cleared out) - it may not even help you but here is the magic:

[douglasdmoe]

Thanks for the quick reply.   It looks like the issue I ran into has finally been identified.   The new bug is CSCwa19573.  I am working with TAC now to confirm their workaround will fix the issue.

 Info: 

Catalina.out file is huge because of SSL audit events
CSCwa19573

Symptom: Catalina.out file is huge Conditions: ISE 2.7 p5 or ISE 3.0 P4 Workaround: Please contact Cisco TAC for Workaround

[Marcelo Morais]

Hi @douglasdmoe ,

 I recommend this workaround (ISE 2.7 P6 also has the issue), it's easy to apply and you only need to reboot the Node.

Hope this helps !!!