topic backup failing in Network Access Control

backup failing

John Vierra — Fri, 31 Aug 2018 17:48:11 GMT

My backups are failing in my lab ISE 2.4 with the below error message. Any ideas on how to fix?

Thanks

Error: The data filesystem is 71 percent full, which is above threshold of 70. Backup aborted

Re: backup failing

Damien Miller — Fri, 31 Aug 2018 19:11:16 GMT

The first thing to check is the disk space on the node. Log in to the CLI and run "sh disk". Depending on which line is utilizing the space you may need TAC to root patch in and clean it up.

If you want you can provide the output of the command above, we can look at it and provide you with your next steps.

Re: backup failing

John Vierra — Fri, 31 Aug 2018 19:15:07 GMT

This is what I get running that command.
ise/admin# show disk

disk repository: 4% used (1016900 of 30106488)

Internal filesystems:
/ : 70% used ( 9880188 of 14987616)
/dev : 0% used ( 0 of 8123384)
/dev/shm : 0% used ( 0 of 8134044)
/run : 1% used ( 1484 of 8134044)
/sys/fs/cgroup : 0% used ( 0 of 8134044)
/tmp : 1% used ( 7060 of 1983056)
/boot : 25% used ( 114106 of 487634)
/storedconfig : 2% used ( 1583 of 95054)
/opt : 71% used ( 120787188 of 180485852)
/run/user/440 : 0% used ( 0 of 1626812)
/run/user/301 : 0% used ( 0 of 1626812)
/run/user/308 : 0% used ( 0 of 1626812)
/run/user/321 : 0% used ( 0 of 1626812)
/opt/docker/runtime/devicemapper/mnt/6177e74d3ad14c3c631fcaded33e50b427193c46156
cf6b87fbc59bab1722aa2 : 1% used ( 403348 of 104805376)
/opt/docker/runtime/containers/2da15de019b9dd6bd185c416c3d473775fab77e031ab0e161
d8cfaedfa6bbc0f/shm : 0% used ( 0 of 65536)
/run/user/0 : 0% used ( 0 of 1626812)
/run/user/304 : 0% used ( 0 of 1626812)
/run/user/303 : 0% used ( 0 of 1626812)
/run/user/322 : 0% used ( 0 of 1626812)
/opt/docker/runtime/devicemapper/mnt/7dc7a52b0607bb8bed5aa804c721fc21edff4b8e418
23f7a8d3962cb9c46adf1 : 1% used ( 566488 of 104805376)
/opt/docker/runtime/containers/9596e16a3be2763b2eed21ea7eea4b3bef661aaedd72ee31c
490a1bc64d7b11a/shm : 0% used ( 0 of 65536)
/opt/docker/runtime/devicemapper/mnt/aaa28fa63dcc9b4122639d6e7b015f7d54470bcbf21
5811c538afb922ece6e6c : 1% used ( 427032 of 104805376)
/opt/docker/runtime/containers/aaa28fa63dcc9b4122639d6e7b015f7d54470bcbf215811c5
38afb922ece6e6c/shm : 0% used ( 0 of 65536)
/opt/docker/runtime/devicemapper/mnt/a778686bb882b07ae952b4494ae0a036650e05bcf42
6b85be10a8a48470cc63c : 1% used ( 394348 of 104805376)
/opt/docker/runtime/containers/a778686bb882b07ae952b4494ae0a036650e05bcf426b85be
10a8a48470cc63c/shm : 0% used ( 0 of 65536)
/opt/docker/runtime/devicemapper/mnt/930eb2cbdb0621031891188254016916a35a422971c
d4c0190ea521a8a9b0305 : 1% used ( 348340 of 104805376)
/opt/docker/runtime/containers/930eb2cbdb0621031891188254016916a35a422971cd4c019
0ea521a8a9b0305/shm : 0% used ( 0 of 65536)
/opt/docker/runtime/devicemapper/mnt/6967600b542eb7614d8f21de92d5d36f22c8fc1a96f
965420a51de792267108e : 1% used ( 347076 of 104805376)
/opt/docker/runtime/containers/5e6152c7527a1d04b601e53a9677e0056c995580dc6cae569
1e6feb361689c43/shm : 0% used ( 0 of 65536)
/run/user/323 : 0% used ( 0 of 1626812)
all internal filesystems have sufficient free space

Re: backup failing

Damien Miller — Fri, 31 Aug 2018 19:43:10 GMT

Engage TAC, / : 70% used ( 9880188 of 14987616) is likely causing your issue and they will install the root patch to determine what is using that space. A couple other 2.4 deployments i'm looking at right now have the same provisioned size, 14987616, but are using only 19% and 31%.

/opt : 71% used ( 120787188 of 180485852) is for your radius/tacacs logs which is high but shouldn't cause the failure. 80% is the high watermark for this, it would be odd if ISE was stopping the backup over it. If you don't care about your log retention you can purge some of this from the log purging maintenance page. https://your.ise.com/admin/#administration/administration_system/administration_system_backup/data_purging

Re: backup failing

John Vierra — Fri, 31 Aug 2018 20:09:07 GMT

What's interesting is I purged everything earlier and it still says 71.

Re: backup failing

Damien Miller — Fri, 31 Aug 2018 20:36:44 GMT

Quite odd, you could try the same from the CLI if you did it from the GUI before.

#3

ise/admin# application configure ise
Selection ISE configuration option
[1]Reset M&T Session Database
[2]Rebuild M&T Unusable Indexes
[3]Purge M&T Operational Data

Re: backup failing

paul — Fri, 31 Aug 2018 21:16:03 GMT

Do not purge your session database that will send you down a spiral of live session/log issues. Problem is there is a bug that when you do a reset it resets back to 2.4 unpatched. So the DB changes in patch 1/2 won't be there and things tend to break especially live sessions.

Re: backup failing

John Vierra — Fri, 31 Aug 2018 21:23:07 GMT

Didn’t seem to help. I’m going to have a WebEx next week with someone so let’s see what they find. Thanks for all the help.

Re: backup failing

Damien Miller — Fri, 31 Aug 2018 22:31:01 GMT

John, let us know what ends up correcting it.

Paul, that's good to know. I'm dealing with an issue where the licensing usage is zero and live sessions are missing. Tac has come back saying that the device ip address column is missing from the radius session table. Now I wonder if that was done in the past.

Re: backup failing

paul — Fri, 31 Aug 2018 22:43:07 GMT

That is the bug you reset DB to patch 0 that field was added in patch 1. The dev team tried adding to manually and it worked for a while. In the end I just rebuilt my M&T nodes.

Re: backup failing

hslai — Sat, 01 Sep 2018 02:34:56 GMT

CSCvh74979 is tracking the issue associated with resetting M&T session and operational databases.