topic Hello Sir, in Network Security

Module Disk Usage: High unmanaged disk usage on /Volume.

Enid Vallja — Tue, 12 Mar 2019 13:27:27 GMT

Hi,

I have a critical notification on my Sourcefire - Health Monitor.

It says "Module Disk Usage: High unmanaged disk usage on /Volume" even though the Sourcefire has a lot of free memory.

Please tell me what can I do with the "/Volume" directory to free some memory.

Best regards,

Enid.

Hi Enid,

Dinesh Verma — Fri, 07 Jul 2017 09:53:17 GMT

Hi Enid,

Mostly /var/tmp/ contains a lot of temporary files. So you might want to try this (This is general):

--> SSH to the module

--> Go to expert mode

--> Become root: sudo su

--> Go to: cd /var/tmp/

--> Remove temp stuff using this command: rm -rf Apply_*

Apply_<random> files are created when you deploy things from FMC to the module. Other place you might want to check is

/var/sf/updates/ (here you can delete all older upgrade files).

/var/common/ (delete old TS and core files)

Command: du -sh * (is useful to list out which directories with size). Find out the directories with huge size and if there is unwanted backup or TS files, delete them.

This article is quite helpful too: http://www.cisco.com/c/en/us/support/docs/security/firesight-management-center/118719-technote-firesight-00.html

Regards,

Hi Dv,

Enid Vallja — Fri, 07 Jul 2017 10:36:39 GMT

Hi Dv,

I don't think this has to do with the /Volume directory because I deleted yesterday some filesto free some memory but it still points the /Volume.

Regards,

Enid.

Run the command: df -TH and

Dinesh Verma — Fri, 07 Jul 2017 10:56:01 GMT

Run the command: df -TH and provide me the output. We will see if it's false health alarm.

This is the output:

Enid Vallja — Fri, 07 Jul 2017 11:03:10 GMT

This is the output (memory.jpg), and please take a look to the alerts (alerts.jpg).

/var (/Volume) looks fine to

Dinesh Verma — Fri, 07 Jul 2017 11:10:50 GMT

/var (/Volume) looks fine to me. Just to double check that you gave me the output from device for which you got the health alarm for disk space. Please check.

If it's the same device then I would say it's a false alarm.You can go to health monitor and click on the device which has alarm and then click on Run All Module and wait for a while and see if alert goes away.

Being root on the device, you can try restarting diskmanager service too:

pmtool restartbyid diskmanager (User should be root).

On totally a different note, it's disk space we're talking here, not memory.

I clicked on Run All Modules

Enid Vallja — Fri, 07 Jul 2017 13:08:48 GMT

I clicked on Run All Modules and still the problem is like in the photo.

And I already did a restart yesterday but the notification won't go away.

Hello Enid,

Jetsy Mathew — Fri, 07 Jul 2017 16:38:04 GMT

Hello Enid,

We need to verify if there is any logs or old backups saved in the device. First of all we need to verify if this is a right health alert or if its a false alarm.

It would be great if you can open a Cisco TAC case so that we can investigate this further.

Regards

Jetsy

That's tricky. As jetsy said,

Dinesh Verma — Mon, 10 Jul 2017 01:27:43 GMT

That's tricky. As jetsy said, open up a TAC case and we will help you fix this health alarm.

Hello,

m.trautes — Tue, 11 Jul 2017 19:50:41 GMT

Hello,

have you any solution - i have the same error since 2 Days.

Mike

Hello Sir,

Dinesh Verma — Wed, 12 Jul 2017 00:12:30 GMT

Hello Sir,

We gave you all possible commands so that you could fix it your own. If you still see the alarm then please open up a TAC case, we will take a look to the logs and help you fix the problem. Webex is always good and gives us more insight to the issue.

Regards,

Hello Mike,

enidvallja — Mon, 17 Jul 2017 08:14:28 GMT

Hello Mike,

Well, I opened a TAC Case and after some lookups they notice something:

The SFR Module is running v5.4.0 however the Defense Center managing it is running v5.4.1.

Also the SFR Module is hitting the following bug: CSCuv99982 - Hardcoded 2GB memcap for RNA and AppId - https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuv99982/?reffering_site=dumpcr this is fixed starting on 5.4.0.6 or 5.4.1.5.

Enid.

Re: Hi Enid,

Lee Win Neng — Tue, 04 Sep 2018 01:28:03 GMT

Hi Dv,

Good day to you.

I had carried out the same procedure and commands as suggested by you and was working fine for me. Is there any bugs that causing this issue of exceed on temporary files on the module that causing "high unmanaged disk usage on /Volume".

Kindly need your assistance and advice for this issue, thank you.

Re: Hi Enid,

heritierday — Mon, 05 Nov 2018 15:16:52 GMT

Hi DV,

It is working for me as i have removed all the old updates in /var/sf/updates/.

Thanks a lot.

Re: Hi Enid,

Barrett Cowan — Tue, 28 May 2019 15:43:19 GMT

This is to build off Dv post, and just to be clear here, this is for a FirePower module, not FMC (although, there wouldn't be much difference). Other places you can look that has helped me as well

/var/sf/SRU/
- Delete all Sourcefire_Rule_Updates but last couple
/var/sf/sru/ (notice the case difference)
- Delete all vrt folders but last couple
/var/sf/updates/
- Delete all unwanted updates
/var/sf/vdb/
- Delete all VDB Fingerprint Database files but last couple
/var/common/
- Delete everything
Everything can be deleted under the following directories
- /var/sf/detection_engines/<some GUID>/backup/
- /var/sf/detection_engines/<some GUID>/instance-1/backup/
- /var/sf/detection_engines/<some GUID>/instance-2/backup/
If you’re feeling frisky, and have some time … /var/sf/upgrades_backup/
- Delete all old backups

After deleting everything above, run the following command if you’re not able to just reboot the module

pmtool restartbyid diskmanager

I’ve experienced this issue on 6.0 – 6.2.2.x, I don’t think I’ve dealt with it since 6.2.3. I currently have 85 deployments of mixed FirePower & FTD (FTD has a LOOONG way to go .. BTW). Been through a lot of TAC cases and issues with FirePower.

Re: Hi Enid,

ArchiTech89 — Wed, 21 Aug 2019 00:41:01 GMT

Barrett has it goin' on. I work at Cisco on Firepower/NGFW devices (FTD) and this is as good a summary of locations for mitigating disk usage problems I've seen. Kudos to him!

Re: Hi Enid,

Barrett Cowan — Mon, 21 Oct 2019 14:42:08 GMT

Thank you @ArchiTech89, much appreciated!

Adding to my previous post, just came by an issue today on v6.1.0.7, classic "High unmanaged disk usage on /Volume". Poking around the directories, I found a file in the /var/log directory named messages.1 that was 26G. Deleted it and rebooted, everything good now.

Just for context, I can reference many other deployments we have out there to compare. This allowed me to figure out how the messages file was too large/out of place.

Re: Hello Mike,

dpereziii — Thu, 31 Oct 2019 14:25:13 GMT

So I am experiencing this same exact issue on one of my modules that currently on 6.3.0.3. Attempting to upgrade to 6.4.0 and it failed about 35% of the way. I followed the directions in clearing out space and ran the pmtoool followed by reboots of the module and FMC without success. One thing I did notice was that the /Volume directory now has a 6.3.0 and a 6.4.0 directory with over 15GB of data each, but it's still on 6.3.0.3. Would it be safe to remove the 6.4.0 directory, run the pmtool, reboot (if necessary) and run the upgrade again?

Re: Hello Mike,

dpereziii — Thu, 31 Oct 2019 14:45:50 GMT

Nevermind, I took the risk and removed the entire directory, ran the pmtool and that fixed it.

Re: Hi Enid,

mansoorsherazi — Tue, 05 Jan 2021 08:57:41 GMT

Thank a lot, I was scratching my head for the last 3 days. I was unable to upgrade my devices from fmc. your solution made my day.

Again Thanks and best of luck