FMC crashed during upgrade and login to the GUI doesn’t work

[Chess Norris]

Hi,

I just did a FMC upgrade from version 6.4 to 6.6.5. The readiness check went through without any issues, but the upgrade failed at 22% and I saw the following in the logfile.

Fatal error: Unable to create /Volume/6.6.5/{tmp,run,sf/run,sf/peers,log/sf,lib/mysql}

./functions.install: line 179: echo: write error: No space left on device

After checking the disk, I can see that /var is 100% full and the reason for the interrupted installation.

The GUI is still reachable but I cannot login to it anymore with my local username.
I have already opened a TAC case asking for some advice on which files I can safely remove to free up some space, but wanted to ask if someone else have experienced this and how you solved it?
Before when an upgrade stopped, I have been able to run the scipt "upgrade_rollback.sh", to get back to the previous version, but I am not sure it's such a good idea to run this when the disk is 100% full.

There is also a "upgrade_resume.sh" script that might be possible to run after a disk cleanup. 

Thanks

/Chess

[Eric R. Jones]

Have checked for the log files in there? If you are sending logs to remote storage I would still see if there are some large logs in there. The FMC doesn't provide a method, that I have found, to setup rotating logs or remove logs automatically by date, size etc.. I found this old forum post they may or may not offer some help, https://community.cisco.com/t5/network-security/delete-backups-on-fmc-90-disk-space-used/td-p/4435339.  

We use remote logging so we can go to the FMC, we use 7.0.1, and select and remove files from there or just go to the NFS location and remove them from there.

[Jitendra Kumar]

Yes @Eric R. Jones  you are right to check below thinks.

1.what is occupied most of the space.

2.Delete where most not required.

3 Delete Logs which is not required.

4. Don't forgot to take backup file.

Below is the Article that will help you.

https://www.cisco.com/c/en/us/support/docs/security/firesight-management-center/118719-technote-firesight-00.html

Thanks,

Jitendra

[Chess Norris]

I am working with TAC at the moment. They gave me those commands to check for files that would be safe to delete.

find /var/ -type f -printf '%s %p\n' | sort -nr | head -100

find / -type f -exec du -Sh {} + | sort -rh | head -n 100

I will update  when I hear back from them.

Thanks

/Chess