cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1687
Views
10
Helpful
13
Replies
Beginner

Problem with Prime Infrastucture 3.1.3 /opt disk space full

 

It's a small installation with less than 100 devices

When i run following commands the result showing the following:
ade # df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/smosvg-rootvol
3.9G 324M 3.4G 9% /
tmpfs 32G 11G 21G 34% /dev/shm
/dev/mapper/smosvg-altrootvol
93M 5.6M 83M 7% /altroot
/dev/sda1 485M 48M 412M 11% /boot
/dev/mapper/smosvg-home
93M 6.2M 83M 7% /home
/dev/mapper/smosvg-localdiskvol
244G 188M 231G 1% /localdisk
/dev/mapper/smosvg-optvol
1.4T 1.3T 6.6G 100% /opt
/dev/mapper/smosvg-recvol
93M 5.6M 83M 7% /recovery
/dev/sda2 97M 5.6M 87M 7% /storedconfig
/dev/mapper/smosvg-storeddatavol
9.5G 151M 8.9G 2% /storeddata
/dev/mapper/smosvg-tmpvol
2.0G 36M 1.8G 2% /tmp
/dev/mapper/smosvg-usrvol
6.8G 967M 5.5G 15% /usr
/dev/mapper/smosvg-varvol
3.9G 194M 3.5G 6% /var
ade #

ade # df /opt
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/smosvg-optvol
1424876728 1345588408 6908724 100% /opt
ade #

ade # du -h --max-depth=6 /opt | grep [0-9]G | sort -k2
du: cannot read directory `/opt/lost+found': Permission denied
du: cannot read directory `/opt/CSCOlumos/rcmds': Permission denied
du: cannot read directory `/opt/oracle/base/fast_recovery_area/WCS/archivelog/2018_04_12': Permission denied
12G /opt/CSCOlumos
1.5G /opt/CSCOlumos/logs
64K /opt/CSCOlumos/matlab/toolbox/matlab/imagesci/@H5G
2.9G /opt/CSCOlumos/staging
1.6G /opt/CSCOlumos/staging/app
3.1G /opt/CSCOlumos/updates
1.4G /opt/CSCOlumos/updates/PI_3_1_3-1.0.16
1.6G /opt/oracle/base/diag/tnslsnr
1.6G /opt/oracle/base/diag/tnslsnr/FMCL-PRIME1
1.6G /opt/oracle/base/diag/tnslsnr/FMCL-PRIME1/wcstns
3.3G /opt/oracle/base/fast_recovery_area
3.1G /opt/oracle/base/fast_recovery_area/WCS
3.1G /opt/oracle/base/fast_recovery_area/WCS/onlinelog
29G /opt/oracle/base/oradata
1.4G /opt/oracle/base/oradata/STBYWCS
1.2G /opt/oracle/base/oradata/STBYWCS/datafile
28G /opt/oracle/base/oradata/WCS
28G /opt/oracle/base/oradata/WCS/datafile
3.0G /opt/oracle/base/product
3.0G /opt/oracle/base/product/12.1.0
3.0G /opt/oracle/base/product/12.1.0/dbhome_1
1.2G /opt/oracle/base/product/12.1.0/dbhome_1/lib
ade #
What files can i remove or minimize to free up more space?

I have tried ncs cleanup this solved the problem temporarily.

13 REPLIES 13
Beginner

Re: Problem with Prime Infrastucture 3.1.3 /opt disk space full

There is nothing you can do to bring the DB to a normal proportion.

 

Do ncs stop.

Now you can add a disk!! Do not make any disk bigger, it will not work.

reboot the system

The OS will see the new disk space on reboot and add it to the partitions.

 

I would suggest to do an export of everything you can export in PI, devices, maps, groups, etc.

Then document everything you cannot export.

Then deploy the OVA again and import.

 

Then check the data retention settings and reduce them to the minimum acceptable for you.

Then make sure you get an alarm from PI when disk space usage on PI goes over the limit.

 

  

Beginner

Re: Problem with Prime Infrastucture 3.1.3 /opt disk space full

The Prime server is not virtual, its a Cisco UCS C220 M4 Server
So it's not easy to just add a disk at the moment, i'm not sure if there is more slots. Need to check that.
Hall of Fame Community Legend

Re: Problem with Prime Infrastucture 3.1.3 /opt disk space full

This is a known bug with 3.1. Upgrade to 3.3 or 3.4.
Beginner

Re: Problem with Prime Infrastucture 3.1.3 /opt disk space full

Can there be a problem to upgrade when there is no space left in /opt ?
Maybe the installation needs more space?
Hall of Fame Community Legend

Re: Problem with Prime Infrastucture 3.1.3 /opt disk space full

The easiest way to upgrade PI is a complete rebuild.
Export all your maps and data. Re-install 3.4 and import the maps and data.
Beginner

Re: Problem with Prime Infrastucture 3.1.3 /opt disk space full

Thanks for reply!

Ok, when you say export maps and data.
Whats the best way to do that? I'll have backup running to external ftp

Should i'll reinstall to 3.4 and then apply the backup from 3.1?
Beginner

Re: Problem with Prime Infrastucture 3.1.3 /opt disk space full

A backup also requires space in /opt even if it is 'remote'

You will have to add a disk, so you can make the system work for a short time, so you can export things.

Or you can try to deploy 3.4 or 3.3 and restore the last backup.

But also then I would do an export of everything possible, deploy the ova again and reimport.

Hall of Fame Community Legend

Re: Problem with Prime Infrastucture 3.1.3 /opt disk space full


@danlin wrote:
Thanks for reply!

Ok, when you say export maps and data.
Whats the best way to do that? I'll have backup running to external ftp

Should i'll reinstall to 3.4 and then apply the backup from 3.1?

That's correct.

Beginner

Re: Problem with Prime Infrastucture 3.1.3 /opt disk space full

This is the graph showing disk space over time :)disk historik prime.png

Beginner

Re: Problem with Prime Infrastucture 3.1.3 /opt disk space full

The setting of a threshold bigger or equal to 100% is asking for troubles.

This is manually set by someone.

I believe out-of-the-box it is 40%

 

Beginner

Re: Problem with Prime Infrastucture 3.1.3 /opt disk space full

After some deepdiving in the filesystem i found the source of problem.

In this directory below there was thousands of directories named incdir_XXXXX

and they contained very big .trc and .trm files.

/opt/oracle/base/diag/rdbms/wcs/wcs/incident

 

After contact with TAC the solution was to remove files, and now i got more than 1TB empty space

This may be some useful info to someone.

 

I don't now why the alarm threshold is 90% but i'm pretty sure that's nothing we have changed since installation. Checked an other installation and there was also 90%?!
But now it's changed to 65%

 

Highlighted

Re: Problem with Prime Infrastucture 3.1.3 /opt disk space full

Thanks a lot for your information.

Do you delete directories or files inside directories?

 

Beginner

Re: Problem with Prime Infrastucture 3.1.3 /opt disk space full

I deleted the directories with files within.


@antonio.dinapoli wrote:

Thanks a lot for your information.

Do you delete directories or files inside directories?

 


 

CreatePlease to create content
Content for Community-Ad
July's Community Spotlight Awards