cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
6104
Views
10
Helpful
13
Replies

Problem with Prime Infrastucture 3.1.3 /opt disk space full

danlin
Level 1
Level 1

 

It's a small installation with less than 100 devices

When i run following commands the result showing the following:
ade # df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/smosvg-rootvol
3.9G 324M 3.4G 9% /
tmpfs 32G 11G 21G 34% /dev/shm
/dev/mapper/smosvg-altrootvol
93M 5.6M 83M 7% /altroot
/dev/sda1 485M 48M 412M 11% /boot
/dev/mapper/smosvg-home
93M 6.2M 83M 7% /home
/dev/mapper/smosvg-localdiskvol
244G 188M 231G 1% /localdisk
/dev/mapper/smosvg-optvol
1.4T 1.3T 6.6G 100% /opt
/dev/mapper/smosvg-recvol
93M 5.6M 83M 7% /recovery
/dev/sda2 97M 5.6M 87M 7% /storedconfig
/dev/mapper/smosvg-storeddatavol
9.5G 151M 8.9G 2% /storeddata
/dev/mapper/smosvg-tmpvol
2.0G 36M 1.8G 2% /tmp
/dev/mapper/smosvg-usrvol
6.8G 967M 5.5G 15% /usr
/dev/mapper/smosvg-varvol
3.9G 194M 3.5G 6% /var
ade #

ade # df /opt
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/smosvg-optvol
1424876728 1345588408 6908724 100% /opt
ade #

ade # du -h --max-depth=6 /opt | grep [0-9]G | sort -k2
du: cannot read directory `/opt/lost+found': Permission denied
du: cannot read directory `/opt/CSCOlumos/rcmds': Permission denied
du: cannot read directory `/opt/oracle/base/fast_recovery_area/WCS/archivelog/2018_04_12': Permission denied
12G /opt/CSCOlumos
1.5G /opt/CSCOlumos/logs
64K /opt/CSCOlumos/matlab/toolbox/matlab/imagesci/@H5G
2.9G /opt/CSCOlumos/staging
1.6G /opt/CSCOlumos/staging/app
3.1G /opt/CSCOlumos/updates
1.4G /opt/CSCOlumos/updates/PI_3_1_3-1.0.16
1.6G /opt/oracle/base/diag/tnslsnr
1.6G /opt/oracle/base/diag/tnslsnr/FMCL-PRIME1
1.6G /opt/oracle/base/diag/tnslsnr/FMCL-PRIME1/wcstns
3.3G /opt/oracle/base/fast_recovery_area
3.1G /opt/oracle/base/fast_recovery_area/WCS
3.1G /opt/oracle/base/fast_recovery_area/WCS/onlinelog
29G /opt/oracle/base/oradata
1.4G /opt/oracle/base/oradata/STBYWCS
1.2G /opt/oracle/base/oradata/STBYWCS/datafile
28G /opt/oracle/base/oradata/WCS
28G /opt/oracle/base/oradata/WCS/datafile
3.0G /opt/oracle/base/product
3.0G /opt/oracle/base/product/12.1.0
3.0G /opt/oracle/base/product/12.1.0/dbhome_1
1.2G /opt/oracle/base/product/12.1.0/dbhome_1/lib
ade #
What files can i remove or minimize to free up more space?

I have tried ncs cleanup this solved the problem temporarily.

13 Replies 13

m.hegeraat
Level 1
Level 1

There is nothing you can do to bring the DB to a normal proportion.

 

Do ncs stop.

Now you can add a disk!! Do not make any disk bigger, it will not work.

reboot the system

The OS will see the new disk space on reboot and add it to the partitions.

 

I would suggest to do an export of everything you can export in PI, devices, maps, groups, etc.

Then document everything you cannot export.

Then deploy the OVA again and import.

 

Then check the data retention settings and reduce them to the minimum acceptable for you.

Then make sure you get an alarm from PI when disk space usage on PI goes over the limit.

 

  

The Prime server is not virtual, its a Cisco UCS C220 M4 Server
So it's not easy to just add a disk at the moment, i'm not sure if there is more slots. Need to check that.

Leo Laohoo
Hall of Fame
Hall of Fame
This is a known bug with 3.1. Upgrade to 3.3 or 3.4.

Can there be a problem to upgrade when there is no space left in /opt ?
Maybe the installation needs more space?

The easiest way to upgrade PI is a complete rebuild.
Export all your maps and data. Re-install 3.4 and import the maps and data.

Thanks for reply!

Ok, when you say export maps and data.
Whats the best way to do that? I'll have backup running to external ftp

Should i'll reinstall to 3.4 and then apply the backup from 3.1?

A backup also requires space in /opt even if it is 'remote'

You will have to add a disk, so you can make the system work for a short time, so you can export things.

Or you can try to deploy 3.4 or 3.3 and restore the last backup.

But also then I would do an export of everything possible, deploy the ova again and reimport.


@danlin wrote:
Thanks for reply!

Ok, when you say export maps and data.
Whats the best way to do that? I'll have backup running to external ftp

Should i'll reinstall to 3.4 and then apply the backup from 3.1?

That's correct.

This is the graph showing disk space over time :)disk historik prime.png

The setting of a threshold bigger or equal to 100% is asking for troubles.

This is manually set by someone.

I believe out-of-the-box it is 40%

 

After some deepdiving in the filesystem i found the source of problem.

In this directory below there was thousands of directories named incdir_XXXXX

and they contained very big .trc and .trm files.

/opt/oracle/base/diag/rdbms/wcs/wcs/incident

 

After contact with TAC the solution was to remove files, and now i got more than 1TB empty space

This may be some useful info to someone.

 

I don't now why the alarm threshold is 90% but i'm pretty sure that's nothing we have changed since installation. Checked an other installation and there was also 90%?!
But now it's changed to 65%

 

Thanks a lot for your information.

Do you delete directories or files inside directories?

 

I deleted the directories with files within.


@antonio.dinapoli wrote:

Thanks a lot for your information.

Do you delete directories or files inside directories?

 


 

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: