cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
65926
Views
5
Helpful
12
Comments
Rollin Kibbe
Cisco Employee
Cisco Employee

Background:  During NCS' short life, a number of customers had problems (NCS failing to start, backups failing, etc.) where exhausting disk space was at the root, but they had overlooked the gentle warnings that a critical alarm gave.  Prime Infrastructure 1.2 has disk space messages that are highly intrusive, and were meant to be intrusive so customers wouldn't run out of space and not know it.

The threshold for these alarms is a hard set 50% utilization.  At the most basic level, in order to copy out temp files to create a backup copy of the database, the /opt partition shouldn't be more than 50% utilized.  Say, for example, one's /opt partition is 100GB, and their database is 75 GB, where is Prime Infrastructure supposed to be able to put 75 GB of temp files in order to create a backup?  The 50% threshold is very important, particularly if NFS mounts are not being used to hold these temp files or the final backup.tar.gpg files the backup process creates. 

In all installations, if the utilization level of the /opt partition (or /dev/mapper/smosvg-optvol in the Prime Infrastructure GUI under Administration > Appliance) is less than 60%, it's possible that the temporary archive log files from the Oracle database have accumulated and can be cleaned with the following commands at the Prime Infrastructure command line interface.

Prime Infrastructure release 1.3.0.20 resolved a number of backup/restore, migration and database issues.  The subsequent 1.3.1 patch (filename PI_1.3.0.20_Update_1-12.tar.gz ) has fixed even more issues with backup/restore and reduced the filesize of the resulting backup.tar.gpg file that the backup process creates.  Be sure to keep up to date with the latest bug fixes by upgrading to 1.3.0.20 and any subsequent patches, as soon as practicably possible.

ncs stop
ncs cleanup
ncs start

NOTE:  It's critical that Prime Infrastructure be stopped BEFORE starting an ncs cleanup.    In some installations that are on the border of alarming, the "ncs cleanup" command may clear enough space to stop the alarms. 

In installations with a virtual appliance, adding additional disk space is straightforward:  Stop Prime Infrastructure with the "halt" command, add a second virtual hard disk to the VM, then power the VM back on.  On boot up, the file system table will be updated with the new disk, and Prime Infrastructure will detect it and add it to both the /opt and /localdisk partitions in an 85%/15% mix.  On logging into the GUI after adding the additional virtual hard disk, the disk space alarms will be gone.  Unfortunately, increasing the size of the existing virtual hard disk in the virtual machine is not an option, as Prime Infrastructure's disk grow utility will not see that;  it is looking for an additional virtual hard disk to be added to the VM.

In installations where High Availability (HA) is configured, remove HA

Administration > High Availability > HA Configuration > Remove button 

before adding additional disk space. The primary and secondary can receive the additional disk space separately and brought back online, then HA restored.

Comments
djrutledge
Level 1
Level 1

Hi Rollin,

I have an NCS install of 1.1.1.24 and would like to run the ncs cleanup command to free up space. We currenly have a warning of "NCS disk volume 'optvol' is above the recommended disk usage, more than '50'% used." It is at 62% now.

Is content such as floorplans, APs, and controllers preserved when running the cleanup?

Also when logging into the console on the admin account I do not see the  /dev/ or /localdisk/ directories on the disk when using 'dir'. Any ideas  why?

sschmidt
Cisco Employee
Cisco Employee

Hello,

"Is content such as floorplans, APs, and controllers preserved when running the cleanup? "

Yes these are not affected.

"Also when logging into the console on the admin account I do not see  the  /dev/ or /localdisk/ directories on the disk when using 'dir'. Any  ideas  why?" 

You would not see these directories from the cli as admin.  NCS is a locked down system so the underlying linux OS was not available unless a rootpatch was applied.  This restriction has beem removed in Prime Infrastructure.

djrutledge
Level 1
Level 1

Thanks!

bjmcmaster
Level 1
Level 1

will adding MSE with location services increase prime db size significantly?   If so, is there a ratio of clients/disk space?

Craig Le-Butt
Level 1
Level 1

By the look of it, Primes backup does include MSE data.

We've only had it installed for 8 days, and the back up has doubled in size.

In the WCS the avarage back up was around 2.5Gb, the full export from WCS for Prime install was 4.5Gb.


hope this helps

Hi

I have done the ncs cleanup on a 1.3.0.20 software, but the 3 point deleting files from users i said "no".

After cleanup and ncs start the error is the same!!!

93%disk full

can i delete manually the two big files??

ade # ls -l

total 304640

drwxr-xr-x 2 root root        4096 Oct  1 04:17 Prime_Backups

lrwxrwxrwx 1 root root          20 Feb 14  2013 crash -> /opt/CSCOlumos/crash

drwxr-xr-x 2 root root        4096 Oct 15 18:11 defaultRepo

-rw-rw-rw- 1 root gadmin  26938358 Apr  2  2013 dump.dump.gz

drwxr-xr-x 3 root root        4096 Jul 11 10:48 ftp

-rw-rw-rw- 1 root gadmin        85 Mar 22  2013 launchout

-rw-rw-rw- 1 root gadmin 284656177 Mar 22  2013 launchout.log

drwx------ 2 root root       16384 Dec 12  2012 lost+found

drwxr-xr-x 2 root root        4096 Oct 18 11:02 tftp

ade #

ade #

Disk Source     Available Space(KB)     Used Space(KB)     % Used

/dev/mapper/smosvg-rootvol     3451440     311440     8

/dev/mapper/smosvg-storeddatavol     9238544     153708     2

/dev/mapper/smosvg-localdiskvol     3041360     39283452     93

/dev/sda2     88342     5688     6

/dev/mapper/smosvg-home     84612     5668     6

/dev/mapper/smosvg-optvol     104744464     93907328     47

/dev/mapper/smosvg-varvol     3619304     143576     4

/dev/mapper/smosvg-usrvol     5466184     1096260     17

/dev/mapper/smosvg-tmpvol     1828796     37576     2

/dev/mapper/smosvg-recvol     84616     5664     6

/dev/mapper/smosvg-altrootvol     84616     5664     6

/dev/sda1     452600     17644     4

tmpfs     5795072     5447640     48

Craig Le-Butt
Level 1
Level 1

My database got corrupted last week,  so I did the reinilise the database, then imported the databack, this clear up the space issue.

You may find it easier to rebuild the VM server, not as bad as it sounds, this has curred the space issue.

One of the problems is you can't defrag the drive like on windows.

Radium Fu
Level 1
Level 1

I wondering is there a way to setup a auto run job, to do this disk clean up things, maybe every 2 week for my environment, as I got warring every other week.

alicia.villegas
Community Member

Hi Rollin,

If I add more space to the Cisco Prime VM, will I loose any data. I can not backup because of running out of space alarm?

From my experience you'll not loose any data. But you will need resize LVM group in order to make available new disk space to underlying Linux OS. You need to open case in TAC to do it.

fredrik.lundberg
Community Member

We have been running into the "full database" problem a few times. This is what I did. (The prime is running on VMWare - so take a snapshot before )

1. login with ssh and type "shell".

2. Check "df -h" if the /opt is full..

3. Shut down the database by "exit" the shell and type "ncs stop".

4. "Halt" the system and go to VMWare to extend the disk.

5. Then I booted the server with Gparted and extended the filesystem. (use any partition software)

6. Boot up the Prime and login to the "shell" again. Make sure the database is stopped "ncs stop".

7. Check the volume group: ade # sudo vgdisplay 

There should now be "free PE / Size" to use.

Also find out the LV path to use:  ade # sudo lvdisplay  (Look for the /optvol)

8. Extend the logical volume: ade # sudo lvextend -L+10G /dev/smosvg/optvol

9. Rezise the filesystem by: ade # sudo resize2fs /dev/smosvg/optvol

10. Reboot the server "reboot".

Aomar bahloul
Spotlight
Spotlight

Hi Frederik,

Thank you for your post it was very helpful.

Aomar.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: