Cisco Prime Infra 2.2 Crash

[saxenanitesh8522]

Hi,

We had a issue with cisco prime 2.2 which we have been using for long time.

Now we had a issue where the prime is crashed and the only available backup which we have is 2 years old.

We had two to three issues:-

1. OPT Volume was full --> NCS Clean up fixed this issue

2. VAR volume is full --> Not able to fix this issue? --> Any suggestions how to fix this issue?

3. In the logs it was oracle memory issue --> i think is released to space issue which could be fixed if i fix point 2.

Post doing NCS clean up --> now its staying the DB not start.

I understand we can do ncs db reinit --> but will i lose all the data? if i am trying to do a backup as well its not working as it didnt get finished only.

Also is it possible to get the wireless map data from the CLI some place as we are using that mosly.

Any suggestions??

[saxenanitesh8522]

ncs start verbose

Starting Prime Infrastructure...

Reporting Server Heap size = 4096m
XMP Server Heap size = 6144m
Starting Health Monitor
Starting Health Monitor as a primary
Checking for Port 8082 availability... OK
CERT MATCHED :
Updating web server configuration file ...
Starting Health Montior Web Server...
Health Monitor Web Server Started.
Setting UID to 441:110
UID set to 441:110
Starting Health Monitor Server...
Health Monitor Server Started.

Failure during Prime Infrastructure startup. Check launchout.log for details.

Starting strongSwan 5.0.1 IPsec [starter]...

When seeing the status is showing db error

------------

/dev/mapper/smosvg-optvol 647G 454G 160G 74% /opt

/dev/mapper/smosvg-varvol 3.8G 3.8G 0 100% /var --> Concern on this

[marce1000]

   >VAR volume is full --> Not able to fix this issue? --> Any suggestions how to fix this issue?

 - You should probably cleanup the directory under a root shell, as far as I remember in Prime 2.x you become root with the shell-command root.  Below is a link to finding biggest directories and files. Of course not everything can be deleted but look for directory related to logging (e.g.). Same applies for other partitions that are full :

            https://www.tecmint.com/find-top-large-directories-and-files-sizes-in-linux/

 M,

[saxenanitesh8522]

Will doing a ncs db reinit --> as a solution can that help in re-initializing the database or will it delete all my database?

[marce1000]

 - Yes, it will delete the content of the database and can be considered as being a 'factory defaults reset' for Cisco Prime.

 M.

[saxenanitesh8522]

hi..

i checked --> /dev/mapper/smosvg-varvol  3.8G 3.8G 0 100% /var 

 i should be deleting these logs but not sure can you please let me know if that okay to delete?

So i checked under /var/logs --> 

ade # cd log/
ade # ls -l
total 163604

86556 cron
275570 cron.1
241104 cron.2
275361 cron.3
275352 cron.4
147168 lastlog
221835 maillog
696894 maillog.1
543885 maillog.2
701135 maillog.3
703399 maillog.4
15508693 secure
39034880 secure.1
22847488 secure.2
41873408 secure.3
43315200 secure.4

[marce1000]

 - You can delete all the red-colored-files , except for the first maillog (the one without extension). The other ones are result for log-rotating and will appear again after some time.

 M.

[saxenanitesh8522]

but these are in MB i cant find anything which i think can remove like 100 or 200 mb files from the same.

Is there any place else in /var other than logs.

[marce1000]

 - Following root-shell command examples from the link below, you may look for big files on the  whole of the  /var partition, if their names relate to logging and or temporary usage, you can delete them (too) :

            https://www.cyberciti.biz/faq/linux-find-largest-file-in-directory-recursively-using-find-du/

 M.

[saxenanitesh8522]

hi,

tried deleting those under secure.1 to .4 didnt help and its showing /var 100%

ran the command for top 20 in /var folder

ade # sudo find /var -type f -printf "%s\t%p\n" | sort -n | tail -20
1041141 /var/log/sa/sar27
1041141 /var/log/sa/sar28
1041141 /var/log/sa/sar29
1060080 /var/log/sa/sa24
1060080 /var/log/sa/sa25
1060080 /var/log/sa/sa26
1060080 /var/log/sa/sa27
1060080 /var/log/sa/sa28
1060080 /var/log/sa/sa29
1060080 /var/log/sa/sa31
1318912 /var/lib/rpm/__db.002
1388779 /var/log/audit/audit.log
2641920 /var/lib/rpm/Filemd5s
2945024 /var/lib/rpm/Basenames
5243040 /var/log/audit/audit.log.3
5243060 /var/log/audit/audit.log.1
5243067 /var/log/audit/audit.log.2
15514164 /var/log/secure
20729856 /var/lib/rpm/Packages
23988767 /var/lib/mlocate/mlocate.db

--------------

backup ciscprime repository defaultRepo application NCS

DO NOT press ^C while the backup is in progress
Aborting backup with a ^C may terminate the backup operation or the backup file may be corrupted

Checking for arhivelog cleanup job
Backup Started at : 06/09/20 15:19:42
DB size is 318 GB --> this even possible?
Available size is 172 GB
you might run out of disk space in repository, Do you want to proceed with backup
do you want to proceed [y/n]?

How am i suppose to take a backup if there is not enough space 

[marce1000]

 - You may want to cleanup the database with ncs stop ; ncs cleanup ; ncs start. Subsequently if there is still not enough space you could backup to an external repository,

 M.

[saxenanitesh8522]

hi

i did the ncs cleanup but no luck that was my first thing to do.

that /var vloume i am not able to cleanup space and it keeps stuck on db error when trying to start the services.

[marce1000]

 - You may then need to re-install Cisco prime and restore from a valid backup. Note that as far as partition sizing being provided , choose the installation type according to the size of your network and or the number of devices which will be entered into Prime, check the following info-resources :

    https://www.cisco.com/c/en/us/td/docs/net_mgmt/prime/infrastructure/3-7/quickstart/guide/bk_Cisco_Prime_Infrastructure_3_7_0_Quick_Start_Guide.html#con_1070235

  https://www.cisco.com/c/en/us/td/docs/net_mgmt/prime/infrastructure/3-3/quickstart/guide/bk_Cisco_Prime_Infrastructure_3_3_0_Quick_Start_Guide.html#con_1070021

 M.