06-16-2012 02:48 AM
Good day,
We have an ANM Virtual Appliance, version 5.2, were we login and can go no further. This was working for fine for approximately two and half weeks. We created a backup and re-loaded the system via CLI with the same result. We logged in again via SSH and we have noted the following:
cscoanmsa/admin# sh disk
temp. space 4% used (141244 of 4951688)
disk: 7% used (353916 of 5935604)
Internal filesystems:
warning - /var is 100% used (89219000 of 89258112)
cscoanmsa/admin# sh application status ANM
% Error finding status information for the application: ANM
cscoanmsa/admin#
cscoanmsa/admin#
cscoanmsa/admin#
cscoanmsa/admin#
cscoanmsa/admin# sh application status ANM
The monit daemon 4.9 uptime: 23m
Process 'dcm' initializing
Process 'dal' initializing
Process 'ip-disc' Execution failed
Process 'licman' Execution failed
Process 'anm-fw-mon' running
Process 'mysql' Execution failed
System 'cscoanmsa' running
Java Processes:
licman : Stopped (None) [N/A]
dcm : Stopped (None) [N/A]
dal : Stopped (None) [N/A]
ip-disc : Stopped (None) [N/A]
Other Processes:
anm-fw-mon: Running (3641) [2012-06-15 15:48:55]
mysql : Stopped (None) [N/A]
cscoanmsa/admin# sh mem
total memory: 4148032 kB
free memory: 3616320 kB
cached: 342332 kB
swap-cached: 0 kB
Is there any way to access and clean out the /var directory from the CLI. is this achieved simply via the "delete" command with the full path ?
I believe this may be the issue, and we need to resolve this before we can move forward.
Any assistance or guidance would be greatly appreciated.
Thank you in advance.
Paul.
06-18-2012 12:49 AM
Hi Paul,
I'm afraid it's not easy. There is a patch that TAC can provide (you will need to open a case for it) that allows access to the filesystem. Unfortunately, this patch requires some disk space to install, so, in your situation, I'm not sure you will be able to.
The "/var" partition mostly holds the log files, so, another alternative is to make a backup of your current system, deploy a new virtual machine and restore the backup there. Since logs are not part of the backup, this will most likely solve your issue, at least for now.
Regards
Daniel
06-18-2012 01:57 AM
Hi Daniel,
Thank you for your response, If we are able to follow the route of logging a TAC case and installing the patch, how do we gain access to the file system? With the ADEOS login? Or similar to Stanalone server, via an OS login similar to "root"?
Would TAC provide all these details/processes, disk space requirements, etc. along with patch?
Also, we would eventually end up in a simialr situation, so a long term solution would be? To not have ACE send syslog's to ANM? There are approx. 18 ACE units currently implemented with around 20 contexts on each. Logging is set to informational. So, drop logging level or send syslog elsewhwere? Have you encountered this before on ANM VA? Could this also be from the Historical data being exported or not? Different location?
Will have to enquire on dropping level with customer.
Thanks again for your reply.
Regards
Paul.
06-18-2012 02:08 AM
Hi Paul,
I think I was not clear enough on my answer. You will not get access to the filesystem, the TAC engineer will need to get remote access to your ANM over a Webex session and he will get access to the system with a special password.
As for the final solution, you would first need to understand why the disk is getting full so quickly. Again, the TAC engineer should be able to assist with this if he gets access to the server.
Regards
Daniel
08-14-2012 04:33 AM
Any solution to this? facing same issue, unable to do backup since MySql is not running
08-14-2012 04:42 AM
Hi Michael,
I ended up trashing the VM and restoring from a backup. The issue I had was that my devices were sending syslogs from sev 0-5, I changed this to 0-2 as recommended by the documentation.
However i8f you log a TAC case they can supply you with a patch that will enable them to have "root" access onto the server and can delete all the large log files.
I am hoping that Cisco build in some kind of log rotation system in future releases, that would save us alot of time.
I hope the patch works for you !
Joan
05-08-2014 06:01 AM
We've seen this on ANM 5.2.1 as well. In our case it was that /var/log/messages was taking up nearly 85GB on the var volume. ANM, at least in 5.2.1, doesn't seem to rotate this log file.
The way we fixed it was to mount the var volume using a Linux live CD like Knoppix and clear our /var/log/messages. Very unsupported, but gets you out of a hole:
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: