cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2909
Views
0
Helpful
6
Replies

ANM 5.2 Virtual Appliance Internal filesystems problem

Paul Pinto
Level 1
Level 1

Good day,

We have an ANM Virtual Appliance, version 5.2, were we login and can go no further. This was working for fine for approximately two and half weeks. We created a backup and re-loaded the system via CLI with the same result. We logged in again via SSH and we have noted the following:

cscoanmsa/admin# sh disk

temp. space 4% used (141244 of 4951688)

disk: 7% used (353916 of 5935604)

Internal filesystems:

  warning - /var is 100% used (89219000 of 89258112)

cscoanmsa/admin# sh application status ANM

% Error finding status information for the application: ANM

cscoanmsa/admin#

cscoanmsa/admin#

cscoanmsa/admin#

cscoanmsa/admin#

cscoanmsa/admin# sh application status ANM

The monit daemon 4.9 uptime: 23m

Process 'dcm'                       initializing

Process 'dal'                       initializing

Process 'ip-disc'                   Execution failed

Process 'licman'                    Execution failed

Process 'anm-fw-mon'                running

Process 'mysql'                     Execution failed

System 'cscoanmsa'                  running

Java Processes:

licman    : Stopped (None)  [N/A]

dcm       : Stopped (None)  [N/A]

dal       : Stopped (None)  [N/A]

ip-disc   : Stopped (None)  [N/A]

Other Processes:

anm-fw-mon: Running (3641)  [2012-06-15 15:48:55]

mysql     : Stopped (None)  [N/A]

cscoanmsa/admin# sh mem

total memory:    4148032 kB

free memory:     3616320 kB

cached:           342332 kB

swap-cached:           0 kB

Is there any way to access and clean out the /var directory from the CLI. is this achieved simply via the "delete" command with the full path ?

I believe this may be the issue, and we need to resolve this before we can move forward.

Any assistance or guidance would be greatly appreciated.

Thank you in advance.

Paul.

6 Replies 6

Daniel Arrondo Ostiz
Cisco Employee
Cisco Employee

Hi Paul,

I'm afraid it's not easy. There is a patch that TAC can provide (you will need to open a case for it) that allows access to the filesystem. Unfortunately, this patch requires some disk space to install, so, in your situation, I'm not sure you will be able to.

The "/var" partition mostly holds the log files, so, another alternative is to make a backup of your current system, deploy a new virtual machine and restore the backup there. Since logs are not part of the backup, this will most likely solve your issue, at least for now.

Regards

Daniel

Hi Daniel,

Thank you for your response, If we are able to follow the route of logging a TAC case and installing the patch, how do we gain access to the file system? With the ADEOS login? Or similar to Stanalone server, via an OS login similar to "root"?

Would TAC provide all these details/processes, disk space requirements, etc. along with patch?

Also, we would eventually end up in a simialr situation, so a long term solution would be? To not have ACE send syslog's to ANM? There are approx. 18 ACE units currently implemented with around 20 contexts on each. Logging is set to informational. So, drop logging level or send syslog elsewhwere? Have you encountered this before on ANM VA? Could this also be from the Historical data being exported or not? Different location?

Will have to enquire on dropping level with customer.

Thanks again for your reply.

Regards

Paul.

Hi Paul,

I think I was not clear enough on my answer. You will not get access to the filesystem, the TAC engineer will need to get remote access to your ANM over a Webex session and he will get access to the system with a special password.

As for the final solution, you would first need to understand why the disk is getting full so quickly. Again, the TAC engineer should be able to assist with this if he gets access to the server.

Regards

Daniel

Any solution to this? facing same issue, unable to do backup since MySql is not running

Hi Michael,

I ended up trashing the VM and restoring from a backup. The issue I had was that my devices were sending syslogs from sev 0-5, I changed this to 0-2 as recommended by the documentation.

However i8f you log a TAC case they can supply you with a patch that will enable them to have "root" access onto the server and can delete all the large log files.

I am hoping that Cisco build in some kind of log rotation system in future releases, that would save us alot of time.

I hope the patch works for you !

Joan

echristi1
Level 1
Level 1

We've seen this on ANM 5.2.1 as well. In our case it was that /var/log/messages was taking up nearly 85GB on the var volume. ANM, at least in 5.2.1, doesn't seem to rotate this log file.

The way we fixed it was to mount the var volume using a Linux live CD like Knoppix and clear our /var/log/messages. Very unsupported, but gets you out of a hole:

Boot the ANM machine with Knoppix (KNOPPIX_V7.2.0DVD-2013-06-16-EN.iso)
 
As root run:
 
pvs (to work out LVM physical volumes and volume groups)
 
PV        VG
/dev/sda2 smosvg
 
vgchange -ay smosvg (to activate the volume group)
 
lvscan (to work out what logical volumes are available)
 
ACTIVE '/dev/smosvg/varvol'
 
mount /dev/mapper/smosvg-varvol /mnt
echo > /mnt/log/messages
umount /mnt
reboot
Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: