topic Re: Critical Alarm : User space core files were detected (WAAS) in Application Networking

Critical Alarm : User space core files were detected (WAAS)

ohaddad — Mon, 05 Apr 2010 10:56:37 GMT

I have major alarms on WAVE-274 (Edge-WAAS) and WAVE-7341(Core-WAAS).

Alarm Name: Core_dump

Alarm informations: "User space core files were detected"

Is it possible to learn more about this - what caused it, ...ect

Any ideas on how to proceed?

Re: Critical Alarm : User space core files were detected (WAAS)

claudiols — Tue, 06 Apr 2010 22:31:30 GMT

Go to subdirectory crash, probably there is a dump file that you need to open a TAC case and send it

Re: Critical Alarm : User space core files were detected (WAAS)

ohaddad — Wed, 07 Apr 2010 16:21:15 GMT

Hi Claudiols;

Thank you for your reply,

But I wanted to know the exact subdirectory crash, and the dump file name if it's possible.

thank

Re: Critical Alarm : User space core files were detected (WAAS)

claudiols — Wed, 07 Apr 2010 16:45:55 GMT

the name of file can be very diferent, but the directory are

/local1/core_dir or /local1/crash

the file would be something like core.java.4.1.3.b55.cnbuild.5940.gz

Re: Critical Alarm : User space core files were detected (WAAS)

Harrisson Braga — Thu, 30 Sep 2010 13:08:37 GMT

What can I extract of this file core.java.4.1.3.b55.cnbuild.5940.gz ? It´s only information of
crash?

Only the TAC to resolve this problem?

regards

Re: Critical Alarm : User space core files were detected (WAAS)

fbergamo — Mon, 11 Oct 2010 18:16:57 GMT

Hi,

This is correct; you should open a TAC case and attach not only the core or crash files found in the directories mentioned, but also a sysreport generated in the device where the issue was observed as this investigation encompasses the analysis of several logs and possible correlation with known issues.

To generate a sysreport and store it to an FTP server, use this form of the command: copy sysreport ftp server-ip remote-directory remote-file-name

Best regards,

Fabio Bergamo

Cisco TAC

Re: Critical Alarm : User space core files were detected (WAAS)

Felix Arrieta — Thu, 02 Aug 2012 01:10:39 GMT

when you see alarm "core file detected" most of the time means that an internal process crash, it is hard to tell what exactly happens by looking at the core file itself so you better open a TAC case.

If you still want to learn about it, I can help with the following steps ( this is not documented anywhere so I don't promise covering all the steps, you'd need to be already familiar with reviewing WAAS logs to give it more sense)

scenario sample:

1- 'core files detected alarm shows up and you have no idea what it means.

2- We need to know when exactly the core file was created.

do WAE#dir find the folders "core_dir" and "crash" and open them

with core_dir folder :

WAE#dir core_dir

size time of last change name

-------------- ------------------------- -----------

618747 Sat Apr 19 06:30:14 2012 core.cnbuild.29405.gz

361568 Mon Aug 08 13:20:22 2008 core.cnbuild.11689.gz

so we found two core files over here and it's respective date.

***

delete the core files that are older ( deleting them fixes the alarm issue, why would we need to be alarm for a core file from a long time ago? corefiles are logs that help our developers to understand what process failed so this means whatever issue happen WAAS already recover from it, at least that's what we expect.

< how to delete a core file>

WAE#cd core_dir

WAE#delfile core.cnbuild.11689.gz

give it a minute

WAE# show alarms < the alarm will go again >

***

So once we know the date when the core file was created go and check the *syslog*

If you don't know how to check the syslog:

a- log/save your putty session ( so you can later open the log on a wordpad or any application to simplify research)

b- WAE#type-tail syslog.txt 500 < this number indicates how many lines you want the log to be, this can vary depending on how much log lines your WAE generates or how long ago the core file was created> for example if the core file was first seen 24 hours ago, I guess a syslog of 200 lines should do it but it could be more or less.

c- open the saved session(notepad or whatever..) search for the closer date to the core file date's, and we should see helpful logs that we could easily do research on the internet.

Most of the time if the core file is coming from a problem that requires urgent assistance you'll notice how the alarm that looks critical often repeats itself through the log ( because WAAS will try to recover from the unhealthy state) sometimes you delete a core file*clear the alarm* and it comes back few hours later, that's bad ..so you either help yourself trobleshooting it based on the logs or open a TAC case soon cause it might be causing trouble to your applications)

On the other hand if it was just one time behavior the core file can be easily deleted.

for example:

WAE device stops responding at 8:00 then it comes back online at 8:05, when it came back an internal process got crash, it then generates a core file when the service get's restore (let's say at 8:15) a week later you're trying to figure out what's the core file alarm about.

If you follow the steps above you'll end up reviewing the syslog and realize that before the core file was created the WAAS box got rebooted at 8:05!! end of the research...you might think why was this box rebooting? may be a power outage,hardware issue or someone messing up at the DC?

I am not saying this will give you an exact answer but you'd have something more than a core file not saying much, I have help many customers without asking higher support to decrypt the core file itself.

note:

-There are many logs for many different reasons the syslog I mention on this post is the 'general' log to say it like that... for more detailed logs for specific processes go to WAE#dir errorlog

-Don't forget to vist the WAAS troubleshooting page

http://docwiki.cisco.com/wiki/Cisco_WAAS_Troubleshooting_Guide_for_Release_4.1.3_and_Later_--_Troubleshooting_Optimization

-Don't start playing with WAAS on production devices!!!

I know this post is from months ago but I hope it can help other people too, cheers!

Felix

hi! 4 years after I have the

Harrisson Braga — Tue, 27 Jan 2015 18:30:24 GMT

hi! 4 years after I have the same problem, but in to directory core_dir no have files, in to the crash have the files above:

     size          time of last change             name
-------------- -------------------------          -----------
   17053068576 Tue Nov 11 11:18:43 2014           vmcore
           434 Tue Nov 11 13:53:42 2014           waas_version.txt

Can I delete vmcore? That file don't have extension .gz

Regards

Hi,Felix

antenehasnakeCSCO1311611171 — Sat, 20 May 2017 05:50:08 GMT

Hi,Felix

Thanks for clearing the air ,have one question on how can I download the core damp file in my local directory using CLI Command or GUI just like "sysreport", using FTP Server and or tftp server.

Hi,

finn.poulsen — Tue, 06 Jun 2017 10:44:51 GMT

Hi,

cd crash (or cd core_dir)

copy disk ftp <ftp-srv IP> <remote dir> <remote filename> <local filename>

...it will prompt you for credentials for the FTP server !

you can use the "." (period) as <remote dir> for the current (working) directory of the ftp server

Best regards

Finn