ASA 5510 crashes frequently

lars.hecking · ‎09-02-2013

The ASA 5510 here is crashing frequently. We have been unable to find out why exactly this is happening, and the device is not under a service contract. The logging that is in place doesn't give any hints as to why this is happening. Since the last time we had problems, it was working fine for months, then it started crashing again every other day. When this happens, the device simply locks up and needs a power-cycle since console access is not working either.

Is this a know failure mode on this type of device, or with this version of the software? I can supply more detail if required.

ASA > show version

Cisco Adaptive Security Appliance Software Version 8.2(4)

Device Manager Version 6.3(5)

Compiled on Tue 14-Dec-10 12:00 by builders

System image file is "disk0:/asa824-k8.bin"

Config file at boot was "startup-config"

ASA up 20 hours 27 mins

Hardware: ASA5510, 256 MB RAM, CPU Pentium 4 Celeron 1599 MHz

Internal ATA Compact Flash, 64MB

BIOS Flash M50FW080 @ 0xffe00000, 1024KB

Encryption hardware device : Cisco ASA-55x0 on-board accelerator (revision 0x0)

Boot microcode : CN1000-MC-BOOT-2.00

SSL/IKE microcode: CNLite-MC-SSLm-PLUS-2.03

IPSec microcode : CNlite-MC-IPSECm-MAIN-2.05

0: Ext: Ethernet0/0 : address is 0017.5a88.735c, irq 9

1: Ext: Ethernet0/1 : address is 0017.5a88.735d, irq 9

2: Ext: Ethernet0/2 : address is 0017.5a88.735e, irq 9

3: Ext: Ethernet0/3 : address is 0017.5a88.735f, irq 9

4: Ext: Management0/0 : address is 0017.5a88.7360, irq 11

5: Int: Internal-Data0/0 : address is 0000.0001.0002, irq 11

6: Int: Internal-Control0/0 : address is 0000.0001.0001, irq 5

Licensed features for this platform:

Maximum Physical Interfaces : Unlimited

Maximum VLANs : 100

Inside Hosts : Unlimited

Failover : Active/Active

VPN-DES : Enabled

VPN-3DES-AES : Enabled

Security Contexts : 2

GTP/GPRS : Disabled

SSL VPN Peers : 2

Total VPN Peers : 250

Shared License : Disabled

AnyConnect for Mobile : Disabled

AnyConnect for Cisco VPN Phone : Disabled

AnyConnect Essentials : Disabled

Advanced Endpoint Assessment : Disabled

UC Phone Proxy Sessions : 2

Total UC Proxy Sessions : 2

Botnet Traffic Filter : Disabled

This platform has an ASA 5510 Security Plus license.

Serial Number: JMX1020K0R2

Running Activation Key: 0xcb091579 0xfccd86cb 0x5ce1f108 0xb24cc8fc 0x803dde88

Configuration register is 0x1

Configuration has not been modified since last system restart.

Jouni Forss · ‎09-02-2013

Hi,

I guess you already missing 2 of the usual options I would probably take since you dont have a service contract. And those would be software upgrade to the lastest version in your current software level (8.2(5)) and/or opening a TAC case.

How often does the device crash?

Has the device written Crash Info to the Flash?

dir flash:

show crashinfo

show crashinfo save

Is there anything special with the CPU and Memory utilization?

show cpu

show memory

Is there perhaps some configuration change that was done at the same time when the crashes started? Maybe you have some older backup configurations that you could compare to the current running configurations? For example save both configurations to their own files and compare the difference of these configurations with MS Word or some other software.

I am afraid that I probably wont be able to provide any help with this other than trying to think of place where to look.

I wasnt able to find any bug listed for 8.2(4) software that would cause crashes or I just didnt search with the correct keywords.

- Jouni

lars.hecking · ‎09-02-2013

Thanks, Jouni.

The device crashed/hung several times in March/April this year. Then it was fine for months and hung again on the 28th July, 26th August, 27th August, 31st August. In April, our network people cleaned up flash: since there seem to have been some space issues, which appeared to have helped.

The latest crash info on the device was from May 2009, so we may be looking at some kind of lock-up rather than crash.

ASA# dir flash:

Directory of disk0:/

96 -rwx 15261696 10:10:08 Apr 18 2011 asa824-k8.bin

97 -rwx 14812604 11:00:16 Sep 23 2011 asdm-635.bin

99 -rwx 13879296 07:42:42 May 11 2009 asa804-32-k8.bin

10 drwx 4096 03:42:44 Nov 22 2007 crypto_archive

101 drwx 4096 07:43:18 Mar 13 2009 sdesktop

3 drwx 4096 06:04:48 Mar 13 2009 log

11 drwx 4096 16:10:18 Apr 15 2011 coredumpinfo

62881792 bytes total (18407424 bytes free)

ASA# sh crashinfo

: Saved_Crash

Thread Name: aaa (Old pc 0x08069626 ebp 0xd45bd290)

Page fault: Address not mapped

vector 0x0000000e

edi 0xd45bd1f0

esi 0xd5e5e3c4

ebp 0xd45bd1c0

esp 0xd45bd1a8

ebx 0x00000000

edx 0x00000000

ecx 0x00000000

eax 0x00000000

error code 0x00000004

eip 0x08440898

cs 0x00000073

eflags 0x00013292

CR2 0x00000008

Cisco Adaptive Security Appliance Software Version 8.0(4)

Compiled on Thu 07-Aug-08 20:53 by builders

Hardware: ASA5510

Crashinfo collected on 14:02:31.703 UTC Wed May 6 2009

ASA# show crashinfo save

crashinfo save enable

ASA#

ASA# show cpu

CPU utilization for 5 seconds = 1%; 1 minute: 2%; 5 minutes: 2%

ASA# show memory

Free memory: 133770744 bytes (50%)

Used memory: 134664712 bytes (50%)

------------- ----------------

Total memory: 268435456 bytes (100%)

ASA#

I have a record of "soft" and "hard" crashes/hangs last year. Soft means the device stopped responding, but recovered by itself, probably within a few hours. Hard means that power-cycle was required. I have about four such events recorded for the Apr-Jun 2012 time frame, and no configuration changes between Oct 2011 and July 2012.

I will ask our NOC whether they can provide me with 8.2(5). Is there any way I can monitor cpu/mem on a more continuous basis, e.g. through snmp/mrtg?

iec1128759 · ‎09-02-2013

Check if the version of IOS has the minimum requirements, I had a similar problem with version 8.2 (5) and 256MB of RAM.

The ASA consumed all RAM memory, and crashed.

Greetings.

lars.hecking · ‎09-03-2013

The release notes for 8.2 say thta it uses more base memory and a RAM upgrade to 1GB is recommended if the device has less than 20% RAM free. This is not the case here - I've never seen memory consumption go higher than 50%, and cpu is usually well below 10%. But I have finally put some SNMP monitoring in place and will observe for a few days.

Marius Gunnerud · ‎09-03-2013

I have had a couple ASAs crash frequently on me in the past an in both cases it was a memory leak and replacing the RAM sorted things out.

Remember to take a backup of your config before doing any hardware changes if you haven't done so already.

--
Please remember to select a correct answer and rate helpful posts

lars.hecking · ‎09-05-2013

The status light at the back is flashing amber. Some research here suggests that this may indicate bad ram. I'll upgrade to 1GB and see how that goes.

lars.hecking · ‎09-26-2013

It's not a memory leak - used memory remains flat.

The status light at the back is still flashing amber after the DIMM upgrade. Any idea what that means?

After the RAM upgrade, I can really only be certain the problem has gone if it doesn't happen for at least six months as we have usually seen it happening anywhere from once a day to every few months.

An interesting observation: after the RAM upgrade, memory consumption jumped right up from around 135MB to 202MB. I was wondering why that is - does the device use excess RAM to copy data that is normally in firmware or PCI space, like certain desktop and workstation models do?

lars.hecking · ‎08-15-2014

Problem solved.

ASA was placed under maintenance and replaced the next time it crashed. No more problems since, which is a strong indication that the box had developed a fault.