topic Show crashinfo doesn't say in Network Security

Ask the Expert: Troubleshooting Crashes in the Adaptive Security Appliances (ASA)

Monica Lluis — Tue, 12 Mar 2019 07:37:50 GMT

This session will provide an opportunity to learn and ask questions about how to troubleshoot issues with the Cisco Adaptive Security Appliance (ASAs), such as crashes, high CPUs, and other common issues. To participate in this event, ask your questions below by clicking on the "reply" button.

Ask questions from Monday May 2nd to Friday May 13, 2016

Featured Experts

Puneesh Chhabra is a Customer Support Engineer in the Cisco High-Touch Technical Services (HTTS) team based in Bangalore, India. He has total of 8 years of experience in network security. He has delivered multiple trainings on Cisco firewalls and VPN solutions. Prior to joining Cisco, he worked at IBM and HCL as network security consultant. Chhabra holds bachelor of Science degree in Computer Sciences from Kurukshetra University. He has achieved his CCIE certification in Security. (CCIE Security #30128)

Aditya Ganjoo is a Customer Support Engineer in the Cisco High-Touch Technical Services (HTTS) team based in Bangalore, India.He has been working with TAC from past 5 years in Security domains like Firewall, VPN and AAA. Aditya has delivered trainings on ASA and VPN technologies.Aditya holds a Bachelor's degree in Information Technology College – M.I.E.T College of Engineering and Technology from University of Jammu.He has achieved certifications for CCNA, CCNA-Security and is currently pursuing CCIE Security

Find other https://supportforums.cisco.com/expert-corner/events.

** Ratings Encourage Participation! **
Please be sure to rate the Answers to Questions

To ask your question, please use the reply button below.

Hello experts

Henrik Grankvist — Tue, 03 May 2016 11:11:00 GMT

Hello experts

I recently had failover in a HA-pair and it looks like the reason was that the primary firewall crashed. But the logs and show commands doesn't say much.

What would be the first step in troubleshooting the reason for this failover?

Hi Henrik,

Puneesh Chhabra — Tue, 03 May 2016 11:48:01 GMT

Hi Henrik,

Can you get me the "show version" and "show crashinfo" from the primary firewall ?

Regards,

Puneesh

Hi Henrik,

Aditya Ganjoo — Tue, 03 May 2016 11:49:44 GMT

Hi Henrik,

We need to check if there is any crash file generated on the primary ASA.

Use the command show crashinfo on the ASA to check that.

Also as per the show failover history I do not see any reason for failover except "Set by the config command" which means we have manually triggered the failover on the ASA.

Regards,

Aditya

Show crashinfo doesn't say

Henrik Grankvist — Tue, 03 May 2016 12:27:44 GMT

Show crashinfo doesn't say anything. Could it mean it didn't crash? The uptime on the primary firewall is 14 days which means it did atleast reload.

vpn-remote/pri/act#  show crashinfo 
INFO: This module has no crashinfo available.
INFO: Crashinfo file (flash:/crash.txt) not found 

------------------ show crashinfo module 1 ------------------

INFO: There is no module in slot 1 to retrieve crashinfo from.
INFO: This module has no crashinfo available.

vpn-remote/pri/act# show version

Cisco Adaptive Security Appliance Software Version 9.1(7) 
Device Manager Version 7.5(1)112

Compiled on Thu 14-Jan-16 09:37 by builders
System image file is "disk0:/asa917-k8.bin"
Config file at boot was "startup-config"

vpn-remote up 14 days 12 hours
failover cluster up 3 years 325 days

Hardware:   ASA5540, 2048 MB RAM, CPU Pentium 4 2000 MHz,
Internal ATA Compact Flash, 256MB
BIOS Flash M50FW080 @ 0xfff00000, 1024KB

Encryption hardware device : Cisco ASA-55xx on-board accelerator (revision 0x0)
                             Boot microcode        : CN1000-MC-BOOT-2.00 
                             SSL/IKE microcode     : CNlite-MC-SSLm-PLUS-2.08
                             IPSec microcode       : CNlite-MC-IPSECm-MAIN-2.09
                             Number of accelerators: 1

 0: Ext: GigabitEthernet0/0  : address is 0017.5a88.785e, irq 9
 1: Ext: GigabitEthernet0/1  : address is 0017.5a88.785f, irq 9
 2: Ext: GigabitEthernet0/2  : address is 0017.5a88.7860, irq 9
 3: Ext: GigabitEthernet0/3  : address is 0017.5a88.7861, irq 9
 4: Ext: Management0/0       : address is 0017.5a88.7862, irq 11
 5: Int: Not used            : irq 11
 6: Int: Not used            : irq 5

Licensed features for this platform:
Maximum Physical Interfaces       : Unlimited      perpetual
Maximum VLANs                     : 200            perpetual
Inside Hosts                      : Unlimited      perpetual
Failover                          : Active/Active  perpetual
Encryption-DES                    : Enabled        perpetual
Encryption-3DES-AES               : Enabled        perpetual
Security Contexts                 : 2              perpetual
GTP/GPRS                          : Disabled       perpetual
AnyConnect Premium Peers          : 2              perpetual
AnyConnect Essentials             : 2500           perpetual
Other VPN Peers                   : 5000           perpetual
Total VPN Peers                   : 5000           perpetual
Shared License                    : Disabled       perpetual
AnyConnect for Mobile             : Enabled        perpetual
AnyConnect for Cisco VPN Phone    : Disabled       perpetual
Advanced Endpoint Assessment      : Disabled       perpetual
UC Phone Proxy Sessions           : 2              perpetual
Total UC Proxy Sessions           : 2              perpetual
Botnet Traffic Filter             : Disabled       perpetual
Intercompany Media Engine         : Disabled       perpetual
Cluster                           : Disabled       perpetual

This platform has an ASA 5540 VPN Premium license.


Failover cluster licensed features for this platform:
Maximum Physical Interfaces       : Unlimited      perpetual
Maximum VLANs                     : 200            perpetual
Inside Hosts                      : Unlimited      perpetual
Failover                          : Active/Active  perpetual
Encryption-DES                    : Enabled        perpetual
Encryption-3DES-AES               : Enabled        perpetual
Security Contexts                 : 4              perpetual
GTP/GPRS                          : Disabled       perpetual
AnyConnect Premium Peers          : 4              perpetual
AnyConnect Essentials             : 2500           perpetual
Other VPN Peers                   : 5000           perpetual
Total VPN Peers                   : 5000           perpetual
Shared License                    : Disabled       perpetual
AnyConnect for Mobile             : Enabled        perpetual
AnyConnect for Cisco VPN Phone    : Disabled       perpetual
Advanced Endpoint Assessment      : Disabled       perpetual
UC Phone Proxy Sessions           : 4              perpetual
Total UC Proxy Sessions           : 4              perpetual
Botnet Traffic Filter             : Disabled       perpetual
Intercompany Media Engine         : Disabled       perpetual
Cluster                           : Disabled       perpetual

Yes, it doesn't look like it

Puneesh Chhabra — Tue, 03 May 2016 13:10:06 GMT

Yes, it doesn't look like it crashed. Did you check syslogs at the time of reload ?

Could it be a power glitch ?

Regards,

Puneesh

I included the syslog in my

Henrik Grankvist — Tue, 03 May 2016 13:41:46 GMT

I included the syslog in my first post, but it doesn't say much. Maybe I had the logging level set too high to show the reason.

Could be the power, but it gets its power through a UPS.

What logging settings would you recommend so that in the future see more information if this scenario happened again?

Hi Team,

Subin Kallidumpil — Tue, 03 May 2016 13:47:04 GMT

Hi Team,

I have a query around one of the deployment of ASA 5585.

Does access-list count have an affect on the ASA memory ?

I have a pretty high number of ACL on my ASA, are there any workarounds to optimize high memory ?

Regards

K Subin

Hi Subin,

Aditya Ganjoo — Tue, 03 May 2016 14:20:47 GMT

Hi Subin,

Yes ACE count on the ASA has a significant affect on the memory.

Use the command show access-list | in elements to check the ACE count.

As per Cisco on ASA 5585 here is the number of ACE's supported:

SSP-10: 500K
SSP-20: 750K
SSP-40: 1M
SSP-60: 2M

ACL count increase is a result of fully expanding original object-group ACEs to individual ACEs.

The magnitude of the explosion of an object-group ACE is determined by

1. The number of object groups that are present in the ACE, and
2. The number of objects (elements) in the object groups

The actual number of ACEs after expansion is obtained by multiplying the above two factors. For example, if an object-group ACE consists of a source-IP object group of 1000 elements, a destination-IP object group of 100 elements, and a service object group of 10 elements, then the object-group ACE is expanded to 1 million ACEs. The ACEs are then compiled into a lookup table for layer-3/4 packet classification.

You can use the command object-group-search access-control for reducing the ACE count on the ASA post 8.2 code.

Regards,

Aditya

Hi Henrik,

Aditya Ganjoo — Tue, 03 May 2016 14:44:21 GMT

Hi Henrik,

If this occurs in near future you can do the following things:

1) Enable the coredump on all ASA. The command for same is

Coredump enable filesystem <flash media>

///Please note , the ASA 5585 creates around 1 GB of file size on flash, in case you don't have that much of space on internal ASA flash, you can also use Cisco certified USB flash drives, in case they are available on all ASA unit.

*enabling coredump delay the reload of the system in the event of software forced reload. So expect extra time for ASA to reload and come back on line. Exact time will depend on size of coredump.

https://supportforums.cisco.com/document/59021/enabling-coredump-asa

2) Configure "informational" level syslog on ASA .

https://supportforums.cisco.com/document/73511/how-enable-syslogs-asa

Regards,

Aditya

Hello !We have ASA HA Pair

Murali — Tue, 03 May 2016 15:25:43 GMT

Hello !

We have ASA HA Pair running 8.1(2) code , suddenly active box reloaded and HA communication lost due to some interface issues.
We are interested in why Active box reloaded , is there any relation it could break the HA ? . i have attached crashinfo file.

Also is there any online tools available to analyze the crashinfo of ASA for general public ? Thanks in advance for your time.

HN/act# show ver | i up
Config file at boot was "startup-config"
pfw2-lnd30a-dclg up 6 years 155 days
failover cluster up 6 years 183 days

HN/stby# sh ver | i up
Config file at boot was "startup-config"
pfw2-lnd30a-dclg up 5 days 13 hours<-----------------
failover cluster up 6 years 183 days

HN/act# sh failover
Failover On
Failover unit Secondary
Failover LAN Interface: CROSSOVER GigabitEthernet3/3 (up)
Unit Poll frequency 1 seconds, holdtime 3 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 8 of 250 maximum
Version: Ours 8.1(2), Mate 8.1(2)
Last Failover at: 14:55:20 UTC Mar 10 2012
        This host: Secondary - Active
                Active time: 132100155 (sec)
                slot 0: ASA5580 hw/sw rev (1.0/8.1(2)) status (Up Sys)
                  Interface Admin (x.x.x.x): Normal
                  Interface LBCommon (x.x.x.x): Normal
                  Interface Stag (x.x.x.x): Normal
                  Interface Dev (x.x.x.x): Normal (Waiting)
                  Interface DB (x.x.x.x): Normal
                  Interface TesaDB (x.x.x.x): Normal
                  Interface ESX (x.x.x.x): Normal
                  Interface DBackup (x.x.x.x): Normal
        Other host: Primary - Failed
                Active time: 0 (sec)
                slot 0: ASA5580 hw/sw rev (1.0/8.1(2)) status (Up Sys)
                  Interface Admin (): Normal
                  Interface LBCommon (): Normal
                  Interface Stag (): Normal
                  Interface Dev (): No Link (Waiting)
                  Interface DB (): Normal
                  Interface TesaDB (): Normal
                  Interface ESX (): Normal
                  Interface DBackup (): Normal

Stateful Failover Logical Update Statistics
        Link : CROSSOVER GigabitEthernet3/3 (up)
        Stateful Obj    xmit       xerr       rcv        rerr
        General         598637093 0          411326427 1635
        sys cmd         27023867   0          27023866   0
        up time         0          0          0          0
        RPC services    0          0          0          0
        TCP conn        490335076 0          360266707 0
        UDP conn        40862947   0          645        0
        ARP tbl         40415203   0          24035209   1635
        Xlate_Timeout   0          0          0          0
        VPN IKE upd     0          0          0          0
        VPN IPSEC upd   0          0          0          0
        VPN CTCP upd    0          0          0          0
        VPN SDI upd     0          0          0          0
        VPN DHCP upd    0          0          0          0
        SIP Session     0          0          0          0

        Logical Update Queue Information
                        Cur     Max     Total
        Recv Q:         0       17      481911780
        Xmit Q:         0       1024    730744931

Hi Uda,

Puneesh Chhabra — Tue, 03 May 2016 15:51:20 GMT

Hi Uda,

Please attach complete "show version"

Regards,

Puneesh

/stby# sh verCisco

Murali — Tue, 03 May 2016 16:39:36 GMT

<HS>/stby# sh ver

Cisco Adaptive Security Appliance Software Version 8.1(2)
Device Manager Version 6.1(5)

Compiled on Thu 09-Oct-08 10:28 by builders
System image file is "disk0:/asa812-smp-k8.bin"
Config file at boot was "startup-config"

HS up 5 days 14 hours
failover cluster up 6 years 183 days

Hardware:   ASA5580-20, 8192 MB RAM, CPU AMD Opteron 2600 MHz
            2 CPUs, 4 cores
Internal ATA Compact Flash, 1024MB
BIOS Flash MX29LV320 @ 0xffc00000, 4096KB

Encryption hardware device : Cisco ASA-5580 on-board accelerator (revision 0x0)
                             Boot microcode   : CNPx-MC-BOOT-2.00
                             SSL/IKE microcode: CNPx-MC-SSLm-PLUS-2.01
                             IPSec microcode : CNPx-MC-IPSEC-MAIN-0002
Baseboard Management Controller (revision 0x1) Firmware Version: 1.119

0: Ext: Management0/0       : address is 0023.7d54.f87c, irq 11
1: Ext: Management0/1       : address is 0023.7d54.f87e, irq 10
2: Ext: GigabitEthernet3/0 : address is 0015.17c3.cc6c, irq 5
3: Ext: GigabitEthernet3/1 : address is 0015.17c3.cc6d, irq 11
4: Ext: GigabitEthernet3/2 : address is 0015.17c3.cc6e, irq 11
5: Ext: GigabitEthernet3/3 : address is 0015.17c3.cc6f, irq 10
6: Ext: GigabitEthernet4/0 : address is 0015.17c3.ccd0, irq 11
7: Ext: GigabitEthernet4/1 : address is 0015.17c3.ccd1, irq 10
8: Ext: GigabitEthernet4/2 : address is 0015.17c3.ccd2, irq 10
9: Ext: GigabitEthernet4/3 : address is 0015.17c3.ccd3, irq 11
10: Ext: GigabitEthernet6/0 : address is 0015.17c8.84ac, irq 10
11: Ext: GigabitEthernet6/1 : address is 0015.17c8.84ad, irq 11
12: Ext: GigabitEthernet6/2 : address is 0015.17c8.84ae, irq 11
13: Ext: GigabitEthernet6/3 : address is 0015.17c8.84af, irq 5

Licensed features for this platform:
Maximum Physical Interfaces : Unlimited
Maximum VLANs                : 250
Inside Hosts                 : Unlimited
Failover                     : Active/Active
VPN-DES                      : Enabled
VPN-3DES-AES                 : Enabled
Security Contexts            : 2
GTP/GPRS                     : Disabled
SSL VPN Peers                : 2
Total VPN Peers              : 10000
AnyConnect for Mobile        : Disabled
AnyConnect for Linksys phone : Disabled
Advanced Endpoint Assessment : Disabled
Licensed Cores               : 4

This platform has an ASA5580-20 VPN Premium license.

Serial Number: XXXXXXXX
Running Activation Key: XXXXXXXXXXXXX
Configuration register is 0x1
Configuration last modified by enable_15 at 11:07:57.838 UTC Thu Apr 28 2016

Hi Uda,

Puneesh Chhabra — Wed, 04 May 2016 06:22:34 GMT

Hi Uda,

I decoded the traceback and it looks like a watchdog failure. I could not point towards any known defect. However, there are quite a few Watchdog defects in the code you're running.

The code (8.1.2) is already end of sw maintenance and will soon be end of support:

http://www.cisco.com/c/en/us/products/collateral/security/asa-5500-series-next-generation-firewalls/end_of_life_c51_640618.html

I would suggest this action plan:

1. Enable the coredump on ASA to capture more detailed information if the crash occurs again.

Coredump enable filesystem <flash media>

Warning: Enabling coredump delay the reload of the system in the event of software forced reload. So expect extra time for ASA to reload and come back on line. Exact time will depend on size of coredump.

https://supportforums.cisco.com/document/59021/enabling-coredump-asa

2. Consider upgrading the ASA software to a newer image.

Regards,

Puneesh

Thanks for the review and

Murali — Wed, 04 May 2016 10:29:11 GMT

Thanks for the review and recommendations Puneesh !

Murali.

Hi Experts,

pathaksahil11 — Wed, 04 May 2016 12:04:55 GMT

Hi Experts,

We encountered an issue with ASA 5580 last month which became unresponsive. There was no internet access and it appeared the firewall was dropping all traffic. Physically, the lights seemed ok. After unsuccessful attempts to try and login to the asa, we decided to reboot the firewall, 15 minutes later everything started working!
As per TAC, nothing could be gathered from the logs.

Need to know if there is an option where ASA can failover to standby during such a scenario and can we prevent control/management plane being affected during an issue to enable an administrator to login and troubleshoot.

Hi ,

Puneesh Chhabra — Wed, 04 May 2016 12:43:31 GMT

Hi ,

Did you check the "show crashinfo" and uptime to verify if the ASA reloaded or crashed ?

Unfortunately, there is no way for the standby to take over unless it triggers one of the following:

•The unit has a hardware failure or a power failure.

•The unit has a software failure.

•Too many monitored interfaces fail. (Looks like in your case the hellos on the interface were sent and received properly. So, the rest of the interface tests were not performed)

•The no failover active command is entered on the active unit or the failover active command is entered on the standby unit.

To get to the root cause of what exactly happened at that time, we may require syslogs.

Also, please provide the software version you are running on the firewall.

Regards,

Puneesh

Thanks Puneesh Chhabra for

pathaksahil11 — Wed, 04 May 2016 12:52:13 GMT

Thanks Puneesh Chhabra for the insight!

As far as I know, no crashinfo was generated and the device has been running fine since then.

Hi Aditya,

Andreas Heykaus — Sun, 08 May 2016 19:12:57 GMT

Hi Aditya,

is there any overview about the no. of ACE's supported by ASA model?

Thanks

Andreas

Hi Andreas,

Aditya Ganjoo — Mon, 09 May 2016 04:32:54 GMT

Hi Andreas,

Yes, there is.

Here is the list:

ASA5512-x 100K
ASA5515-x 200K
ASA5525-x 300K
ASA5545-x 500K
ASA5555-x 500K

ASA5585

SSP-10: 500K
SSP-20: 750K
SSP-40: 1M
SSP-60: 2M

ASA-5540

500K ACE ACLs. Tested till 700K

ASA-5520:

200K ACE ACLs. Tested till 300K

ASA-5510:

80K ACE

ASA-5505:

25K ACE

Regards,

Aditya

Please rate helpful posts.