Ask the Expert: Troubleshooting Crashes in the Adaptive Security Appliances (ASA)

[Monica Lluis]

This session will provide an opportunity to learn and ask questions about how to troubleshoot issues with the Cisco Adaptive Security Appliance (ASAs), such as crashes, high CPUs, and other common issues. To participate in this event, ask your questions below by clicking on the "reply" button. 

Ask questions from Monday May 2nd  to Friday May 13, 2016

Featured Experts

Puneesh Chhabra is a Customer Support Engineer in the Cisco High-Touch Technical Services (HTTS) team based in  Bangalore, India. He has total of 8 years of experience in network security. He has delivered multiple trainings on Cisco firewalls and VPN solutions. Prior to joining Cisco, he worked at IBM and HCL as network security consultant. Chhabra holds bachelor of Science degree in Computer Sciences from Kurukshetra University. He has achieved his CCIE certification in Security. (CCIE Security #30128)

Aditya Ganjoo is a Customer Support Engineer in the Cisco High-Touch Technical Services (HTTS) team based in  Bangalore, India.He has been working with TAC from past 5 years in Security domains like Firewall, VPN and AAA. Aditya has delivered trainings on ASA and VPN technologies.Aditya holds a Bachelor's degree in Information Technology College – M.I.E.T College of Engineering and Technology from University of Jammu.He has achieved certifications for CCNA, CCNA-Security and is currently pursuing CCIE Security

Find other  https://supportforums.cisco.com/expert-corner/events.

** Ratings Encourage Participation! **
Please be sure to rate the Answers to Questions

To ask your question, please use the reply button below.
 

[Aditya Ganjoo]

Hi Andreas,

Yes, there is.

Here is the list:

ASA5512-x 100K 
ASA5515-x 200K 
ASA5525-x 300K 
ASA5545-x 500K 
ASA5555-x 500K 

ASA5585

SSP-10: 500K
SSP-20: 750K
SSP-40: 1M
SSP-60: 2M

ASA-5540

500K ACE ACLs. Tested till 700K

ASA-5520:

200K ACE ACLs. Tested till 300K

ASA-5510:

80K ACE

ASA-5505:

25K ACE

Regards,

Aditya

Please rate helpful posts.

[Murali]

Hello !

We have ASA HA Pair running 8.1(2) code , suddenly active box reloaded and HA communication lost due to some interface issues.
We are interested in why Active box reloaded , is there any relation it could break the HA ? . i have attached crashinfo file.

Also is there any online tools available to analyze the crashinfo of ASA for general public ? Thanks in advance for your time.

HN/act# show ver | i up
Config file at boot was "startup-config"
pfw2-lnd30a-dclg up 6 years 155 days
failover cluster up 6 years 183 days

HN/stby# sh ver | i up
Config file at boot was "startup-config"
pfw2-lnd30a-dclg up 5 days 13 hours<-----------------
failover cluster up 6 years 183 days

HN/act# sh failover
Failover On
Failover unit Secondary
Failover LAN Interface: CROSSOVER GigabitEthernet3/3 (up)
Unit Poll frequency 1 seconds, holdtime 3 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 8 of 250 maximum
Version: Ours 8.1(2), Mate 8.1(2)
Last Failover at: 14:55:20 UTC Mar 10 2012
        This host: Secondary - Active
                Active time: 132100155 (sec)
                slot 0: ASA5580 hw/sw rev (1.0/8.1(2)) status (Up Sys)
                  Interface Admin (x.x.x.x): Normal
                  Interface LBCommon (x.x.x.x): Normal
                  Interface Stag (x.x.x.x): Normal
                  Interface Dev (x.x.x.x): Normal (Waiting)
                  Interface DB (x.x.x.x): Normal
                  Interface TesaDB (x.x.x.x): Normal
                  Interface ESX (x.x.x.x): Normal
                  Interface DBackup (x.x.x.x): Normal
        Other host: Primary - Failed
                Active time: 0 (sec)
                slot 0: ASA5580 hw/sw rev (1.0/8.1(2)) status (Up Sys)
                  Interface Admin (): Normal
                  Interface LBCommon (): Normal
                  Interface Stag (): Normal
                  Interface Dev (): No Link (Waiting)
                  Interface DB (): Normal
                  Interface TesaDB (): Normal
                  Interface ESX (): Normal
                  Interface DBackup (): Normal

Stateful Failover Logical Update Statistics
        Link : CROSSOVER GigabitEthernet3/3 (up)
        Stateful Obj    xmit       xerr       rcv        rerr
        General         598637093  0          411326427  1635
        sys cmd         27023867   0          27023866   0
        up time         0          0          0          0
        RPC services    0          0          0          0
        TCP conn        490335076  0          360266707  0
        UDP conn        40862947   0          645        0
        ARP tbl         40415203   0          24035209   1635
        Xlate_Timeout   0          0          0          0
        VPN IKE upd     0          0          0          0
        VPN IPSEC upd   0          0          0          0
        VPN CTCP upd    0          0          0          0
        VPN SDI upd     0          0          0          0
        VPN DHCP upd    0          0          0          0
        SIP Session     0          0          0          0

        Logical Update Queue Information
                        Cur     Max     Total
        Recv Q:         0       17      481911780
        Xmit Q:         0       1024    730744931

[Puneesh Chhabra]

Hi Uda,

Please attach complete "show version"

Regards,

Puneesh

[Murali]

   
<HS>/stby# sh ver

Cisco Adaptive Security Appliance Software Version 8.1(2)
Device Manager Version 6.1(5)

Compiled on Thu 09-Oct-08 10:28 by builders
System image file is "disk0:/asa812-smp-k8.bin"
Config file at boot was "startup-config"

HS up 5 days 14 hours
failover cluster up 6 years 183 days

Hardware:   ASA5580-20, 8192 MB RAM, CPU AMD Opteron 2600 MHz
            2 CPUs, 4 cores
Internal ATA Compact Flash, 1024MB
BIOS Flash MX29LV320 @ 0xffc00000, 4096KB

Encryption hardware device : Cisco ASA-5580 on-board accelerator (revision 0x0)
                             Boot microcode   : CNPx-MC-BOOT-2.00
                             SSL/IKE microcode: CNPx-MC-SSLm-PLUS-2.01
                             IPSec microcode  : CNPx-MC-IPSEC-MAIN-0002
Baseboard Management Controller (revision 0x1) Firmware Version: 1.119

 0: Ext: Management0/0       : address is 0023.7d54.f87c, irq 11
 1: Ext: Management0/1       : address is 0023.7d54.f87e, irq 10
 2: Ext: GigabitEthernet3/0  : address is 0015.17c3.cc6c, irq 5
 3: Ext: GigabitEthernet3/1  : address is 0015.17c3.cc6d, irq 11
 4: Ext: GigabitEthernet3/2  : address is 0015.17c3.cc6e, irq 11
 5: Ext: GigabitEthernet3/3  : address is 0015.17c3.cc6f, irq 10
 6: Ext: GigabitEthernet4/0  : address is 0015.17c3.ccd0, irq 11
 7: Ext: GigabitEthernet4/1  : address is 0015.17c3.ccd1, irq 10
 8: Ext: GigabitEthernet4/2  : address is 0015.17c3.ccd2, irq 10
 9: Ext: GigabitEthernet4/3  : address is 0015.17c3.ccd3, irq 11
10: Ext: GigabitEthernet6/0  : address is 0015.17c8.84ac, irq 10
11: Ext: GigabitEthernet6/1  : address is 0015.17c8.84ad, irq 11
12: Ext: GigabitEthernet6/2  : address is 0015.17c8.84ae, irq 11
13: Ext: GigabitEthernet6/3  : address is 0015.17c8.84af, irq 5

Licensed features for this platform:
Maximum Physical Interfaces  : Unlimited
Maximum VLANs                : 250
Inside Hosts                 : Unlimited
Failover                     : Active/Active
VPN-DES                      : Enabled
VPN-3DES-AES                 : Enabled
Security Contexts            : 2
GTP/GPRS                     : Disabled
SSL VPN Peers                : 2
Total VPN Peers              : 10000
AnyConnect for Mobile        : Disabled
AnyConnect for Linksys phone : Disabled
Advanced Endpoint Assessment : Disabled
Licensed Cores               : 4

This platform has an ASA5580-20 VPN Premium license.

Serial Number: XXXXXXXX
Running Activation Key: XXXXXXXXXXXXX
Configuration register is 0x1
Configuration last modified by enable_15 at 11:07:57.838 UTC Thu Apr 28 2016

[Puneesh Chhabra]

Hi Uda,

I decoded the traceback and it looks like a watchdog failure.  I could not point towards any known defect.  However, there are quite a few Watchdog defects in the code you're running. 

The code (8.1.2) is already end of sw maintenance and will soon be end of support:

http://www.cisco.com/c/en/us/products/collateral/security/asa-5500-series-next-generation-firewalls/end_of_life_c51_640618.html

I would suggest this action plan:

1.  Enable the coredump on ASA to capture more detailed information if the crash occurs again.

Coredump enable filesystem <flash media>

Warning:  Enabling coredump delay the reload of the system in the event of software forced reload. So expect extra time for ASA to reload and come back on line. Exact time will depend on size of coredump.

https://supportforums.cisco.com/document/59021/enabling-coredump-asa

2.  Consider upgrading the ASA software to a newer image.

Regards,

Puneesh

[Murali]

Thanks for the review and recommendations Puneesh !

Murali.

[pathaksahil11]

Hi Experts,

We encountered an issue with ASA 5580 last month which became unresponsive. There was no internet access and it appeared the firewall was dropping all traffic. Physically, the lights seemed ok. After unsuccessful attempts to try and login to the asa, we decided to reboot the firewall, 15 minutes later everything started working!
As per TAC, nothing could be gathered from the logs.

Need to know if there is an option where ASA can failover to standby during such a scenario and can we prevent control/management plane being affected during an issue to enable an administrator to login and troubleshoot.

[Puneesh Chhabra · Accepted solution]

Hi ,

Did you check the "show crashinfo" and uptime to verify if the ASA reloaded or crashed ?

Unfortunately, there is no way for the standby to take over unless it triggers one of the following:

•The unit has a hardware failure or a power failure.

•The unit has a software failure.

•Too many monitored interfaces fail.  (Looks like in your case the hellos on the interface were sent and received properly.  So, the rest of the interface tests were not performed)

•The no failover active command is entered on the active unit or the failover active command is entered on the standby unit.

To get to the root cause of what exactly happened at that time, we may require syslogs.  

Also, please provide the software version you are running on the firewall.

Regards,

Puneesh

[pathaksahil11]

Thanks Puneesh Chhabra for the insight!

As far as I know, no crashinfo was generated and the device has been running fine since then. 

[miteshbj123]

Hi Aditya,

I had a similar problem. This article was really helpful.

Great work.

Thanks,

[Aditya Ganjoo]

Hi Mitesh,

Glad to assist :)

Regards,

Aditya

[Monica Lluis]

Thank you for your comment. When you see helpful content in the community, kindly rate it, as it helps users get to it faster and experts to keep doing a great job helping others in the community.

Monica Lluis

Community Manager

[Suman Kumar]

Hi, our cisco asa 5525 is getting rebooted, please find the attached crashinfo.

[Puneesh Chhabra]

Hi Suman,

I decoded the crashinfo and looks like you're running into a known defect here:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuy03024/?reffering_site=dumpcr

Please verify if the conditions are matching in your case and try the workaround as well:

Conditions:This crash occurs in following two scenarios:
1. When there is an FQDN object in configuration and users tries executes either "show running-config" or modifies acccess-list config while FQDN object is getting resolved.
2. When user tries to access or modify access-list config, while it is being modified from another session either from telnet/ssh/csm/asdm.

Both the above will lead to this crash as both the operations will result in dlist corruption.
Workaround:For first, the crash requires usage of FQDN ACL entries, changing from FQDN to standard IP may help avoid this crash in case for FQDN.
For second, avoid accessing and/or modifying access-list from multiple places at same time.

Also, it seems to have been fixed in 9.5(2.3)

Regards,

Puneesh

[Suman Kumar]

Hi Puneesh,

Thanks for your reply, yes we had FQDN ACLs defined we already removed it, let see if it resolve the issue,

can you also please suggest which ios version is more stable,

PFB the show version output.

Bharti-FW# sh ver

Cisco Adaptive Security Appliance Software Version 9.5(2)203
Device Manager Version 7.5(2)

Compiled on Wed 27-Jan-16 13:50 PST by builders
System image file is "disk0:/asa952-203-smp-k8.bin"
Config file at boot was "startup-config"

Bharti-FW up 1 day 23 hours
failover cluster up 12 days 0 hours

Hardware:   ASA5525, 8192 MB RAM, CPU Lynnfield 2394 MHz, 1 CPU (4 cores)
            ASA: 3518 MB RAM, 1 CPU (1 core)
Internal ATA Compact Flash, 8192MB
BIOS Flash MX25L6445E @ 0xffbb0000, 8192KB

Encryption hardware device : Cisco ASA Crypto on-board accelerator (revision 0x1)