Ask the Expert: Troubleshooting Crashes in the Adaptive Security Appliances (ASA)

[Monica Lluis]

This session will provide an opportunity to learn and ask questions about how to troubleshoot issues with the Cisco Adaptive Security Appliance (ASAs), such as crashes, high CPUs, and other common issues. To participate in this event, ask your questions below by clicking on the "reply" button. 

Ask questions from Monday May 2nd  to Friday May 13, 2016

Featured Experts

Puneesh Chhabra is a Customer Support Engineer in the Cisco High-Touch Technical Services (HTTS) team based in  Bangalore, India. He has total of 8 years of experience in network security. He has delivered multiple trainings on Cisco firewalls and VPN solutions. Prior to joining Cisco, he worked at IBM and HCL as network security consultant. Chhabra holds bachelor of Science degree in Computer Sciences from Kurukshetra University. He has achieved his CCIE certification in Security. (CCIE Security #30128)

Aditya Ganjoo is a Customer Support Engineer in the Cisco High-Touch Technical Services (HTTS) team based in  Bangalore, India.He has been working with TAC from past 5 years in Security domains like Firewall, VPN and AAA. Aditya has delivered trainings on ASA and VPN technologies.Aditya holds a Bachelor's degree in Information Technology College – M.I.E.T College of Engineering and Technology from University of Jammu.He has achieved certifications for CCNA, CCNA-Security and is currently pursuing CCIE Security

Find other  https://supportforums.cisco.com/expert-corner/events.

** Ratings Encourage Participation! **
Please be sure to rate the Answers to Questions

To ask your question, please use the reply button below.
 

[Henrik Grankvist]

Hello experts

I recently had failover in a HA-pair and it looks like the reason was that the primary firewall crashed. But the logs and show commands doesn't say much.

What would be the first step in troubleshooting the reason for this failover?

[Puneesh Chhabra]

Hi Henrik,

Can you get me the "show version" and "show crashinfo" from the primary firewall ?

Regards,

Puneesh

[Henrik Grankvist]

Show crashinfo doesn't say anything. Could it mean it didn't crash? The uptime on the primary firewall is 14 days which means it did atleast reload.

vpn-remote/pri/act#  show crashinfo 
INFO: This module has no crashinfo available.
INFO: Crashinfo file (flash:/crash.txt) not found 

------------------ show crashinfo module 1 ------------------

INFO: There is no module in slot 1 to retrieve crashinfo from.
INFO: This module has no crashinfo available.

vpn-remote/pri/act# show version

Cisco Adaptive Security Appliance Software Version 9.1(7) 
Device Manager Version 7.5(1)112

Compiled on Thu 14-Jan-16 09:37 by builders
System image file is "disk0:/asa917-k8.bin"
Config file at boot was "startup-config"

vpn-remote up 14 days 12 hours
failover cluster up 3 years 325 days

Hardware:   ASA5540, 2048 MB RAM, CPU Pentium 4 2000 MHz,
Internal ATA Compact Flash, 256MB
BIOS Flash M50FW080 @ 0xfff00000, 1024KB

Encryption hardware device : Cisco ASA-55xx on-board accelerator (revision 0x0)
                             Boot microcode        : CN1000-MC-BOOT-2.00 
                             SSL/IKE microcode     : CNlite-MC-SSLm-PLUS-2.08
                             IPSec microcode       : CNlite-MC-IPSECm-MAIN-2.09
                             Number of accelerators: 1

 0: Ext: GigabitEthernet0/0  : address is 0017.5a88.785e, irq 9
 1: Ext: GigabitEthernet0/1  : address is 0017.5a88.785f, irq 9
 2: Ext: GigabitEthernet0/2  : address is 0017.5a88.7860, irq 9
 3: Ext: GigabitEthernet0/3  : address is 0017.5a88.7861, irq 9
 4: Ext: Management0/0       : address is 0017.5a88.7862, irq 11
 5: Int: Not used            : irq 11
 6: Int: Not used            : irq 5

Licensed features for this platform:
Maximum Physical Interfaces       : Unlimited      perpetual
Maximum VLANs                     : 200            perpetual
Inside Hosts                      : Unlimited      perpetual
Failover                          : Active/Active  perpetual
Encryption-DES                    : Enabled        perpetual
Encryption-3DES-AES               : Enabled        perpetual
Security Contexts                 : 2              perpetual
GTP/GPRS                          : Disabled       perpetual
AnyConnect Premium Peers          : 2              perpetual
AnyConnect Essentials             : 2500           perpetual
Other VPN Peers                   : 5000           perpetual
Total VPN Peers                   : 5000           perpetual
Shared License                    : Disabled       perpetual
AnyConnect for Mobile             : Enabled        perpetual
AnyConnect for Cisco VPN Phone    : Disabled       perpetual
Advanced Endpoint Assessment      : Disabled       perpetual
UC Phone Proxy Sessions           : 2              perpetual
Total UC Proxy Sessions           : 2              perpetual
Botnet Traffic Filter             : Disabled       perpetual
Intercompany Media Engine         : Disabled       perpetual
Cluster                           : Disabled       perpetual

This platform has an ASA 5540 VPN Premium license.


Failover cluster licensed features for this platform:
Maximum Physical Interfaces       : Unlimited      perpetual
Maximum VLANs                     : 200            perpetual
Inside Hosts                      : Unlimited      perpetual
Failover                          : Active/Active  perpetual
Encryption-DES                    : Enabled        perpetual
Encryption-3DES-AES               : Enabled        perpetual
Security Contexts                 : 4              perpetual
GTP/GPRS                          : Disabled       perpetual
AnyConnect Premium Peers          : 4              perpetual
AnyConnect Essentials             : 2500           perpetual
Other VPN Peers                   : 5000           perpetual
Total VPN Peers                   : 5000           perpetual
Shared License                    : Disabled       perpetual
AnyConnect for Mobile             : Enabled        perpetual
AnyConnect for Cisco VPN Phone    : Disabled       perpetual
Advanced Endpoint Assessment      : Disabled       perpetual
UC Phone Proxy Sessions           : 4              perpetual
Total UC Proxy Sessions           : 4              perpetual
Botnet Traffic Filter             : Disabled       perpetual
Intercompany Media Engine         : Disabled       perpetual
Cluster                           : Disabled       perpetual

[Puneesh Chhabra]

Yes, it doesn't look like it crashed.  Did you check syslogs at the time of reload ?

Could it be a power glitch ?

Regards,

Puneesh

[Henrik Grankvist]

I included the syslog in my first post, but it doesn't say much. Maybe I had the logging level set too high to show the reason.

Could be the power, but it gets its power through a UPS.

What logging settings would you recommend so that in the future see more information if this scenario happened again?

[Aditya Ganjoo]

Hi Henrik,

If this occurs in near future you can do the following things:

1) Enable the coredump on all ASA. The command for same is

Coredump enable filesystem <flash media>

///Please note , the ASA 5585 creates around 1 GB of file size on flash, in case you don't have that much of space on internal ASA flash, you can also use Cisco certified USB flash drives, in case they are available on all ASA unit.

*enabling coredump delay the reload of the system in the event of software forced reload. So expect extra time for ASA to reload and come back on line. Exact time will depend on size of coredump.

https://supportforums.cisco.com/document/59021/enabling-coredump-asa

2) Configure "informational" level syslog on ASA .

https://supportforums.cisco.com/document/73511/how-enable-syslogs-asa

Regards,

Aditya

[gunjan.k1]

Hi Aditya,

Thank you for sharing this information. This was very helpful. Keep up doing great job.

Thanks,

Gunjan

[Aditya Ganjoo]

Glad to assist :)

Regards,

Aditya

[Mohammed Zia]

Hi Experts,

I have ASA 5520 which is rebooting every hour 4 times a day.

Im not sure if there is an Issue with Software or hardware.

your Inputs are Highly Appreciated

 I have attached show tech output.

-regards

Ayaz Basha

[Aditya Ganjoo]

Hi Ayaz,

Hope you are doing fine.

Checking the traces for the crash , it seems related to the ASA device HW:

CTM ERROR: ASA hardware accelerator init failed, cause: boot_init completion timeout

It is a panic crash related to CTM thread so I would suggest you to replace the device.

Also since you are running 8.4.3 code with ikev1/ikev2 VPN on the ASA  i would recommend you to upgrade to the fixed versions as per the :

Cisco ASA IKEv1 and IKEv2 Buffer Overflow Vulnerability CSCux29978

https://tools.cisco.com/bugsearch/bug/CSCux29978/?reffering_site=dumpcr

Known Fixed Releases:

8.2(5.59)

8.4(7.30)

8.7(1.18)

9.0(4.38)

9.1(6.11)

9.1(7)

9.2(4.5)

9.3(3.7)

9.4(2.4)

9.4(2.99)

9.5(2.2)

Regards,

Aditya

Please rate helpful posts and mark correct answers.

[Aditya Ganjoo]

Hi Henrik,

We need to check if there is any crash file generated on the primary ASA.

Use the command show crashinfo on the ASA to check that.

Also as per the show failover history I do not see any reason for failover except "Set by the config command" which means we have manually triggered the failover on the ASA.

Regards,

Aditya

[Subin Kallidumpil]

Hi Team,

I have a query around one of the deployment of ASA 5585.

Does access-list count have an affect on the ASA memory ?

I have a pretty high number of ACL on my ASA, are there any workarounds to optimize high memory ?

Regards

K Subin

[Aditya Ganjoo · Accepted solution]

Hi Subin,

Yes ACE count on the ASA has a significant affect on the memory.

Use the command show access-list | in elements to check the ACE count.

As per Cisco on ASA 5585 here is the number of ACE's supported:

1. SSP-10: 500K
2. SSP-20: 750K
3. SSP-40: 1M
4. SSP-60: 2M

ACL count increase is a result of fully expanding original object-group ACEs to individual ACEs.

The magnitude of the explosion of an object-group ACE is determined by

1. The number of object groups that are present in the ACE, and
2. The number of objects (elements) in the object groups

The actual number of ACEs after expansion is obtained by multiplying the above two factors. For example, if an object-group ACE consists of a source-IP object group of 1000 elements, a destination-IP object group of 100 elements, and a service object group of 10 elements, then the object-group ACE is expanded to 1 million ACEs. The ACEs are then compiled into a lookup table for layer-3/4 packet classification.

You can use the command object-group-search access-control for reducing the ACE count on the ASA post 8.2 code.

Regards,

Aditya

[Andreas Heykaus]

Hi Aditya,

is there any overview about the no. of ACE's supported by ASA model?

Thanks

Andreas