Re: ASA 5520 with overruns and underruns but low CPU

graham.fleming · ‎03-23-2012

ASA has 3 interfaces: outside, inside, DMZ

Average total throughput is between 20-200mbps. Majority of throughput would be between inside and DMZ interfaces. CPU never goes above 70% (and doesn't seem to spike much with respect to traffic patterns).

We are seeing lots of errors on the interface. Inside output:

Interface GigabitEthernet0/1 "Inside", is up, line protocol is up

Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)

Input flow control is unsupported, output flow control is unsupported

Description: Inside LAN interface

9097466566 packets input, 6650618441463 bytes, 0 no buffer

Received 449 broadcasts, 0 runts, 0 giants

75942 input errors, 0 CRC, 0 frame, 75942 overrun, 0 ignored, 0 abort

0 L2 decode drops

9251999409 packets output, 3811434498126 bytes, 85579 underruns

0 pause output, 0 resume output

0 output errors, 0 collisions, 0 interface resets

0 late collisions, 0 deferred

0 input reset drops, 0 output reset drops, 0 tx hangs

input queue (blocks free curr/low): hardware (255/230)

output queue (blocks free curr/low): hardware (255/0)

The ASA is a VPN headend for 3 remote branches. Approximately 100 VOIP phones between the sites register to a phone server at the main site (where 5520 is).

I'm baffled by these overruns and underruns.

Code version 8.2(2).

I refuse to believe that I need to upgrade the firewall when a 5520 can supposedly process 450mbps (aware this is best case but our average is close to 80-100mbps) and should do 300,000 bps wheras on average we are pushing 10-15,000.

Is there anything we can do to find out why the ASA is having such a hard time processing all of the packets we are sending it?

Thanks guys!

patrick.preuss · ‎03-24-2012

Hi

you can try follwing. "show asp drop"

command refercence:

http://www.cisco.com/en/US/docs/security/asa/asa82/command/reference/s2.html#wp1391007

PIX/ASA: Monitor and Troubleshoot Performance Issues

http://www.cisco.com/en/US/products/hw/vpndevc/ps2030/products_tech_note09186a008009491c.shtml

HTH

Patrick

Message was edited by: Patrick Marc Preuss

graham.fleming · ‎03-26-2012

Thanks for the reply. Output is shown below. This morning we had some terrible voice quality between HO and a branch site and there was no related CPU or interface bps spike during this time. Although we did see ASA inside interface errors/drops at the time. So not sure why this is happening.

Frame drop:

Invalid encapsulation (invalid-encap) 1953077

Invalid IP length (invalid-ip-length) 1256

Invalid TCP Length (invalid-tcp-hdr-length) 5

Invalid UDP Length (invalid-udp-length) 4

No valid adjacency (no-adjacency) 35369

Flow is denied by configured rule (acl-drop) 18776203

Invalid SPI (np-sp-invalid-spi) 6188

NAT-T keepalive message (natt-keepalive) 852142

First TCP packet not SYN (tcp-not-syn) 2572558

Bad TCP flags (bad-tcp-flags) 5752

TCP Dual open denied (tcp-dual-open) 36688

TCP data send after FIN (tcp-data-past-fin) 476

TCP failed 3 way handshake (tcp-3whs-failed) 2184310

TCP RST/FIN out of order (tcp-rstfin-ooo) 2587928

TCP SEQ in SYN/SYNACK invalid (tcp-seq-syn-diff) 2280

TCP SYNACK on established conn (tcp-synack-ooo) 196

TCP packet SEQ past window (tcp-seq-past-win) 61410

TCP invalid ACK (tcp-invalid-ack) 2626

TCP replicated flow pak drop (tcp-fo-drop) 43

TCP ACK in 3 way handshake invalid (tcp-discarded-ooo) 3

TCP Out-of-Order packet buffer full (tcp-buffer-full) 152769

TCP global Out-of-Order packet buffer full (tcp-global-buffer-full) 3

TCP Out-of-Order packet buffer timeout (tcp-buffer-timeout) 68013

TCP RST/SYN in window (tcp-rst-syn-in-win) 23879

TCP dup of packet in Out-of-Order queue (tcp-dup-in-queue) 432750

TCP packet failed PAWS test (tcp-paws-fail) 391464

CTM returned error (ctm-error) 2

IPSEC tunnel is down (ipsec-tun-down) 3799

Slowpath security checks failed (sp-security-failed) 55047

Expired flow (flow-expired) 1

ICMP Inspect seq num not matched (inspect-icmp-seq-num-not-matched) 19

ICMP Error Inspect no existing conn (inspect-icmp-error-no-existing-conn) 239

DNS Inspect invalid domain label (inspect-dns-invalid-domain-label) 565

DNS Inspect packet too long (inspect-dns-pak-too-long) 366

DNS Inspect id not matched (inspect-dns-id-not-matched) 1248

FP L2 rule drop (l2_acl) 6460539

Interface is down (interface-down) 26

Dropped pending packets in a closed socket (np-socket-closed) 4639

Last clearing: Never

Flow drop:

NAT failed (nat-failed) 241040

NAT reverse path failed (nat-rpf-failed) 5820

Tunnel being brought up or torn down (tunnel-pending) 30

Need to start IKE negotiation (need-ike) 63014

Inspection failure (inspect-fail) 33182

SSL handshake failed (ssl-handshake-failed) 3

SSL malloc error (ssl-malloc-error) 2

SSL received close alert (ssl-received-close-alert) 12

IPSec inner policy mismatch failure (ipsec-selector-failure) 8

patrick.preuss · ‎03-26-2012

Hi Graham

is your installation new ?

what has changed rechently?

Have you configured qos?

What is the Traffic Pattern over this time?

What inspection is configured?

What say the log?

Can you clear the counters for this and place than again..

I see you have some drops due to tcp-buffers ...

HTH

patrick.preuss · ‎03-26-2012

Hi graham

have you tried the output interpreter

Interface Inside - GigabitEthernet0/1 (up/up)
WARNING: There have been 85579 'underruns' reported.
This indicates the number of times that the transmitter has been running faster
than the ASA/PIX can handle.
TRY THIS: Monitor the level of underruns over time. If they continue increasing,
consider upgrading hardware. Also confirm that there is not a bad client network
card or a virus on the segment causing traffic bursts. Use the show local-host
ASA/PIX command to see if one IP is creating too many connections. Then, try
the same command using the suspected ip to check the host’s activity.
REFERENCE: For more information, see Troubleshooting Ethernet.
WARNING: There have been 75942 'overruns' reported.
This shows the number of times that the receiver hardware was incapable of handling
received data to a hardware buffer because the input rate exceeded the receiver's
capability to handle the data. If the overruns are equal to input errors and
there are no CRC errors then at one point the ASA/PIX received packets faster
than it can handle. This is not a cause of concern and can be ignored.
TRY THIS: Verify that speed and duplex settings are hard-coded on the ASA/PIX
and on the other directly connected devices. Use show blocks ASA/PIX command.
A zero in the LOW column indicates a previous event where memory exhausted. A
zero in the CNT column means memory is exhausted now. If the memory is continuously
exhausted and traffic is not moving, then consider upgrading the interface to
Gigabit or the ASA/PIX to a higher model. If this is DMZ interface, you can use
other unused interfaces by splitting your current DMZ into 2 networks. If very
large object-groups or large access-lists are used on ASA/PIX then use object-group-search
keyword in the access-list ASA/PIX command to specify that access-list search
is performed on object groups that are contained in access-list instead of searching
the entire expanded access-list.
REFERENCE: For more information, see Troubleshooting Ethernet.

NOTE: Output Interpreter will perform analysis of ASA/PIX interface statistics
for all interfaces that are not administratively shutdown. Ensure that interface
statistics reflect the current state of the interfaces (<24hrs) by periodically clearing the
counters with the 'clear interface' command.

patrick.preuss · ‎03-26-2012

Hi Graham

is it possible for you to split the interfaces for inside?

HTH