Cisco ASA and lots of connections per second - cpu usage

WebOps eDreams · ‎05-05-2015

Hi!

We have an asa5585-x which Works with no problems; but sometimes we can see huge amounts of connections more tan 10k-20k that triggers high cpu usage.

Do you know how can we detect the source of this connections?

Added youll see asdm with the connections and cpu usage.

thanks!

Vibhor Amrodia · ‎05-05-2015

Hi,

I think this is a Cluster setup and we do see an increase in the connections at that time.

Have you been able to get some more information on the connections that are seen during this time.

Some of the commands that would help in troubleshooting this issue would be:-

1) cluster exec show perfmon

2) cluster exec show conn count

3) cluster exec show asp drop

You should collect multiple outputs from the Master and cluster exec would print the output from the slave also on the master unit.

Thanks and Regards,

Vibhor Amrodia

WebOps eDreams · ‎05-05-2015

hi vibhor,

thanks for your reply and yes this is a cluster; actually a cisco 5585-X with sourcefire blade. i executed the commands you said. The problem is that i cannot locate the source of this connections; this is the output from the commands:

bcn1-fw-asa3# cluster exec show perfmon
ASA3A(LOCAL):*********************************************************

PERFMON STATS:                     Current      Average
Xlates                               64/s          8/s
Connections                       24262/s         13/s
TCP Conns                            55/s         11/s
UDP Conns                            49/s         10/s
URL Access                            0/s          0/s
URL Server Req                        0/s          0/s
TCP Fixup                             0/s          0/s
TCP Intercept Established Conns       0/s          0/s
TCP Intercept Attempts                0/s          0/s
TCP Embryonic Conns Timeout           1/s          1/s
HTTP Fixup                            0/s          0/s
FTP Fixup                             0/s          0/s
AAA Authen                            0/s          0/s
AAA Author                            0/s          0/s
AAA Account                           0/s          0/s

VALID CONNS RATE in TCP INTERCEPT: Current Average
N/A 114.29%

ASA3B:****************************************************************

PERFMON STATS:                     Current      Average
Xlates                                0/s          0/s
Connections                         167/s         24/s
TCP Conns                            11/s          9/s
UDP Conns                            39/s         25/s
URL Access                            0/s          0/s
URL Server Req                        0/s          0/s
TCP Fixup                             0/s          0/s
TCP Intercept Established Conns       0/s          0/s
TCP Intercept Attempts                0/s          0/s
TCP Embryonic Conns Timeout           0/s          0/s
HTTP Fixup                            0/s          0/s
FTP Fixup                             0/s          0/s
AAA Authen                            0/s          0/s
AAA Author                            0/s          0/s
AAA Account                           0/s          0/s

VALID CONNS RATE in TCP INTERCEPT: Current Average
N/A 0.00%
bcn1-fw-asa3# cluster exec show conn count
ASA3A(LOCAL):*********************************************************
24260 in use, 31367 most used
Cluster stub connections: 7276 in use, 3118042 most used

ASA3B:****************************************************************
7305 in use, 11317 most used
Cluster stub connections: 24117 in use, 1486057 most used
bcn1-fw-asa3#
bcn1-fw-asa3#
bcn1-fw-asa3# cluster exec show asp drop
ASA3A(LOCAL):*********************************************************

Frame drop:
Invalid encapsulation (invalid-encap) 13097

No valid adjacency (no-adjacency) 18

Unexpected packet (unexpected-packet) 192

No route to host (no-route) 6714

Reverse-path verify failed (rpf-violated) 1560

Flow is denied by configured rule (acl-drop) 2707737

First TCP packet not SYN (tcp-not-syn) 472390

Bad TCP checksum (bad-tcp-cksum) 31

Bad TCP flags (bad-tcp-flags) 1907

TCP data send after FIN (tcp-data-past-fin) 244

TCP failed 3 way handshake (tcp-3whs-failed) 18412

TCP RST/FIN out of order (tcp-rstfin-ooo) 259948

TCP SEQ in SYN/SYNACK invalid (tcp-seq-syn-diff) 17965

TCP SYNACK on established conn (tcp-synack-ooo) 1223

TCP packet SEQ past window (tcp-seq-past-win) 6561

TCP invalid ACK (tcp-invalid-ack) 2973

TCP ACK in 3 way handshake invalid (tcp-discarded-ooo) 33

TCP RST/SYN in window (tcp-rst-syn-in-win) 4800

TCP packet failed PAWS test (tcp-paws-fail) 449

Connection limit reached (conn-limit) 85

CTM returned error (ctm-error) 2593

Slowpath security checks failed (sp-security-failed) 2689854
Expired flow (flow-expired) 18539

ICMP Error Inspect no existing conn (inspect-icmp-error-no-existing-conn)
8
SFR Module requested drop (sfr-request) 3180394

FP L2 rule drop (l2_acl) 290386

Unable to obtain connection lock (connection-lock) 2790

Interface is down (interface-down) 1706488

Dropped pending packets in a closed socket (np-socket-closed) 18

Cluster packet rcvd over CCL, unit has stub flow and unknown role (cluster-ccl
-unknown-stub) 61
NAT invalid cluster input (nat-cluster-input) 4155

Layer 3 protocol of the packet is not IP (cluster-non-ip-pkt) 86337

Last clearing: Never

Flow drop:
Inspection failure (inspect-fail) 1560

SSL bad record detected (ssl-bad-record-detect) 6

SSL handshake failed (ssl-handshake-failed) 5232

Flow removed, packet sent to owner (cluster-redirect) 2982803310

Last clearing: Never

ASA3B:****************************************************************

Frame drop:
Invalid encapsulation (invalid-encap) 13139

Invalid TCP Length (invalid-tcp-hdr-length) 1

No valid adjacency (no-adjacency) 28

No route to host (no-route) 35378

Reverse-path verify failed (rpf-violated) 7444

Flow is denied by configured rule (acl-drop) 1185136

First TCP packet not SYN (tcp-not-syn) 185283

Bad TCP flags (bad-tcp-flags) 178

TCP data send after FIN (tcp-data-past-fin) 53

TCP failed 3 way handshake (tcp-3whs-failed) 1149

TCP RST/FIN out of order (tcp-rstfin-ooo) 18512

TCP SEQ in SYN/SYNACK invalid (tcp-seq-syn-diff) 1

TCP packet SEQ past window (tcp-seq-past-win) 388

TCP invalid ACK (tcp-invalid-ack) 8

TCP RST/SYN in window (tcp-rst-syn-in-win) 47

Slowpath security checks failed (sp-security-failed) 1405118

Expired flow (flow-expired) 5043

SFR Module requested drop (sfr-request) 893746

FP L2 rule drop (l2_acl) 1826974

Unable to obtain connection lock (connection-lock) 73

Interface is down (interface-down) 1592141

Cluster packet rcvd over CCL on backup (cluster-ccl-backup) 39497

Layer 3 protocol of the packet is not IP (cluster-non-ip-pkt) 86339

Last clearing: Never

Flow drop:
Inspection failure (inspect-fail) 1270

Flow removed, packet sent to owner (cluster-redirect) 56953870

Last clearing: Never
bcn1-fw-asa3#

isaacalves27 · ‎09-28-2015

Hello Vibhor

I am facing a similar issue on our 5585-X. But I think that everyting is as expected.

A cluster with two ASA 5585-X SSP20 can behandle a max of 170.000 CPS - is that correct understood?- 0.7*(125.000+125.000).

Best regards

Robert Rowland III · ‎05-19-2015

1) setup packet capture with rotating capture files and stop capture during an event and analyze ?

with Wireshark you can filter on "SYN"s to see connection attempts

2) does your firewall connect to any routers that can do "top talkers" ?, turn if on, you may have a small number of devices getting busy.

3) using Ciscocmd check on the xlates on a regular basis and see what is increasing - may have to use Excel and/or some command line sorting

these should help you see what is going on.