Re: 4 Identical set up FTD....Snort Problems

frazreid2 · ‎02-20-2024

Hi,

I have 4 FTD's and on 2 of them snort is getting the CPU load to 60%+ on the other 2 the CPU including Snort ist less than 5% (this was the aim of the new devices) All devices have the same basic configuration but of course a different rulebase.

I have tried a few commands to try to find out the root cause of the SNORT high CPU usage but nothing found.

Commands and results are

> show kernel cgroup-controller cpuset | begin lina

group "restricted/lina"

cpuset.cpus: 2-5,14-17

cpuset.mems: 0

tasks:

13738 13860 13918 13919

13920 13921 14061 14090

14091

group "restricted/qemu"

cpuset.cpus: 6-11,18-23

cpuset.mems: 0

tasks:

> show memory

Free memory: 41601053766 bytes (82%)

Used memory: 9390567968 bytes (18%)

------------- ------------------

Total memory: 50991621734 bytes (100%)

Note: Free memory is the free system memory. Additional memory may

be available from memory pools internal to the firewall process.

Use 'show memory detail' to see this information, but use it

with care since it may cause CPU hogs and packet loss under load.

> show blocks

SIZE MAX LOW CNT FAILED

0 2700 2667 2700 0

4 100 99 99 0

80 6147 5687 6147 0

256 9248 8885 9143 0

1550 6254 5658 6251 0

2048 100 98 100 0

2560 164 162 164 0

4096 100 89 100 0

8192 100 100 100 0

9344 25000 24999 25000 0

16384 100 100 100 0

65664 16 16 16 0

top - 12:20:30 up 248 days, 3:06, 2 users, load average: 15.15, 14.42, 14.53

Tasks: 355 total, 1 running, 354 sleeping, 0 stopped, 0 zombie

%Cpu(s): 56.7 us, 2.3 sy, 0.1 ni, 40.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st

MiB Mem : 95235.8 total, 70051.2 free, 18060.6 used, 7123.9 buff/cache

MiB Swap: 6979.9 total, 6979.4 free, 0.5 used. 74540.2 avail Mem

Unknown command - try 'h' for help

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

13920 root 0 -20 11.8g 874724 341628 S 800.0 0.9 47639,13 lina

21443 root 1 -19 9966.1m 2.3g 253008 S 616.9 2.5 684056:57 snort3

13712 root 25 5 427972 6028 4564 S 7.3 0.0 25087:03 loggerd

14150 root 25 5 628696 89232 8128 S 2.7 0.1 2346:39 EventHandler

13719 root 20 0 407600 18196 4948 S 1.0 0.0 910:05.10 sftunnel

3273 root 20 0 3068560 132712 49964 S 0.7 0.1 203:36.02 SFDataCorrelato

4167 admin 20 0 4080 2660 1996 R 0.7 0.0 0:00.15 top

12 root 20 0 0 0 0 I 0.3 0.0 131:17.93 kworker/0:1-events

13730 root 20 0 1654600 55660 33460 S 0.3 0.1 752:10.93 adi

13731 sfsnort 20 0 142720 4392 2604 S 0.3 0.0 23:23.33 bltd

13733 root 1 -19 286944 5140 4204 S 0.3 0.0 280:14.59 ndmain.bin

14469 root 20 0 6145128 426616 15892 S 0.3 0.4 652:27.59 java

14706 root 0 -20 2540 1844 1648 S 0.3 0.0 45:56.32 sfifd

15077 ntp 20 0 74376 2852 2180 S 0.3 0.0 16:04.91 ntpd

21336 root 20 0 1155944 4516 2664 S 0.3 0.0 131:29.82 diskmanager

1 root 20 0 2460 1888 1764 S 0.0 0.0 1:54.35 init

Then on a "healthy" FTD....

> show kernel cgroup-controller cpuset | begin lina

group "restricted/lina"

cpuset.cpus: 2-5,14-17

cpuset.mems: 0

tasks:

20050 20226 20271 20272

20273 20274 20411 20457

20458

group "restricted/qemu"

cpuset.cpus: 6-11,18-23

cpuset.mems: 0

tasks:

> show memory

Free memory: 40895792076 bytes (81%)

Used memory: 9800550656 bytes (19%)

------------- ------------------

Total memory: 50696342732 bytes (100%)

Note: Free memory is the free system memory. Additional memory may

be available from memory pools internal to the firewall process.

Use 'show memory detail' to see this information, but use it

with care since it may cause CPU hogs and packet loss under load.

> show blocks

SIZE MAX LOW CNT FAILED

0 2700 2667 2700 0

4 100 99 99 0

80 6000 5932 6000 0

256 9248 8324 9143 0

1550 6454 6265 6450 0

2048 100 98 100 0

2560 164 162 164 0

4096 100 98 100 0

8192 100 100 100 0

9344 25000 25000 25000 0

16384 120 120 120 0

65664 16 16 16 0

top - 12:22:53 up 189 days, 26 min, 2 users, load average: 8.93, 8.99, 9.10

Tasks: 356 total, 1 running, 353 sleeping, 0 stopped, 2 zombie

%Cpu(s): 34.3 us, 1.0 sy, 0.2 ni, 64.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st

MiB Mem : 95235.8 total, 68617.6 free, 18852.0 used, 7766.2 buff/cache

MiB Swap: 6968.4 total, 6968.2 free, 0.2 used. 73886.1 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

20273 root 0 -20 11.8g 874580 341484 S 800.0 0.9 36289,46 lina

20989 root 1 -19 13.2g 3.2g 254832 S 43.2 3.4 60691:16 snort3

20022 root 25 5 428000 5708 4540 S 7.3 0.0 18632:18 loggerd

20578 root 25 5 694240 136956 9464 S 2.0 0.1 2020:27 EventHandler

20573 root 20 0 193520 185816 9644 S 1.3 0.2 170:55.10 perl5.24.4

1859 root 20 0 3068560 169628 49884 S 0.7 0.2 185:14.37 SFDataCorrelato

20023 mysql 20 0 7203900 382668 21456 S 0.7 0.4 499:05.54 mariadbd

5163 root 20 0 4856 4032 2764 S 0.3 0.0 210:39.21 ssp_heimdall_in

5502 root 20 0 0 0 0 Z 0.3 0.0 0:00.01 ps

5893 root 20 0 84444 7856 3912 S 0.3 0.0 140:28.44 rng_tools_start

5937 root 20 0 4512 2412 1484 S 0.3 0.0 436:32.01 1block_process.

6051 root 20 0 11336 8456 3784 S 0.3 0.0 182:35.68 chm_daemon_star

20042 root 20 0 1653316 56988 41160 S 0.3 0.1 551:01.76 adi

20044 root 20 0 234452 15932 15284 S 0.3 0.0 254:17.68 pdts_proc

20045 root 1 -19 286944 4972 4032 S 0.3 0.0 204:58.13 ndmain.bin

21120 root 20 0 6145128 333568 16160 S 0.3 0.3 436:18.79 java

31073 root 20 0 407592 14992 4976 S 0.3 0.0 648:34.68 sftunnel

1 root 20 0 2460 1812 1692 S 0.0 0.0 1:28.49 init

Also done

"Show asp inspect-dp snort" and on the low CPU it shows snort is around 2% but on the other FTD all threads at above 60%.

Where to go from here ?

MHM Cisco World · ‎02-20-2024

All have same ver. ?

frazreid2 · ‎02-20-2024

Yes - all4 and FMC are on 7.2.4

MHM Cisco World · ‎02-20-2024

> show asp drop
share this for both FTD low and high cpu
thanks

MHM

frazreid2 · ‎02-20-2024

High

Frame drop:

Invalid IP length (invalid-ip-length) 494397

Invalid TCP Length (invalid-tcp-hdr-length) 7

No valid adjacency (no-adjacency) 315

No valid V4 adjacency. Check ARP table (show arp) has entry for nexthop. (no-v4-adjacency) 6

No route to host (no-route) 3979

Flow is denied by configured rule (acl-drop) 39458250

First TCP packet not SYN (tcp-not-syn) 44681743

TCP data send after FIN (tcp-data-past-fin) 1

TCP failed 3 way handshake (tcp-3whs-failed) 3703361

TCP RST/FIN out of order (tcp-rstfin-ooo) 37020260

TCP SEQ in SYN/SYNACK invalid (tcp-seq-syn-diff) 25778

TCP SYN on established conn (tcp-syn-ooo) 49184

TCP SYNACK on established conn (tcp-synack-ooo) 9

TCP packet SEQ past window (tcp-seq-past-win) 62201

TCP invalid ACK (tcp-invalid-ack) 325

TCP replicated flow pak drop (tcp-fo-drop) 28844

TCP RST/SYN in window (tcp-rst-syn-in-win) 4802

TCP packet failed PAWS test (tcp-paws-fail) 2336

Slowpath security checks failed (sp-security-failed) 7335860

Dropped by standby unit (fo-standby) 496

Expired flow (flow-expired) 17627

ICMP Inspect bad icmp code (inspect-icmp-bad-code) 24

ICMP Inspect seq num not matched (inspect-icmp-seq-num-not-matched) 27801

ICMP Error Inspect no existing conn (inspect-icmp-error-no-existing-conn) 825

DNS Inspect id not matched (inspect-dns-id-not-matched) 19552

Snort instance is busy (snort-busy) 186

FP L2 rule drop (l2_acl) 26187

Unable to obtain connection lock (connection-lock) 1224

Interface is down (interface-down) 6437816

Async lock queue limit exceeded (async-lock-queue-limit) 235

Connection to PAT address without pre-existing xlate (nat-no-xlate-to-pat-pool) 172

Received a multicast packet in the non-active device (mcast-in-nonactive-device) 179570

Not a blocking packet (none) 188

Fragment reassembly failed (fragment-reassembly-failed) 109

Packet is blocked as requested by snort (snort-block) 495800

Dispatch queue tail drops (dispatch-queue-limit) 313660

Last clearing: Never

Flow drop:

Flow is denied by access rule (acl-drop) 730968

NAT reverse path failed (nat-rpf-failed) 32

Inspection failure (inspect-fail) 28476

Last clearing: Never

Low

Frame drop:

Expired VPN context (vpn-context-expired) 3

Invalid IP length (invalid-ip-length) 579470

Invalid TCP Length (invalid-tcp-hdr-length) 9629

Invalid UDP Length (invalid-udp-length) 220

No valid adjacency (no-adjacency) 463789

No valid V4 adjacency. Check ARP table (show arp) has entry for nexthop. (no-v4-adjacency) 2

No route to host (no-route) 288465

Flow is denied by configured rule (acl-drop) 679513246

Invalid SPI (np-sp-invalid-spi) 6

First TCP packet not SYN (tcp-not-syn) 32684759

Bad TCP flags (bad-tcp-flags) 15

TCP data send after FIN (tcp-data-past-fin) 20

TCP failed 3 way handshake (tcp-3whs-failed) 3919441

TCP RST/FIN out of order (tcp-rstfin-ooo) 64608391

TCP SEQ in SYN/SYNACK invalid (tcp-seq-syn-diff) 768587

TCP SYN on established conn (tcp-syn-ooo) 37727

TCP SYNACK on established conn (tcp-synack-ooo) 716

TCP packet SEQ past window (tcp-seq-past-win) 68652

TCP invalid ACK (tcp-invalid-ack) 8088

TCP replicated flow pak drop (tcp-fo-drop) 796

TCP Out-of-Order packet buffer timeout (tcp-buffer-timeout) 22

TCP RST/SYN in window (tcp-rst-syn-in-win) 30488

TCP dup of packet in Out-of-Order queue (tcp-dup-in-queue) 83

TCP packet failed PAWS test (tcp-paws-fail) 18284

Slowpath security checks failed (sp-security-failed) 470953

Dropped by standby unit (fo-standby) 519

Expired flow (flow-expired) 5

ICMP Inspect bad icmp code (inspect-icmp-bad-code) 1192

ICMP Inspect seq num not matched (inspect-icmp-seq-num-not-matched) 5309

ICMP Error Inspect no existing conn (inspect-icmp-error-no-existing-conn) 2456

DNS Inspect invalid packet (inspect-dns-invalid-pak) 151

DNS Inspect invalid domain label (inspect-dns-invalid-domain-label) 8911

DNS Inspect packet too long (inspect-dns-pak-too-long) 6745

DNS Inspect id not matched (inspect-dns-id-not-matched) 713427

Snort instance is busy (snort-busy) 54993

FP L2 rule drop (l2_acl) 11453328

Unable to obtain connection lock (connection-lock) 1

Interface is down (interface-down) 7606957

IKE new SA limit exceeded (ike-sa-rate-limit) 12

NAT failed (nat-xlate-failed) 82

NAT failed due to pool exhaustion (nat-xlate-pool-exhausted) 70158

Connection to PAT address without pre-existing xlate (nat-no-xlate-to-pat-pool) 190687158

Received a multicast packet in the non-active device (mcast-in-nonactive-device) 18910

Blocked or blacklisted by the firewall preprocessor (firewall) 54272

Blocked or blacklisted by the session preprocessor (session-preproc) 1489

Fragment reassembly failed (fragment-reassembly-failed) 176

Packet is blacklisted by snort (snort-blacklist) 1

Packet is blocked as requested by snort (snort-block) 1577175

Failover link is not ready for processing NLP packets (ha-nlp-lu-link-not-ready) 8

Dispatch queue tail drops (dispatch-queue-limit) 413953

Last clearing: Never

Flow drop:

Need to start IKE negotiation (need-ike) 5416

VPN decryption missing (vpn-missing-decrypt) 530

Flow is denied by access rule (acl-drop) 3382

NAT reverse path failed (nat-rpf-failed) 1044

Inspection failure (inspect-fail) 3833968

Last clearing: Never

MHM Cisco World · ‎02-20-2024

are you sure the order
the first output for high and second for low
can you make double check

MHM

frazreid2 · ‎02-21-2024

done it again.....

High

Frame drop:

Invalid IP length (invalid-ip-length) 494397

Invalid TCP Length (invalid-tcp-hdr-length) 7

No valid adjacency (no-adjacency) 315

No valid V4 adjacency. Check ARP table (show arp) has entry for nexthop. (no-v4-adjacency) 6

No route to host (no-route) 3979

Flow is denied by configured rule (acl-drop) 39659127

First TCP packet not SYN (tcp-not-syn) 45059751

TCP data send after FIN (tcp-data-past-fin) 1

TCP failed 3 way handshake (tcp-3whs-failed) 3754348

TCP RST/FIN out of order (tcp-rstfin-ooo) 37170115

TCP SEQ in SYN/SYNACK invalid (tcp-seq-syn-diff) 25890

TCP SYN on established conn (tcp-syn-ooo) 49205

TCP SYNACK on established conn (tcp-synack-ooo) 9

TCP packet SEQ past window (tcp-seq-past-win) 62201

TCP invalid ACK (tcp-invalid-ack) 325

TCP replicated flow pak drop (tcp-fo-drop) 28844

TCP RST/SYN in window (tcp-rst-syn-in-win) 4828

TCP packet failed PAWS test (tcp-paws-fail) 2336

Slowpath security checks failed (sp-security-failed) 7374940

Dropped by standby unit (fo-standby) 496

Expired flow (flow-expired) 17750

ICMP Inspect bad icmp code (inspect-icmp-bad-code) 24

ICMP Inspect seq num not matched (inspect-icmp-seq-num-not-matched) 27918

ICMP Error Inspect no existing conn (inspect-icmp-error-no-existing-conn) 825

DNS Inspect id not matched (inspect-dns-id-not-matched) 19618

Snort instance is busy (snort-busy) 186

FP L2 rule drop (l2_acl) 26221

Unable to obtain connection lock (connection-lock) 1232

Interface is down (interface-down) 6437816

Async lock queue limit exceeded (async-lock-queue-limit) 235

Connection to PAT address without pre-existing xlate (nat-no-xlate-to-pat-pool) 172

Received a multicast packet in the non-active device (mcast-in-nonactive-device) 179570

Not a blocking packet (none) 188

Fragment reassembly failed (fragment-reassembly-failed) 109

Packet is blocked as requested by snort (snort-block) 495800

Dispatch queue tail drops (dispatch-queue-limit) 313660

Last clearing: Never

Flow drop:

Flow is denied by access rule (acl-drop) 735636

NAT reverse path failed (nat-rpf-failed) 32

Inspection failure (inspect-fail) 28476

Last clearing: Never

Low

Frame drop:

Expired VPN context (vpn-context-expired) 3

Invalid IP length (invalid-ip-length) 582021

Invalid TCP Length (invalid-tcp-hdr-length) 9730

Invalid UDP Length (invalid-udp-length) 220

No valid adjacency (no-adjacency) 464150

No valid V4 adjacency. Check ARP table (show arp) has entry for nexthop. (no-v4-adjacency) 2

No route to host (no-route) 288465

Flow is denied by configured rule (acl-drop) 683442015

Invalid SPI (np-sp-invalid-spi) 6

First TCP packet not SYN (tcp-not-syn) 32867754

Bad TCP flags (bad-tcp-flags) 15

TCP data send after FIN (tcp-data-past-fin) 20

TCP failed 3 way handshake (tcp-3whs-failed) 3938424

TCP RST/FIN out of order (tcp-rstfin-ooo) 64936421

TCP SEQ in SYN/SYNACK invalid (tcp-seq-syn-diff) 768612

TCP SYN on established conn (tcp-syn-ooo) 37732

TCP SYNACK on established conn (tcp-synack-ooo) 716

TCP packet SEQ past window (tcp-seq-past-win) 68698

TCP invalid ACK (tcp-invalid-ack) 8144

TCP replicated flow pak drop (tcp-fo-drop) 796

TCP Out-of-Order packet buffer timeout (tcp-buffer-timeout) 22

TCP RST/SYN in window (tcp-rst-syn-in-win) 30515

TCP dup of packet in Out-of-Order queue (tcp-dup-in-queue) 83

TCP packet failed PAWS test (tcp-paws-fail) 18360

Slowpath security checks failed (sp-security-failed) 473568

Dropped by standby unit (fo-standby) 519

Expired flow (flow-expired) 5

ICMP Inspect bad icmp code (inspect-icmp-bad-code) 1192

ICMP Inspect seq num not matched (inspect-icmp-seq-num-not-matched) 5383

ICMP Error Inspect no existing conn (inspect-icmp-error-no-existing-conn) 2456

DNS Inspect invalid packet (inspect-dns-invalid-pak) 151

DNS Inspect invalid domain label (inspect-dns-invalid-domain-label) 8944

DNS Inspect packet too long (inspect-dns-pak-too-long) 6754

DNS Inspect id not matched (inspect-dns-id-not-matched) 716738

Snort instance is busy (snort-busy) 54993

FP L2 rule drop (l2_acl) 11518879

Unable to obtain connection lock (connection-lock) 1

Interface is down (interface-down) 7606957

IKE new SA limit exceeded (ike-sa-rate-limit) 12

NAT failed (nat-xlate-failed) 82

NAT failed due to pool exhaustion (nat-xlate-pool-exhausted) 70158

Connection to PAT address without pre-existing xlate (nat-no-xlate-to-pat-pool) 191621601

Received a multicast packet in the non-active device (mcast-in-nonactive-device) 18910

Blocked or blacklisted by the firewall preprocessor (firewall) 54305

Blocked or blacklisted by the session preprocessor (session-preproc) 1492

Fragment reassembly failed (fragment-reassembly-failed) 176

Packet is blacklisted by snort (snort-blacklist) 1

Packet is blocked as requested by snort (snort-block) 1579977

Failover link is not ready for processing NLP packets (ha-nlp-lu-link-not-ready) 8

Dispatch queue tail drops (dispatch-queue-limit) 416935

Last clearing: Never

Flow drop:

Need to start IKE negotiation (need-ike) 5448

VPN decryption missing (vpn-missing-decrypt) 536

Flow is denied by access rule (acl-drop) 3394

NAT reverse path failed (nat-rpf-failed) 1048

Inspection failure (inspect-fail) 3872922

Last clearing: Never

tvotna · ‎02-20-2024

You can use "show perfstat" or better perfstats from Linux to understand load and traffic profile. On Snort2 systems this would be something like:

cd /var/sf/detection_engines/<DE-UUID>
for i in `ls ./ | grep "instance-"`; do echo $i; perfstats -q < $i/now; done;

Not sure if this is still the same for Snort3.