High CPU Utilization on Cisco ASA 5520 - Dispatch Unit Process

shafhuss · ‎04-26-2018

Hi Team,

I am observing high cpu utlization on cisco asa 5520. I have found that Dispatch Unit process is causing high cpu. I have captured some of the outputs for following commands for your reference. Also, when i look into it, there are lots of LDAP queries and DNS Drops.

I see lots of input errors and overruns on Inside and Outside interface of the firewall as well. Can someone suggest what is causing this.

****************************************************************

Process: hostscan_token_cleaner, PROC_PC_TOTAL: 39, MAXHOG: 5, LASTHOG: 2

LASTHOG At: 05:46:28 UTC Mar 19 2018

PC: 0x096b4c0c (suspend)

Process: hostscan_token_cleaner, NUMHOG: 30, MAXHOG: 5, LASTHOG: 2

LASTHOG At: 05:46:28 UTC Mar 19 2018

PC: 0x096b4c0c (suspend)

Call stack: 0x096b4c0c 0x0806a65c

Process: snmp, PROC_PC_TOTAL: 1, MAXHOG: 9, LASTHOG: 9

LASTHOG At: 08:44:46 UTC Mar 19 2018

PC: 0x091b4cd9 (suspend)

Process: snmp, NUMHOG: 1, MAXHOG: 9, LASTHOG: 9

LASTHOG At: 08:44:46 UTC Mar 19 2018

PC: 0x091b4cd9 (suspend)

Call stack: 0x091b4cd9 0x09066edf 0x09041bc5 0x0903fb94 0x0806a65c

Process: Unicorn Admin Handler, NUMHOG: 3, MAXHOG: 10, LASTHOG: 4

LASTHOG At: 10:17:31 UTC Apr 11 2018

PC: 0x0806d64e (suspend)

Call stack: 0x0806d64e 0x084e361d 0x084e1bf1 0x08cb1a33 0x0857a368 0x08572469 0x085726f2

0x08572a2e 0x0857910b 0x0806a65c

Process: Unicorn Proxy Thread, PROC_PC_TOTAL: 87, MAXHOG: 6, LASTHOG: 6

LASTHOG At: 10:18:14 UTC Apr 11 2018

PC: 0x091bdc05 (suspend)

Process: Unicorn Proxy Thread, NUMHOG: 87, MAXHOG: 6, LASTHOG: 6

LASTHOG At: 10:18:14 UTC Apr 11 2018

PC: 0x091bdc05 (suspend)

Call stack: 0x0806d5e0 0x091bdc05 0x091be11d 0x091d248d 0x091d3dec 0x092eed2b 0x092ed446

0x092e8fdb 0x092e869a 0x092e9d15 0x092d9fda 0x092d966b 0x092d973c 0x092d2325

Process: ssh, PROC_PC_TOTAL: 26, MAXHOG: 9, LASTHOG: 6

LASTHOG At: 10:19:47 UTC Apr 11 2018

PC: 0x08c887fd (suspend)

Process: ssh, NUMHOG: 26, MAXHOG: 9, LASTHOG: 6

LASTHOG At: 10:19:47 UTC Apr 11 2018

PC: 0x08c887fd (suspend)

Call stack: 0x08c887fd 0x098e589e 0x08ea3e44 0x08c91e64 0x08c93901 0x080f7f0b 0x080fa046

0x080faf7c 0x0806a65c

Process: Dispatch Unit, PROC_PC_TOTAL: 773, MAXHOG: 5, LASTHOG: 2

LASTHOG At: 14:54:57 UTC Apr 12 2018

PC: 0x082a4169 (suspend)

Process: Dispatch Unit, NUMHOG: 773, MAXHOG: 5, LASTHOG: 2

LASTHOG At: 14:54:57 UTC Apr 12 2018

PC: 0x082a4169 (suspend)

Call stack: 0x082a4169 0x0806a65c

Process: ssh_init, NUMHOG: 178, MAXHOG: 3, LASTHOG: 2

LASTHOG At: 20:27:18 UTC Apr 12 2018

PC: 0x0911079d (suspend)

Call stack: 0x0911079d 0x09122720 0x09128c99 0x091162b4 0x0806a65c

Process: snmp, PROC_PC_TOTAL: 51, MAXHOG: 11, LASTHOG: 11

LASTHOG At: 21:06:02 UTC Apr 12 2018

PC: 0x0906e5de (suspend)

Process: snmp, NUMHOG: 51, MAXHOG: 11, LASTHOG: 11

LASTHOG At: 21:06:02 UTC Apr 12 2018

PC: 0x0906e5de (suspend)

Call stack: 0x0906e5de 0x0906d5b0 0x0906a50e 0x0906d146 0x09041d9b 0x0903fb94 0x0806a65c

Process: emweb/https, NUMHOG: 4, MAXHOG: 5, LASTHOG: 2

LASTHOG At: 06:23:53 UTC Apr 13 2018

PC: 0x0806d64e (suspend)

Call stack: 0x0806d64e 0x0832901f 0x0806a65c

Process: emweb/https, NUMHOG: 186, MAXHOG: 7, LASTHOG: 7

LASTHOG At: 07:39:20 UTC Apr 13 2018

PC: 0x0806d64e (suspend)

Call stack: 0x0806d64e 0x08328c1d 0x0806a65c

Process: Session Manager, PROC_PC_TOTAL: 254, MAXHOG: 15, LASTHOG: 3

LASTHOG At: 09:50:18 UTC Apr 13 2018

PC: 0x08ff142d (suspend)

Process: Session Manager, NUMHOG: 254, MAXHOG: 15, LASTHOG: 3

LASTHOG At: 09:50:18 UTC Apr 13 2018

PC: 0x08ff142d (suspend)

Call stack: 0x08ff142d 0x0806a65c

Process: ssh, PROC_PC_TOTAL: 522, MAXHOG: 49, LASTHOG: 9

LASTHOG At: 09:55:35 UTC Apr 13 2018

PC: 0x0806d64e (suspend)

Process: ssh_init, NUMHOG: 283, MAXHOG: 10, LASTHOG: 9

LASTHOG At: 09:55:35 UTC Apr 13 2018

PC: 0x0806d64e (suspend)

Call stack: 0x0806d64e 0x081996cb 0x0984a75b 0x0984e7fa 0x09840a9e 0x09840cc7 0x0986579f

0x098747e7 0x0986928a 0x081921d0 0x0815a0f2 0x09125ebb 0x09121606 0x0912010e

Process: Unicorn Proxy Thread, PROC_PC_TOTAL: 9989, MAXHOG: 128, LASTHOG: 3

LASTHOG At: 09:55:52 UTC Apr 13 2018

PC: 0x091bce35 (suspend)

Process: Unicorn Proxy Thread, NUMHOG: 9986, MAXHOG: 128, LASTHOG: 3

LASTHOG At: 09:55:52 UTC Apr 13 2018

PC: 0x091bce35 (suspend)

Call stack: 0x0806d5e0 0x091bce35 0x091bfb0d 0x091bfbc5 0x750ec8a4

Process: ldap_client_thread, PROC_PC_TOTAL: 1209860, MAXHOG: 431, LASTHOG: 3

LASTHOG At: 09:58:44 UTC Apr 13 2018

PC: 0x091dba7d (suspend)

Process: ldap_client_thread, NUMHOG: 1198801, MAXHOG: 431, LASTHOG: 3

LASTHOG At: 09:58:44 UTC Apr 13 2018

PC: 0x091dba7d (suspend)

Call stack: 0x0806c9ba 0x091dba7d 0x091bd7aa 0x091bfb0d 0x091bfbc5 0x00000014 0x098d868c

0x091bdbe7 0x091bec97 0x088c0605 0x0806a65c

Process: ssh_init, PROC_PC_TOTAL: 189, MAXHOG: 15, LASTHOG: 12

LASTHOG At: 09:58:45 UTC Apr 13 2018

PC: 0x0911079d (suspend)

Process: ssh, NUMHOG: 8, MAXHOG: 15, LASTHOG: 12

LASTHOG At: 09:58:45 UTC Apr 13 2018

PC: 0x0911079d (suspend)

Call stack: 0x0911079d 0x09122720 0x09124d9e 0x0911552b 0x09115896 0x09115d89 0x09117120

0x08c873fc 0x098e61f0 0x08106c0d 0x080f9ead 0x080faf7c 0x0806a65c

Process: Dispatch Unit, PROC_PC_TOTAL: 1761293, MAXHOG: 679, LASTHOG: 4

LASTHOG At: 09:58:46 UTC Apr 13 2018

PC: 0x082a4208 (suspend)

Process: Dispatch Unit, NUMHOG: 1541326, MAXHOG: 679, LASTHOG: 4

LASTHOG At: 09:58:46 UTC Apr 13 2018

PC: 0x082a4208 (suspend)

Call stack: 0x082a4208 0x0806a65c

Process: Dispatch Unit, PROC_PC_TOTAL: 21167702, MAXHOG: 821, LASTHOG: 3

LASTHOG At: 09:58:46 UTC Apr 13 2018

PC: 0x082a445c (suspend)

Process: Dispatch Unit, NUMHOG: 4586109, MAXHOG: 821, LASTHOG: 3

LASTHOG At: 09:58:46 UTC Apr 13 2018

PC: 0x082a445c (suspend)

Call stack: 0x082a445c 0x0806a65c

CPU hog threshold (msec): 2.844

Last cleared: None

shafhuss · ‎04-26-2018

Please see some more command outputs.

IND-HYD-GSD-VPN-FW-01/pri/act# show interface
Interface GigabitEthernet0/0 "Outside", is up, line protocol is up
Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec
Full-Duplex(Full-duplex), 1000 Mbps(1000 Mbps)
Input flow control is unsupported, output flow control is off
Description: - interface connecting to VPN_OUT of Uppal Internet firewall
MAC address 001c.58d3.f8f6, MTU 1500
IP address 58.2.238.23, subnet mask 255.255.255.224
15795135326 packets input, 4094502776608 bytes, 0 no buffer
Received 495531 broadcasts, 0 runts, 0 giants
34239993 input errors, 0 CRC, 0 frame, 34239993 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
24520208688 packets output, 10874305004401 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 2 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops, 0 tx hangs
input queue (blocks free curr/low): hardware (255/230)
output queue (blocks free curr/low): hardware (255/97)
Traffic Statistics for "Outside":
15794944374 packets input, 3781578180407 bytes
24520208688 packets output, 10366793510014 bytes
138424540 packets dropped
1 minute input rate 5673 pkts/sec, 1299714 bytes/sec
1 minute output rate 8783 pkts/sec, 3732798 bytes/sec
1 minute drop rate, 58 pkts/sec
5 minute input rate 5998 pkts/sec, 1413782 bytes/sec
5 minute output rate 9200 pkts/sec, 3745219 bytes/sec
5 minute drop rate, 64 pkts/sec
Interface GigabitEthernet0/1 "Inside", is up, line protocol is up
Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec
Full-Duplex(Full-duplex), 1000 Mbps(1000 Mbps)
Input flow control is unsupported, output flow control is off
Description: - interface connecting to VPN_IN of Uppal Internet firewall
MAC address 001c.58d3.f8f7, MTU 1500
IP address 58.2.64.43, subnet mask 255.255.255.224
13672161203 packets input, 9092690613902 bytes, 0 no buffer
Received 125930 broadcasts, 0 runts, 0 giants
68992986 input errors, 0 CRC, 0 frame, 68992986 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
11138084894 packets output, 2794033178367 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 2 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops, 0 tx hangs
input queue (blocks free curr/low): hardware (255/230)
output queue (blocks free curr/low): hardware (253/0)
Traffic Statistics for "Inside":
13672161173 packets input, 8832327224938 bytes
11138084894 packets output, 2557197084862 bytes
296806020 packets dropped
1 minute input rate 4948 pkts/sec, 3195541 bytes/sec
1 minute output rate 3942 pkts/sec, 831289 bytes/sec
1 minute drop rate, 107 pkts/sec
5 minute input rate 5157 pkts/sec, 3230104 bytes/sec
5 minute output rate 4222 pkts/sec, 945345 bytes/sec
5 minute drop rate, 160 pkts/sec
Interface GigabitEthernet0/2 "state_failover", is up, line protocol is up
Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
Description: LAN/STATE Failover Interface
MAC address 001c.58d3.f8f8, MTU 1500
IP address 192.168.16.1, subnet mask 255.255.255.0
7108037 packets input, 613847480 bytes, 0 no buffer
Received 302 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
402282378 packets output, 494877025372 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 15 interface resets
0 late collisions, 0 deferred
7 input reset drops, 0 output reset drops, 0 tx hangs
input queue (blocks free curr/low): hardware (255/232)
output queue (blocks free curr/low): hardware (255/176)
Traffic Statistics for "state_failover":
7107905 packets input, 457794966 bytes
402282244 packets output, 487635833616 bytes
0 packets dropped
1 minute input rate 1 pkts/sec, 84 bytes/sec
1 minute output rate 162 pkts/sec, 198240 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 1 pkts/sec, 96 bytes/sec
5 minute output rate 181 pkts/sec, 221471 bytes/sec
5 minute drop rate, 0 pkts/sec
Interface GigabitEthernet0/3 "TEST", is down, line protocol is down
Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec
Auto-Duplex, Auto-Speed
Input flow control is unsupported, output flow control is off
MAC address 001c.58d3.f8f9, MTU 1500
IP address 192.168.1.1, subnet mask 255.255.255.0
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
0 packets output, 0 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 2 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops, 0 tx hangs
input queue (blocks free curr/low): hardware (255/255)
output queue (blocks free curr/low): hardware (255/255)
Traffic Statistics for "TEST":
0 packets input, 0 bytes
0 packets output, 0 bytes
1041509 packets dropped
1 minute input rate 0 pkts/sec, 0 bytes/sec
1 minute output rate 0 pkts/sec, 0 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 0 pkts/sec, 0 bytes/sec
5 minute output rate 0 pkts/sec, 0 bytes/sec
5 minute drop rate, 0 pkts/sec
Interface Management0/0 "", is administratively down, line protocol is down
Hardware is i82557, BW 100 Mbps, DLY 100 usec
Auto-Duplex, Auto-Speed
Input flow control is unsupported, output flow control is unsupported
Available but not configured via nameif
MAC address 001c.58d3.f8fa, MTU not set
IP address unassigned
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
0 packets output, 0 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collisions, 0 deferred
0 lost carrier, 0 no carrier
0 input reset drops, 0 output reset drops
input queue (curr/max packets): hardware (0/0) software (0/0)
output queue (curr/max packets): hardware (1/0) software (0/0)
IND-HYD-GSD-VPN-FW-01/pri/act# show interface | include errors
34240055 input errors, 0 CRC, 0 frame, 34240055 overrun, 0 ignored, 0 abort
0 output errors, 0 collisions, 2 interface resets
68993045 input errors, 0 CRC, 0 frame, 68993045 overrun, 0 ignored, 0 abort
0 output errors, 0 collisions, 2 interface resets
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 output errors, 0 collisions, 15 interface resets
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 output errors, 0 collisions, 2 interface resets
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 output errors, 0 collisions, 0 interface resets
IND-HYD-GSD-VPN-FW-01/pri/act#

shafhuss · ‎04-26-2018

Some more

shafhuss · ‎04-26-2018

Last

shafhuss · ‎04-27-2018

Hello All,

Any updates. Please help me with this.

Thanks,

Shaf