04-26-2018 12:09 AM - edited 03-12-2019 05:14 AM
Hi Team,
I am observing high cpu utlization on cisco asa 5520. I have found that Dispatch Unit process is causing high cpu. I have captured some of the outputs for following commands for your reference. Also, when i look into it, there are lots of LDAP queries and DNS Drops.
I see lots of input errors and overruns on Inside and Outside interface of the firewall as well. Can someone suggest what is causing this.
****************************************************************
Process: hostscan_token_cleaner, PROC_PC_TOTAL: 39, MAXHOG: 5, LASTHOG: 2
LASTHOG At: 05:46:28 UTC Mar 19 2018
PC: 0x096b4c0c (suspend)
Process: hostscan_token_cleaner, NUMHOG: 30, MAXHOG: 5, LASTHOG: 2
LASTHOG At: 05:46:28 UTC Mar 19 2018
PC: 0x096b4c0c (suspend)
Call stack: 0x096b4c0c 0x0806a65c
Process: snmp, PROC_PC_TOTAL: 1, MAXHOG: 9, LASTHOG: 9
LASTHOG At: 08:44:46 UTC Mar 19 2018
PC: 0x091b4cd9 (suspend)
Process: snmp, NUMHOG: 1, MAXHOG: 9, LASTHOG: 9
LASTHOG At: 08:44:46 UTC Mar 19 2018
PC: 0x091b4cd9 (suspend)
Call stack: 0x091b4cd9 0x09066edf 0x09041bc5 0x0903fb94 0x0806a65c
Process: Unicorn Admin Handler, NUMHOG: 3, MAXHOG: 10, LASTHOG: 4
LASTHOG At: 10:17:31 UTC Apr 11 2018
PC: 0x0806d64e (suspend)
Call stack: 0x0806d64e 0x084e361d 0x084e1bf1 0x08cb1a33 0x0857a368 0x08572469 0x085726f2
0x08572a2e 0x0857910b 0x0806a65c
Process: Unicorn Proxy Thread, PROC_PC_TOTAL: 87, MAXHOG: 6, LASTHOG: 6
LASTHOG At: 10:18:14 UTC Apr 11 2018
PC: 0x091bdc05 (suspend)
Process: Unicorn Proxy Thread, NUMHOG: 87, MAXHOG: 6, LASTHOG: 6
LASTHOG At: 10:18:14 UTC Apr 11 2018
PC: 0x091bdc05 (suspend)
Call stack: 0x0806d5e0 0x091bdc05 0x091be11d 0x091d248d 0x091d3dec 0x092eed2b 0x092ed446
0x092e8fdb 0x092e869a 0x092e9d15 0x092d9fda 0x092d966b 0x092d973c 0x092d2325
Process: ssh, PROC_PC_TOTAL: 26, MAXHOG: 9, LASTHOG: 6
LASTHOG At: 10:19:47 UTC Apr 11 2018
PC: 0x08c887fd (suspend)
Process: ssh, NUMHOG: 26, MAXHOG: 9, LASTHOG: 6
LASTHOG At: 10:19:47 UTC Apr 11 2018
PC: 0x08c887fd (suspend)
Call stack: 0x08c887fd 0x098e589e 0x08ea3e44 0x08c91e64 0x08c93901 0x080f7f0b 0x080fa046
0x080faf7c 0x0806a65c
Process: Dispatch Unit, PROC_PC_TOTAL: 773, MAXHOG: 5, LASTHOG: 2
LASTHOG At: 14:54:57 UTC Apr 12 2018
PC: 0x082a4169 (suspend)
Process: Dispatch Unit, NUMHOG: 773, MAXHOG: 5, LASTHOG: 2
LASTHOG At: 14:54:57 UTC Apr 12 2018
PC: 0x082a4169 (suspend)
Call stack: 0x082a4169 0x0806a65c
Process: ssh_init, NUMHOG: 178, MAXHOG: 3, LASTHOG: 2
LASTHOG At: 20:27:18 UTC Apr 12 2018
PC: 0x0911079d (suspend)
Call stack: 0x0911079d 0x09122720 0x09128c99 0x091162b4 0x0806a65c
Process: snmp, PROC_PC_TOTAL: 51, MAXHOG: 11, LASTHOG: 11
LASTHOG At: 21:06:02 UTC Apr 12 2018
PC: 0x0906e5de (suspend)
Process: snmp, NUMHOG: 51, MAXHOG: 11, LASTHOG: 11
LASTHOG At: 21:06:02 UTC Apr 12 2018
PC: 0x0906e5de (suspend)
Call stack: 0x0906e5de 0x0906d5b0 0x0906a50e 0x0906d146 0x09041d9b 0x0903fb94 0x0806a65c
Process: emweb/https, NUMHOG: 4, MAXHOG: 5, LASTHOG: 2
LASTHOG At: 06:23:53 UTC Apr 13 2018
PC: 0x0806d64e (suspend)
Call stack: 0x0806d64e 0x0832901f 0x0806a65c
Process: emweb/https, NUMHOG: 186, MAXHOG: 7, LASTHOG: 7
LASTHOG At: 07:39:20 UTC Apr 13 2018
PC: 0x0806d64e (suspend)
Call stack: 0x0806d64e 0x08328c1d 0x0806a65c
Process: Session Manager, PROC_PC_TOTAL: 254, MAXHOG: 15, LASTHOG: 3
LASTHOG At: 09:50:18 UTC Apr 13 2018
PC: 0x08ff142d (suspend)
Process: Session Manager, NUMHOG: 254, MAXHOG: 15, LASTHOG: 3
LASTHOG At: 09:50:18 UTC Apr 13 2018
PC: 0x08ff142d (suspend)
Call stack: 0x08ff142d 0x0806a65c
Process: ssh, PROC_PC_TOTAL: 522, MAXHOG: 49, LASTHOG: 9
LASTHOG At: 09:55:35 UTC Apr 13 2018
PC: 0x0806d64e (suspend)
Process: ssh_init, NUMHOG: 283, MAXHOG: 10, LASTHOG: 9
LASTHOG At: 09:55:35 UTC Apr 13 2018
PC: 0x0806d64e (suspend)
Call stack: 0x0806d64e 0x081996cb 0x0984a75b 0x0984e7fa 0x09840a9e 0x09840cc7 0x0986579f
0x098747e7 0x0986928a 0x081921d0 0x0815a0f2 0x09125ebb 0x09121606 0x0912010e
Process: Unicorn Proxy Thread, PROC_PC_TOTAL: 9989, MAXHOG: 128, LASTHOG: 3
LASTHOG At: 09:55:52 UTC Apr 13 2018
PC: 0x091bce35 (suspend)
Process: Unicorn Proxy Thread, NUMHOG: 9986, MAXHOG: 128, LASTHOG: 3
LASTHOG At: 09:55:52 UTC Apr 13 2018
PC: 0x091bce35 (suspend)
Call stack: 0x0806d5e0 0x091bce35 0x091bfb0d 0x091bfbc5 0x750ec8a4
Process: ldap_client_thread, PROC_PC_TOTAL: 1209860, MAXHOG: 431, LASTHOG: 3
LASTHOG At: 09:58:44 UTC Apr 13 2018
PC: 0x091dba7d (suspend)
Process: ldap_client_thread, NUMHOG: 1198801, MAXHOG: 431, LASTHOG: 3
LASTHOG At: 09:58:44 UTC Apr 13 2018
PC: 0x091dba7d (suspend)
Call stack: 0x0806c9ba 0x091dba7d 0x091bd7aa 0x091bfb0d 0x091bfbc5 0x00000014 0x098d868c
0x091bdbe7 0x091bec97 0x088c0605 0x0806a65c
Process: ssh_init, PROC_PC_TOTAL: 189, MAXHOG: 15, LASTHOG: 12
LASTHOG At: 09:58:45 UTC Apr 13 2018
PC: 0x0911079d (suspend)
Process: ssh, NUMHOG: 8, MAXHOG: 15, LASTHOG: 12
LASTHOG At: 09:58:45 UTC Apr 13 2018
PC: 0x0911079d (suspend)
Call stack: 0x0911079d 0x09122720 0x09124d9e 0x0911552b 0x09115896 0x09115d89 0x09117120
0x08c873fc 0x098e61f0 0x08106c0d 0x080f9ead 0x080faf7c 0x0806a65c
Process: Dispatch Unit, PROC_PC_TOTAL: 1761293, MAXHOG: 679, LASTHOG: 4
LASTHOG At: 09:58:46 UTC Apr 13 2018
PC: 0x082a4208 (suspend)
Process: Dispatch Unit, NUMHOG: 1541326, MAXHOG: 679, LASTHOG: 4
LASTHOG At: 09:58:46 UTC Apr 13 2018
PC: 0x082a4208 (suspend)
Call stack: 0x082a4208 0x0806a65c
Process: Dispatch Unit, PROC_PC_TOTAL: 21167702, MAXHOG: 821, LASTHOG: 3
LASTHOG At: 09:58:46 UTC Apr 13 2018
PC: 0x082a445c (suspend)
Process: Dispatch Unit, NUMHOG: 4586109, MAXHOG: 821, LASTHOG: 3
LASTHOG At: 09:58:46 UTC Apr 13 2018
PC: 0x082a445c (suspend)
Call stack: 0x082a445c 0x0806a65c
CPU hog threshold (msec): 2.844
Last cleared: None
04-26-2018 12:18 AM
Please see some more command outputs.
IND-HYD-GSD-VPN-FW-01/pri/act# show interface
Interface GigabitEthernet0/0 "Outside", is up, line protocol is up
Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec
Full-Duplex(Full-duplex), 1000 Mbps(1000 Mbps)
Input flow control is unsupported, output flow control is off
Description: - interface connecting to VPN_OUT of Uppal Internet firewall
MAC address 001c.58d3.f8f6, MTU 1500
IP address 58.2.238.23, subnet mask 255.255.255.224
15795135326 packets input, 4094502776608 bytes, 0 no buffer
Received 495531 broadcasts, 0 runts, 0 giants
34239993 input errors, 0 CRC, 0 frame, 34239993 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
24520208688 packets output, 10874305004401 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 2 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops, 0 tx hangs
input queue (blocks free curr/low): hardware (255/230)
output queue (blocks free curr/low): hardware (255/97)
Traffic Statistics for "Outside":
15794944374 packets input, 3781578180407 bytes
24520208688 packets output, 10366793510014 bytes
138424540 packets dropped
1 minute input rate 5673 pkts/sec, 1299714 bytes/sec
1 minute output rate 8783 pkts/sec, 3732798 bytes/sec
1 minute drop rate, 58 pkts/sec
5 minute input rate 5998 pkts/sec, 1413782 bytes/sec
5 minute output rate 9200 pkts/sec, 3745219 bytes/sec
5 minute drop rate, 64 pkts/sec
Interface GigabitEthernet0/1 "Inside", is up, line protocol is up
Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec
Full-Duplex(Full-duplex), 1000 Mbps(1000 Mbps)
Input flow control is unsupported, output flow control is off
Description: - interface connecting to VPN_IN of Uppal Internet firewall
MAC address 001c.58d3.f8f7, MTU 1500
IP address 58.2.64.43, subnet mask 255.255.255.224
13672161203 packets input, 9092690613902 bytes, 0 no buffer
Received 125930 broadcasts, 0 runts, 0 giants
68992986 input errors, 0 CRC, 0 frame, 68992986 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
11138084894 packets output, 2794033178367 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 2 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops, 0 tx hangs
input queue (blocks free curr/low): hardware (255/230)
output queue (blocks free curr/low): hardware (253/0)
Traffic Statistics for "Inside":
13672161173 packets input, 8832327224938 bytes
11138084894 packets output, 2557197084862 bytes
296806020 packets dropped
1 minute input rate 4948 pkts/sec, 3195541 bytes/sec
1 minute output rate 3942 pkts/sec, 831289 bytes/sec
1 minute drop rate, 107 pkts/sec
5 minute input rate 5157 pkts/sec, 3230104 bytes/sec
5 minute output rate 4222 pkts/sec, 945345 bytes/sec
5 minute drop rate, 160 pkts/sec
Interface GigabitEthernet0/2 "state_failover", is up, line protocol is up
Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
Description: LAN/STATE Failover Interface
MAC address 001c.58d3.f8f8, MTU 1500
IP address 192.168.16.1, subnet mask 255.255.255.0
7108037 packets input, 613847480 bytes, 0 no buffer
Received 302 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
402282378 packets output, 494877025372 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 15 interface resets
0 late collisions, 0 deferred
7 input reset drops, 0 output reset drops, 0 tx hangs
input queue (blocks free curr/low): hardware (255/232)
output queue (blocks free curr/low): hardware (255/176)
Traffic Statistics for "state_failover":
7107905 packets input, 457794966 bytes
402282244 packets output, 487635833616 bytes
0 packets dropped
1 minute input rate 1 pkts/sec, 84 bytes/sec
1 minute output rate 162 pkts/sec, 198240 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 1 pkts/sec, 96 bytes/sec
5 minute output rate 181 pkts/sec, 221471 bytes/sec
5 minute drop rate, 0 pkts/sec
Interface GigabitEthernet0/3 "TEST", is down, line protocol is down
Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec
Auto-Duplex, Auto-Speed
Input flow control is unsupported, output flow control is off
MAC address 001c.58d3.f8f9, MTU 1500
IP address 192.168.1.1, subnet mask 255.255.255.0
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
0 packets output, 0 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 2 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops, 0 tx hangs
input queue (blocks free curr/low): hardware (255/255)
output queue (blocks free curr/low): hardware (255/255)
Traffic Statistics for "TEST":
0 packets input, 0 bytes
0 packets output, 0 bytes
1041509 packets dropped
1 minute input rate 0 pkts/sec, 0 bytes/sec
1 minute output rate 0 pkts/sec, 0 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 0 pkts/sec, 0 bytes/sec
5 minute output rate 0 pkts/sec, 0 bytes/sec
5 minute drop rate, 0 pkts/sec
Interface Management0/0 "", is administratively down, line protocol is down
Hardware is i82557, BW 100 Mbps, DLY 100 usec
Auto-Duplex, Auto-Speed
Input flow control is unsupported, output flow control is unsupported
Available but not configured via nameif
MAC address 001c.58d3.f8fa, MTU not set
IP address unassigned
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
0 packets output, 0 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collisions, 0 deferred
0 lost carrier, 0 no carrier
0 input reset drops, 0 output reset drops
input queue (curr/max packets): hardware (0/0) software (0/0)
output queue (curr/max packets): hardware (1/0) software (0/0)
IND-HYD-GSD-VPN-FW-01/pri/act# show interface | include errors
34240055 input errors, 0 CRC, 0 frame, 34240055 overrun, 0 ignored, 0 abort
0 output errors, 0 collisions, 2 interface resets
68993045 input errors, 0 CRC, 0 frame, 68993045 overrun, 0 ignored, 0 abort
0 output errors, 0 collisions, 2 interface resets
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 output errors, 0 collisions, 15 interface resets
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 output errors, 0 collisions, 2 interface resets
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 output errors, 0 collisions, 0 interface resets
IND-HYD-GSD-VPN-FW-01/pri/act#
04-26-2018 12:18 AM
04-26-2018 12:20 AM
04-27-2018 01:37 AM
Hello All,
Any updates. Please help me with this.
Thanks,
Shaf
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: