Solved: Dispatch Unit - High CPU

edward ventura · ‎01-30-2012

Hi All,

I'm trying to do some research on the Dispatch Unit process. It seems High CPU and this process go hand in hand. I haven't figured out an effective way of determining what underlying issue is the actual source. Can someone point me in the right direction to try an understand what the Dispatch Unit process is doing? I have an ASA 5550. I have seen the cpu hover around 85% +- 5% for sustained long periods, 30 - 60 min +. I have always been under the impression that around 80% cpu and you're probably dropping packets (that could be an out-dated belief).

Any help to understand this is much appreciated.

-E

Maykol Rojas · ‎02-06-2012

Hello Edward,

That is indeed very true, but that only aleaveates the CPU processing but it does not get rid of the issue, based on the ICMP messages and netbios, there could be loops and packets bouncing, hence the high amount of inspection hits. Adding to the packet processing on the interface itself, the inspection will add another CPU intensing consuming.

As you rightly pointed, the main thing on these cases is to first identify the traffic and then, check if it is normal or not.

Mike

View solution in original post

Marcin Latosiewicz · ‎01-30-2012

Edward,

Dispatch unit is the central packet processing process, it does interface polling and MULTIPLE check on ASP.

Typically when your high cpu is in dispatch, you need to look into traffic pattern, enabling unicast RPF on all interfaces is a good start.

"show int" and "show traff" , "show perfmon" to understand what traffic and on which interface is causing this problem.

M.

edward ventura · ‎01-30-2012

Marcin,

Thank you for the response. I was curious if the ASP "process" was happening within the dispatch unit process. This helps. I did do some more research on unicast RPF. This will take some more research. I've found some good documentation and thank you for that direction.

My interface statistics were clean.

CPU utilization for 5 seconds = 80%; 1 minute: 79%; 5 minutes: 74%

show conn count

76078 in use, 139617 most used

show proc cpu-usage sorted non-zero

PC Thread 5Sec 1Min 5Min Process

0x081d8531 0x1bdc1528 77.2% 78.0% 73.9% Dispatch Unit

0x08ead27c 0x1bdb9d50 0.8% 0.7% 0.7% Logger

0x08ef7d09 0x1bdae270 0.2% 0.2% 0.2% snmp

0x08e8f4ec 0x1bdc0b00 0.2% 0.2% 0.2% ssm4ge_cfg_poll_thread

0x08e6c2f5 0x1bd9de70 0.1% 0.0% 0.0% ssh

Show traffic, the outside int was the busiest:

Outside:

received (in 15571.990 secs):

77762147 packets 12436865813 bytes

4166 pkts/sec 798117 bytes/sec

transmitted (in 15571.990 secs):

58494240 packets 57798648078 bytes

3204 pkts/sec 3711154 bytes/sec

1 minute input rate 2712 pkts/sec, 399665 bytes/sec

1 minute output rate 3130 pkts/sec, 3244847 bytes/sec

1 minute drop rate, 40 pkts/sec

5 minute input rate 4025 pkts/sec, 1922721 bytes/sec

5 minute output rate 4075 pkts/sec, 3665981 bytes/sec

5 minute drop rate, 45 pkts/sec

PERFMON STATS: Current Average

Xlates 2/s 7/s

Connections 744/s 6/s

TCP Conns 470/s 9/s

UDP Conns 259/s 6/s

URL Access 0/s 0/s

URL Server Req 0/s 0/s

TCP Fixup 0/s 0/s

TCP Intercept Established Conns 0/s 0/s

TCP Intercept Attempts 0/s 0/s

TCP Embryonic Conns Timeout 1/s 4/s

HTTP Fixup 0/s 0/s

FTP Fixup 0/s 0/s

AAA Authen 0/s 0/s

AAA Author 0/s 0/s

AAA Account 0/s 0/s

integreon · ‎01-30-2012

Hi,

I agree with Marcin. Share those outputs here to help you further.

Anton

Sent from Cisco Technical Support iPad App

integreon · ‎01-30-2012

Edward,

Plese share the output of "sh service-policy"

Regards,

Anton

edward ventura · ‎01-30-2012

I didn't clear these statistics. These are probably since the last power on.

Global policy:

Service-policy: global_policy

Class-map: inspection_default

Inspect: ftp, packet 263278, drop 0, reset-drop 0

Inspect: h323 h225 _default_h323_map, packet 0, drop 0, reset-drop 0

tcp-proxy: bytes in buffer 0, bytes dropped 0

Inspect: h323 ras _default_h323_map, packet 0, drop 0, reset-drop 0

Inspect: rsh, packet 0, drop 0, reset-drop 0

Inspect: rtsp, packet 0, drop 0, reset-drop 0

tcp-proxy: bytes in buffer 0, bytes dropped 0

Inspect: sqlnet, packet 0, drop 0, reset-drop 0

Inspect: skinny , packet 0, drop 0, reset-drop 0

tcp-proxy: bytes in buffer 0, bytes dropped 0

Inspect: sunrpc, packet 12680779, drop 0, reset-drop 0

tcp-proxy: bytes in buffer 0, bytes dropped 0

Inspect: xdmcp, packet 0, drop 0, reset-drop 0

Inspect: sip , packet 0, drop 0, reset-drop 0

tcp-proxy: bytes in buffer 0, bytes dropped 0

Inspect: netbios, packet 637867192, drop 0, reset-drop 0

Inspect: tftp, packet 0, drop 0, reset-drop 0

Inspect: icmp, packet 10303228, drop 0, reset-drop 0

Inspect: icmp error, packet 948054, drop 18, reset-drop 0

Inspect: ip-options _default_ip_options_map, packet 0, drop 0, reset-drop 0

Class-map: global-class

integreon · ‎01-30-2012

Edward,

I believe you are using inspection_default. If yes, issue below command. This will not tamper your production. Lets observe the CPU for another half an hour

policy-map global_policy

class inspection_default

no inspect icmp

no inspect icmp error

Regards,

Anton

integreon · ‎01-30-2012

Friend,

Is the CPU utilization back to normal?

Regards,

Anton

Arshad Khan · ‎01-31-2012

Hey Integreon,

I have same issue here's my default global policy,

Inspect: icmp error, packet 12451113, drop 40170, reset-drop 0

Inspect: icmp, packet 317250117, drop 20057, reset-drop 0

Is droping packet making cpu utilization higher?

Regards,

Arshad Ahmed

integreon · ‎01-31-2012

Hi Arshad,

Not the drops, but the inspection. What is your ASA software version?

Sent from Cisco Technical Support iPad App

Arshad Khan · ‎01-31-2012

Hi integreon,

Heres my software version.

Cisco Adaptive Security Appliance Software Version 8.2(1)11

Device Manager Version 6.2(3)

Regards,

Arshad Ahmed

edward ventura · ‎01-31-2012

Anton,

Thank you for the suggestion. It will take some time before I can implement the changes. I will post back the out come once I have made the change.

edward ventura · ‎01-31-2012

I removed both inspect icmp and icmp error:

Global policy:

Service-policy: global_policy

Class-map: inspection_default

Inspect: ftp, packet 366, drop 0, reset-drop 0

Inspect: h323 h225 _default_h323_map, packet 0, drop 0, reset-drop 0

tcp-proxy: bytes in buffer 0, bytes dropped 0

Inspect: h323 ras _default_h323_map, packet 0, drop 0, reset-drop 0

Inspect: rsh, packet 0, drop 0, reset-drop 0

Inspect: rtsp, packet 0, drop 0, reset-drop 0

tcp-proxy: bytes in buffer 0, bytes dropped 0

Inspect: sqlnet, packet 0, drop 0, reset-drop 0

Inspect: skinny , packet 0, drop 0, reset-drop 0

tcp-proxy: bytes in buffer 0, bytes dropped 0

Inspect: sunrpc, packet 41429, drop 0, reset-drop 0

tcp-proxy: bytes in buffer 0, bytes dropped 0

Inspect: xdmcp, packet 0, drop 0, reset-drop 0

Inspect: sip , packet 0, drop 0, reset-drop 0

tcp-proxy: bytes in buffer 0, bytes dropped 0

Inspect: netbios, packet 106690219, drop 0, reset-drop 0

Inspect: tftp, packet 0, drop 0, reset-drop 0

Inspect: ip-options _default_ip_options_map, packet 0, drop 0, reset-drop 0

Class-map: global-class

The cpu is still in the 80% range. What struck me as odd was the number of inspected netbios packets.

integreon · ‎01-31-2012

OK. Try to remove netbios too. Also what is your ASA version? This might be because of a cosmetic bug too.

edward ventura · ‎01-31-2012

8.4(2)

I tried removing and adding the different inspect commands to see if they were in direct relation to cpu usage but it didn't follow. Unless there is a bug, i think i need to look more into the traffic hitting the asa that is being processed and tune the configs from there.