Dispatch Unit - High CPU

[edward ventura]

Hi All,

I'm trying to do some research on the Dispatch Unit process.  It seems High CPU and this process go hand in hand.  I haven't figured out an effective way of determining what underlying issue is the actual source.  Can someone point me in the right direction to try an understand what the Dispatch Unit process is doing?  I have an ASA 5550.  I have seen the cpu hover around 85% +- 5% for sustained long periods, 30 - 60 min +.  I have always been under the impression that around 80% cpu and you're probably dropping packets (that could be an out-dated belief).

Any help to understand this is much appreciated.

-E

[Marcin Latosiewicz]

Edward,

Dispatch unit is the central packet processing process, it does interface polling and MULTIPLE check on ASP.

Typically when your high cpu is in dispatch, you need to look into traffic pattern, enabling unicast RPF on all interfaces is a good start.

"show int" and "show traff" , "show perfmon" to understand what traffic and on which interface is causing this problem.

M.

[edward ventura]

Marcin,

Thank you for the response.  I was curious if the ASP "process" was happening within the dispatch unit process.  This helps.  I did do some more research on unicast RPF.  This will take some more research.  I've found some good documentation and thank you for that direction.

My interface statistics were clean.

CPU utilization for 5 seconds = 80%; 1 minute: 79%; 5 minutes: 74%

show conn count

76078 in use, 139617 most used

show proc cpu-usage sorted non-zero

PC         Thread       5Sec     1Min     5Min   Process

0x081d8531   0x1bdc1528    77.2%    78.0%    73.9%   Dispatch Unit

0x08ead27c   0x1bdb9d50     0.8%     0.7%     0.7%   Logger

0x08ef7d09   0x1bdae270     0.2%     0.2%     0.2%   snmp

0x08e8f4ec   0x1bdc0b00     0.2%     0.2%     0.2%   ssm4ge_cfg_poll_thread

0x08e6c2f5   0x1bd9de70     0.1%     0.0%     0.0%   ssh

Show traffic, the outside int was the busiest:

Outside:

        received (in 15571.990 secs):

                77762147 packets        12436865813 bytes

                4166 pkts/sec   798117 bytes/sec

        transmitted (in 15571.990 secs):

                58494240 packets        57798648078 bytes

                3204 pkts/sec   3711154 bytes/sec

      1 minute input rate 2712 pkts/sec,  399665 bytes/sec

      1 minute output rate 3130 pkts/sec,  3244847 bytes/sec

      1 minute drop rate, 40 pkts/sec

      5 minute input rate 4025 pkts/sec,  1922721 bytes/sec

      5 minute output rate 4075 pkts/sec,  3665981 bytes/sec

      5 minute drop rate, 45 pkts/sec

PERFMON STATS:                     Current      Average

Xlates                                2/s          7/s

Connections                         744/s          6/s

TCP Conns                           470/s          9/s

UDP Conns                           259/s          6/s

URL Access                            0/s          0/s

URL Server Req                        0/s          0/s

TCP Fixup                             0/s          0/s

TCP Intercept Established Conns       0/s          0/s

TCP Intercept Attempts                0/s          0/s

TCP Embryonic Conns Timeout           1/s          4/s

HTTP Fixup                            0/s          0/s

FTP Fixup                             0/s          0/s

AAA Authen                            0/s          0/s

AAA Author                            0/s          0/s

AAA Account                           0/s          0/s

[integreon]

Hi,

I agree with Marcin. Share those outputs here to help you further.

Anton

Sent from Cisco Technical Support iPad App

[integreon]

Edward,

Plese share the output of "sh service-policy"

Regards,

Anton

[edward ventura]

I didn't clear these statistics.  These are probably since the last power on.

Global policy:

  Service-policy: global_policy

    Class-map: inspection_default

      Inspect: ftp, packet 263278, drop 0, reset-drop 0

      Inspect: h323 h225 _default_h323_map, packet 0, drop 0, reset-drop 0

               tcp-proxy: bytes in buffer 0, bytes dropped 0

      Inspect: h323 ras _default_h323_map, packet 0, drop 0, reset-drop 0

      Inspect: rsh, packet 0, drop 0, reset-drop 0

      Inspect: rtsp, packet 0, drop 0, reset-drop 0

               tcp-proxy: bytes in buffer 0, bytes dropped 0

      Inspect: sqlnet, packet 0, drop 0, reset-drop 0

      Inspect: skinny , packet 0, drop 0, reset-drop 0

               tcp-proxy: bytes in buffer 0, bytes dropped 0

      Inspect: sunrpc, packet 12680779, drop 0, reset-drop 0

               tcp-proxy: bytes in buffer 0, bytes dropped 0

      Inspect: xdmcp, packet 0, drop 0, reset-drop 0

      Inspect: sip , packet 0, drop 0, reset-drop 0

               tcp-proxy: bytes in buffer 0, bytes dropped 0

      Inspect: netbios, packet 637867192, drop 0, reset-drop 0

      Inspect: tftp, packet 0, drop 0, reset-drop 0

      Inspect: icmp, packet 10303228, drop 0, reset-drop 0

      Inspect: icmp error, packet 948054, drop 18, reset-drop 0

      Inspect: ip-options _default_ip_options_map, packet 0, drop 0, reset-drop 0

    Class-map: global-class

[integreon]

Edward,

I believe you are using inspection_default. If yes, issue below command. This will not tamper your production. Lets observe the CPU for another half an hour

policy-map global_policy

        class inspection_default

          no inspect icmp

          no inspect icmp error

Regards,

Anton

[integreon]

Friend,

Is the CPU utilization back to normal?

Regards,

Anton

[Arshad Khan]

Hey Integreon,

I have same issue here's my default global policy,

Inspect: icmp error, packet 12451113, drop 40170, reset-drop 0

Inspect: icmp, packet 317250117, drop 20057, reset-drop 0

Is droping packet making cpu utilization higher?

Regards,

Arshad Ahmed

[integreon]

Hi Arshad,

Not the drops, but the inspection. What is your ASA software version?

Sent from Cisco Technical Support iPad App

[Arshad Khan]

Hi integreon,

Heres my software version.

Cisco Adaptive Security Appliance Software Version 8.2(1)11

Device Manager Version 6.2(3)

Regards,

Arshad Ahmed

[edward ventura]

Anton,

Thank you for the suggestion.  It will take some time before I can implement the changes.  I will post back the out come once I have made the change.

[edward ventura]

I removed both inspect icmp and icmp error:

Global policy:

  Service-policy: global_policy

    Class-map: inspection_default

      Inspect: ftp, packet 366, drop 0, reset-drop 0

      Inspect: h323 h225 _default_h323_map, packet 0, drop 0, reset-drop 0

               tcp-proxy: bytes in buffer 0, bytes dropped 0

      Inspect: h323 ras _default_h323_map, packet 0, drop 0, reset-drop 0

      Inspect: rsh, packet 0, drop 0, reset-drop 0

      Inspect: rtsp, packet 0, drop 0, reset-drop 0

               tcp-proxy: bytes in buffer 0, bytes dropped 0

      Inspect: sqlnet, packet 0, drop 0, reset-drop 0

      Inspect: skinny , packet 0, drop 0, reset-drop 0

               tcp-proxy: bytes in buffer 0, bytes dropped 0

      Inspect: sunrpc, packet 41429, drop 0, reset-drop 0

               tcp-proxy: bytes in buffer 0, bytes dropped 0

      Inspect: xdmcp, packet 0, drop 0, reset-drop 0

      Inspect: sip , packet 0, drop 0, reset-drop 0

               tcp-proxy: bytes in buffer 0, bytes dropped 0

      Inspect: netbios, packet 106690219, drop 0, reset-drop 0

      Inspect: tftp, packet 0, drop 0, reset-drop 0

      Inspect: ip-options _default_ip_options_map, packet 0, drop 0, reset-drop 0

    Class-map: global-class

The cpu is still in the 80% range.  What struck me as odd was the number of inspected netbios packets.

[integreon]

OK. Try to remove netbios too. Also what is your ASA version? This might be because of a cosmetic bug too.

[edward ventura]

8.4(2)

I tried removing and adding the different inspect commands to see if they were in direct relation to cpu usage but it didn't follow.  Unless there is a bug, i think i need to look more into the traffic hitting the asa that is being processed and tune the configs from there.