3750 HIGH CPU problem

[michael Helmbo]

I have 2 questions i would love to get some help with:

Does anyone know what the HULC DAI process runs when a switch doesn't have arp inspection enabled?

If i run a: 
"clear control cpu-interface" followed by a "show control cpu-interface" 
I see "routing protocol" rising very fast on a 3750, but when I use 
"monitor capture point ip process-swiched" I don't see them in my capture. Can someone tell me what they could be?

I can add that the CPU disappears when we disconnect the user vlan.

[InayathUlla Sharieff]

Check your config to see if you have a default route configured pointing to an interface instead the IP address as seen below:

ip route 0.0.0.0 0.0.0.0 Fastethernet0/0

If yes change the default route to point to on IP address and check the CPU utilization again.

Here is the document for more info:

http://www.cisco.com/en/US/products/hw/switches/ps5023/products_tech_note09186a00807213f5.shtml

HTH

[michael Helmbo]

Thanks for the reply. We currently use ip default-gateway ...

I tried changing it to 0.0.0.0 ... just to see when we started trouble shooting - no difference

[InayathUlla Sharieff]

can you send me the following outputs:-

show ver

show proc cpu sorted | ex 0.00

show proc cpu history

show controllers cpu-interface
show platform port-asic stats drop
“show controllers cpu-interface”  // 
show platform tcam utilization 

Regards

inayath

[michael Helmbo]

Thanks and here it is:

[InayathUlla Sharieff]

CPU utilization for five seconds: 99%/14%; one minute: 99%; five minutes: 99%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process 
 261   504803937   166242963       3036 61.86% 60.59% 60.81%   0 HULC DAI Process 

The HULC DAI process is used for the arp inspection feature.  This feature can be very CPU intensive if there are a high number of ARP packets per second coming in for examination  by the CPU.

Next Action:

=========

do show ip int brief | in up|input|output|drops

and check which interface is broadcasting a lot of traffic and then either shut down that port or configure the  nmsp attachment suppress under the interface.

int x/x

nmsp attachment suppress

end

2- If you cant find the inteface with high input then run the debug and find out the ip address.

HTH

Regards

Inayath

********* Please do not forget to rate the usefull posts **********

[InayathUlla Sharieff]

Please find below page which describe on debug.

https://supportforums.cisco.com/document/12298401/troubleshooting-high-cpu-3750

HTH

Regards

Inayath

[michael Helmbo]

Thanks for the help. Love the "debug platform cpu..." command

like i wrote in the beginning we see ""routing protocol" increase faster than anything. With that command we can now see that its primarily arps on the user vlan. The problem we face now is that we see a large amount of packages dropped in a short time on that when we use the command and it seems like its only our 15... switches all other switches don't have a problem - only 3750E with IOS 15....

[InayathUlla Sharieff]

3750 is not defined to be used to run more Routing hence you need to reduce the routing on this switch.

HTH

Regards

Inayath

************Please do not forget to rate all usefull posts ***********

[michael Helmbo]

There is no routing configured on that switch. There is only a management addr and a default route. We actually have a second 3750 - same IOS - same vlans, same problem - all our switches are only configured with mgmt ip and default gateway. But only those 2 have the problem.

Thats why i in my original post wrote that i would love to know what "routing protocol" in the command : "show control cpu-interface" could mean, cause there is no chance that its actually real routing updates. But maybe random arps from the user machines, but it doesn't seem to effect anything else than the 3750 15....

[the_kirschi]

Hi,

I have exactly the same problem with our 3750X stack. It just started about two weeks ago from out of nowhere. The stack does no ip routing at all although it did about a year ago. Now there is only one IP Interface which is in the management Vlan (not the built-in Management). I can see the same increase in "routing protocol" when running the "show controllers cpu-interface". It really seems to be an identical issue. If you read this - as the thread is already a few years old - did you ever solve the issue? We run Version 15.0(2)SE6.

What did you mean when you said the problem disappeared when you "disconnect the user vlan."?

Thanks

Daniel