cisco l2 switch high cpu history

[PKH1994]

Hello,

 We are using 30 nos of IPCOM aps for client WiFi connections and Cisco WS-C2960 switch. As observed, sometimes we have a heavy latency for a getaway. In CPU history that times showing 100% utilization in CPU% per second (last 60 seconds) table. PFA and pls help me to  resolve the issue.  

[balaji.bandi]

As per the output, we see only high on 72 hours rest for me looks ok.

as per the cpu process it was 22% and most was HULC, so that is normal if you have fully populated switch ?

how many ports active ? what kind of througput proicessing by switch ?

some tips for high CPU still work same on 2960

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html

If possible post the configuration.

[PKH1994]

Thank you all for the reply.

This is l2 switch, 36 ports are connected to AccessPoint, approx 50 to 70 clients are connected to the WiFi network and bandwidth speed is 10 Mbps for each user device.currently user does not affected any issue. but I am unable to access the switch sometimes, getting high latency and packet drops to the gateway. PFA of config of switch. any changes required in configuration/ . 

[balaji.bandi · Accepted solution]

I only see "Hulc LED Process" is high as per the output.

check you can remove any Logging config and snmp disable see any improvement.

check outgoing interface any INPUT/OUTPUT drops?

Note : WS-C2960-48PST-L - quite old ( i do not believe any new IOS available to upgrade, you can only get upgrade due to the requirement)  - Try reboot the switch (in the maintenance window, see if that fix the issue)

[PKH1994]

hello,

 HULC processes using high utilization how can i avoid or any change in configuration for that ?? 

Switch-1#sh proc cpu sor
CPU utilization for five seconds: 7%/0%; one minute: 8%; five minutes: 8%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
96 107877 7414381 14 0.79% 0.25% 0.22% 0 Hulc Storm Contr
105 19130200 35123844 544 0.79% 0.69% 0.77% 0 Hulc LED Process
51 744913 78658429 9 0.31% 0.52% 0.56% 0 Fifo Error Detec
50 138599 59035729 2 0.31% 0.25% 0.24% 0 RedEarth Rx Mana
65 57501 44298686 1 0.15% 0.07% 0.06% 0 HLFM address ret
78 73424 14563071 5 0.15% 0.12% 0.10% 0 hpm main process
132 1727 1494079 1 0.15% 0.01% 0.00% 0 UDLD
63 103641 44292705 2 0.15% 0.23% 0.17% 0 HLFM address lea
31 23164 1483975 15 0.15% 0.05% 0.01% 0 Per-Second Jobs
9 28681 172177 166 0.00% 0.00% 0.00% 0 ARP Input
10 0 1 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT
11 0 2 0 0.00% 0.00% 0.00% 0 AAA high-capacit
13 17 5 3400 0.00% 0.00% 0.00% 0 Entity MIB API
8 0 1 0 0.00% 0.00% 0.00% 0 Crash writer
15 66 24900 2 0.00% 0.00% 0.00% 0 IPC Dynamic Cach
7 0 1 0 0.00% 0.00% 0.00% 0 HRPC asic-stats
6 8 2 4000 0.00% 0.00% 0.00% 0 Timers
12 0 1 0 0.00% 0.00% 0.00% 0 Policy Manager

[Leo Laohoo]

86% in 72 hours?  That is nothing.  

[Georg Pauwen]

Hello,

which IPCOM access point models do you have (e.g. 515/625) ? Most of them have speeds upwards of 300Mbps, you are connected to FastEthernet ports, so chances are your switch does not have the capacity to handle 26 of these access points.

Can you post the output of some of the interfaces:

show interfaces interface FastEthernet0/20

show interfaces interface FastEthernet0/34

[PKH1994]

IPCOM model --AP255 (AP255V2.0)

Switch-1#show interfaces fa0/20
FastEthernet0/20 is up, line protocol is up (connected)
Hardware is Fast Ethernet, address is 6c50.4d98.2494 (bia 6c50.4d98.2494)
Description: ***IPCOM-AP255****
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, media type is 10/100BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 18:56:02, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 95492515
Queueing strategy: fifo
Output queue: 0/0 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
2287976 packets input, 844640334 bytes, 0 no buffer
Received 8084 broadcasts (6174 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 6174 multicast, 0 pause input
0 input packets with dribble condition detected
20173587 packets output, 11372398869 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out

Switch-1#show interfaces fa0/15
FastEthernet0/15 is up, line protocol is up (connected)
Hardware is Fast Ethernet, address is 6c50.4d97.248f (bia 6c50.4d97.248f)
Description: ***IPCOM-AP255****
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, media type is 10/100BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 23:04:40, output 00:00:07, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 95488902
Queueing strategy: fifo
Output queue: 0/0 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
29216939 packets input, 10078826492 bytes, 0 no buffer
Received 312928 broadcasts (222275 multicasts)
391 runts, 0 giants, 0 throttles
391 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 222275 multicast, 0 pause input
0 input packets with dribble condition detected
92981422 packets output, 68661283068 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out

Fiber Uplink port:

Switch-1# show interfaces gigabitEthernet 0/2
GigabitEthernet0/2 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is a418.7557.f9b2 (bia a418.7557.f9b2)
Description: ***UplinkFrom-l3-sw***
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseLX SFP
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:01, output 1w2d, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 3000 bits/sec, 3 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
288993784 packets input, 332396930171 bytes, 0 no buffer
Received 6542023 broadcasts (3628138 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 3628138 multicast, 0 pause input
0 input packets with dribble condition detected
165805659 packets output, 45215680169 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out