Cat 3750XST high CPU load due to Spanning-tree process

[Pepelatz]

Hi.

I see strange CPU load on my catalyst 3750 XST stack due to STP process:

C3750XST-itc-hp6#sh proc cpu sort | ex 0.0
CPU utilization for five seconds: 66%/8%; one minute: 61%; five minutes: 61%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
243 1668149115 3341481293 0 37.69% 37.56% 37.52% 0 Spanning Tree
176 2272927417 2007186851 1132 3.85% 2.36% 2.34% 0 Hulc LED Process
11 255093563 1671948 152573 3.15% 0.37% 0.23% 0 Licensing Auto U
90 1727198064 387707832 4454 1.92% 1.80% 1.80% 0 RedEarth Tx Mana
261 1775695623 97528303 18207 1.92% 2.07% 2.01% 0 PI MATM Aging Pr
133 657608319 100614775 6535 0.87% 0.81% 0.82% 0 hpm counter proc
89 605883529 536134816 1130 0.70% 0.66% 0.64% 0 RedEarth I2C dri
111 105703424 448577845 235 0.52% 0.19% 0.12% 0 HRPC hlfm reques
113 63347071 97532524 649 0.35% 0.15% 0.11% 0 HLFM aging proce
112 88681489 2237731882 0 0.35% 0.16% 0.12% 0 HLFM address lea
59 80331551 106689164 752 0.35% 0.15% 0.15% 0 Per-Second Jobs
192 191997816 19879258 9658 0.17% 0.16% 0.15% 0 HQM Stack Proces
193 121927431 78709560 1549 0.17% 0.10% 0.10% 0 HRPC qos request
C3750XST-itc-hp6#

Software is c3750e-universalk9npe-mz.150-2.SE4.bin

How can I find the reason of such CPU load level? I see no STP-messages in logs or debugs.

[Reza Sharifi]

Hi,

Check and make sure that STP forwarding/blocking is working properly and there is no loop on the network. Also, make sure no devices like hubs connected to the network and causing loop.

HTH

[Pepelatz]

C3750XST-itc-hp6#sis

Port Name Status Vlan Duplex Speed Type
Gi1/0/1 HP2000(Virt2) connected 401 a-full a-100 10/100/1000BaseTX
Gi1/0/2 connected trunk a-full a-1000 10/100/1000BaseTX
Gi1/0/3 ESXI01_ILO connected 401 a-full a-1000 10/100/1000BaseTX
Gi1/0/4 connected trunk a-full a-1000 10/100/1000BaseTX
Gi1/0/5 ESXI03 connected trunk a-full a-1000 10/100/1000BaseTX
Gi1/0/6 connected trunk a-full a-1000 10/100/1000BaseTX
Gi1/0/7 ESXI04 connected trunk a-full a-1000 10/100/1000BaseTX
Gi1/0/8 connected trunk a-full a-1000 10/100/1000BaseTX
Gi1/0/9 ESXI04 connected trunk a-full a-1000 10/100/1000BaseTX
Gi1/0/10 HP2000(Virt1) connected 401 a-full a-100 10/100/1000BaseTX
Gi1/0/11 ESXITEST connected 401 a-full a-1000 10/100/1000BaseTX
Gi1/0/12 Beta_ILO connected 401 a-full a-1000 10/100/1000BaseTX
Gi1/0/13 ESXI04_ILO connected 401 a-full a-1000 10/100/1000BaseTX
Gi1/0/14 ESXI03 connected trunk a-full a-1000 10/100/1000BaseTX
Gi1/0/15 connected 401 a-full a-1000 10/100/1000BaseTX
Gi1/0/16 ILO connected 401 a-full a-1000 10/100/1000BaseTX
Gi1/0/17 QNArch connected 401 a-full a-1000 10/100/1000BaseTX
Gi1/0/18 QNArch connected 401 a-full a-1000 10/100/1000BaseTX
Gi1/0/19 USB-Concentrator connected 401 a-full a-100 10/100/1000BaseTX
Gi1/0/20 WITSRV_ILO connected 401 a-full a-1000 10/100/1000BaseTX
Gi1/0/21 notconnect 401 auto auto 10/100/1000BaseTX
Gi1/0/22 DLP_ILO connected 401 a-full a-1000 10/100/1000BaseTX
Gi1/0/23 FC-Switches connected 401 a-full a-100 10/100/1000BaseTX
Gi1/0/24 CrownePlazaStorage connected 401 a-half a-10 10/100/1000BaseTX
Gi1/1/1 notconnect 1 auto auto Not Present
Gi1/1/2 notconnect 1 auto auto Not Present
Gi1/1/3 notconnect 1 auto auto Not Present
Gi1/1/4 connected trunk a-full a-1000 1000BaseSX SFP
Gi2/0/1 HP2000(Virt2) connected 401 a-full a-100 10/100/1000BaseTX
Gi2/0/2 connected trunk a-full a-1000 10/100/1000BaseTX
Gi2/0/3 ESXI02_ILO connected 401 a-full a-1000 10/100/1000BaseTX
Gi2/0/4 connected trunk a-full a-1000 10/100/1000BaseTX
Gi2/0/5 ESXI03 connected trunk a-full a-1000 10/100/1000BaseTX
Gi2/0/6 connected trunk a-full a-1000 10/100/1000BaseTX
Gi2/0/7 ESXI04 connected trunk a-full a-1000 10/100/1000BaseTX
Gi2/0/8 connected trunk a-full a-1000 10/100/1000BaseTX
Gi2/0/9 ESXI04 connected trunk a-full a-1000 10/100/1000BaseTX
Gi2/0/10 HP2000(Virt1) connected 401 a-full a-100 10/100/1000BaseTX
Gi2/0/11 Zeus connected trunk a-full a-1000 10/100/1000BaseTX
Gi2/0/12 Zeus_iLO connected 401 a-full a-1000 10/100/1000BaseTX
Gi2/0/13 USB-Concentrator notconnect 401 auto auto 10/100/1000BaseTX
Gi2/0/14 CLUSTER-EXPERIMENT notconnect 401 auto auto 10/100/1000BaseTX
Gi2/0/15 CLUSTER-EXPERIMENT notconnect 401 auto auto 10/100/1000BaseTX
Gi2/0/16 ESXI03 connected trunk a-full a-1000 10/100/1000BaseTX
Gi2/0/17 notconnect 1 auto auto 10/100/1000BaseTX
Gi2/0/18 notconnect 1 auto auto 10/100/1000BaseTX
Gi2/0/19 USB-Concentrator notconnect 401 auto auto 10/100/1000BaseTX
Gi2/0/20 notconnect 1 auto auto 10/100/1000BaseTX
Gi2/0/21 WitSrv connected 401 a-full a-1000 10/100/1000BaseTX
Gi2/0/22 WitSrv-Replication connected 401 a-full a-1000 10/100/1000BaseTX
Gi2/0/23 FC-Switches connected 401 a-full a-100 10/100/1000BaseTX
Gi2/0/24 CrownePlazaStorage connected 401 a-half a-10 10/100/1000BaseTX
Gi2/1/1 notconnect 1 auto auto Not Present
Gi2/1/2 notconnect 1 auto auto Not Present
Gi2/1/3 notconnect 1 auto auto Not Present
Gi2/1/4 connected trunk a-full a-1000 1000BaseSX SFP
Po1 C6506E-cmt1-lev3-3 connected trunk a-full a-1000
Po2 QNArch-bond connected 401 a-full a-1000
Fa0 notconnect routed auto auto 10/100BaseTX
C3750XST-itc-hp6#

I see many devices (ESXI, for example), which is connected to both switches of my stack.

But I see only TWO PortChannel interfaces... maybe this is the reason of loop? 

[Leo Laohoo]

Output shows STP, Hulc and Red Earth.  All of which tells me that there is, possibly, a port that is currently flapping.  Is interface link logging disabled?  

Check the logs.

[Pepelatz]

Hi.

I see no flapping messages.

Seems that link logging is enabled... but how can I check this?

[Julio E. Moisa]

Hi

Additional to other members, I have detected some MAC addresses creating a kind of loop, you can detect the MAC address with more concurrences using this command:

no logging console
logging buffered 128000

debug platform cpu-queues software-fwd-q

It can be used in short intervals of time (2 or 3 seconds). To disable it, you can execute: u all 

Then check the logs: show log

Usually it is generated by problems with the drivers on the computers.

Also you can verify what STP instance is changing periodically: 

show spanning-tree detail and check this info:

Number of topology changes 83 last change occurred 1w2d ago
from GigabitEthernet0/0

Hope it is useful

:-)

[Pepelatz]

C3750XST-itc-hp6#sh spanning-tree detail | i Number of topology changes
Number of topology changes 8909 last change occurred 00:01:22 ago
Number of topology changes 8965 last change occurred 00:01:22 ago
Number of topology changes 9012 last change occurred 00:01:22 ago
Number of topology changes 9036 last change occurred 00:01:22 ago
Number of topology changes 9070 last change occurred 00:01:22 ago
Number of topology changes 9098 last change occurred 00:01:22 ago
Number of topology changes 9979 last change occurred 00:01:22 ago
Number of topology changes 9114 last change occurred 00:01:22 ago
Number of topology changes 9122 last change occurred 00:01:22 ago
Number of topology changes 9149 last change occurred 00:01:22 ago
Number of topology changes 9180 last change occurred 00:01:22 ago
Number of topology changes 4763 last change occurred 22:30:16 ago
Number of topology changes 4776 last change occurred 22:30:16 ago
Number of topology changes 4757 last change occurred 22:30:16 ago
Number of topology changes 5911 last change occurred 22:30:16 ago
Number of topology changes 6032 last change occurred 00:12:46 ago
Number of topology changes 5990 last change occurred 00:12:46 ago
Number of topology changes 5983 last change occurred 22:30:16 ago
Number of topology changes 6009 last change occurred 22:30:16 ago
Number of topology changes 9597 last change occurred 00:01:22 ago
Number of topology changes 9368 last change occurred 00:01:22 ago
Number of topology changes 6637 last change occurred 22:30:16 ago
Number of topology changes 3764 last change occurred 22:30:17 ago
Number of topology changes 9506 last change occurred 00:01:23 ago
Number of topology changes 7209 last change occurred 00:01:23 ago
Number of topology changes 7289 last change occurred 00:01:23 ago
Number of topology changes 7219 last change occurred 00:01:23 ago
Number of topology changes 7240 last change occurred 00:01:23 ago
Number of topology changes 9652 last change occurred 00:01:23 ago
Number of topology changes 9717 last change occurred 00:01:23 ago
Number of topology changes 12 last change occurred 00:01:22 ago
Number of topology changes 3296 last change occurred 4d05h ago
Number of topology changes 794 last change occurred 4d05h ago
Number of topology changes 817 last change occurred 21:22:31 ago
Number of topology changes 777 last change occurred 4d05h ago
Number of topology changes 692 last change occurred 4d05h ago
Number of topology changes 5254 last change occurred 00:01:23 ago
Number of topology changes 7844 last change occurred 00:01:23 ago
Number of topology changes 754 last change occurred 4d05h ago
Number of topology changes 844 last change occurred 4d05h ago
Number of topology changes 840 last change occurred 4d05h ago
Number of topology changes 841 last change occurred 4d05h ago
Number of topology changes 844 last change occurred 4d05h ago
Number of topology changes 838 last change occurred 4d05h ago
C3750XST-itc-hp6#

Seems that I have topology changes...

[Julio E. Moisa]

Hi

Yes, it look like a flapping link or L2 loop, do you have the root configured manually for the VLANs? check using: spanning-tree vlan X root details to identify where is the root for this VLAN. Any recent change on the layer 2 network?

You have many recent changes. 

[Pepelatz]

No, there was no L2 changes in my network.

This is the stack of two 3750 switches. As I see, there is a few devices (servers, for example), which connected to both switches of my stack.

See "show interfaces status" output.

******************************************

C3750XST-itc-hp6#sis

Port Name Status Vlan Duplex Speed Type
Gi1/0/1 HP2000(Virt2) connected 401 a-full a-100 10/100/1000BaseTX
Gi1/0/2 connected trunk a-full a-1000 10/100/1000BaseTX
Gi1/0/3 ESXI01_ILO connected 401 a-full a-1000 10/100/1000BaseTX
Gi1/0/4 connected trunk a-full a-1000 10/100/1000BaseTX
Gi1/0/5 ESXI03 connected trunk a-full a-1000 10/100/1000BaseTX
Gi1/0/6 connected trunk a-full a-1000 10/100/1000BaseTX
Gi1/0/7 ESXI04 connected trunk a-full a-1000 10/100/1000BaseTX
Gi1/0/8 connected trunk a-full a-1000 10/100/1000BaseTX
Gi1/0/9 ESXI04 connected trunk a-full a-1000 10/100/1000BaseTX
Gi1/0/10 HP2000(Virt1) connected 401 a-full a-100 10/100/1000BaseTX
Gi1/0/11 ESXITEST connected 401 a-full a-1000 10/100/1000BaseTX
Gi1/0/12 Beta_ILO connected 401 a-full a-1000 10/100/1000BaseTX
Gi1/0/13 ESXI04_ILO connected 401 a-full a-1000 10/100/1000BaseTX
Gi1/0/14 ESXI03 connected trunk a-full a-1000 10/100/1000BaseTX
Gi1/0/15 connected 401 a-full a-1000 10/100/1000BaseTX
Gi1/0/16 ILO connected 401 a-full a-1000 10/100/1000BaseTX
Gi1/0/17 QNArch connected 401 a-full a-1000 10/100/1000BaseTX
Gi1/0/18 QNArch connected 401 a-full a-1000 10/100/1000BaseTX
Gi1/0/19 USB-Concentrator connected 401 a-full a-100 10/100/1000BaseTX
Gi1/0/20 WITSRV_ILO connected 401 a-full a-1000 10/100/1000BaseTX
Gi1/0/21 notconnect 401 auto auto 10/100/1000BaseTX
Gi1/0/22 DLP_ILO connected 401 a-full a-1000 10/100/1000BaseTX
Gi1/0/23 FC-Switches connected 401 a-full a-100 10/100/1000BaseTX
Gi1/0/24 CrownePlazaStorage connected 401 a-half a-10 10/100/1000BaseTX
Gi1/1/1 notconnect 1 auto auto Not Present
Gi1/1/2 notconnect 1 auto auto Not Present
Gi1/1/3 notconnect 1 auto auto Not Present
Gi1/1/4 connected trunk a-full a-1000 1000BaseSX SFP
Gi2/0/1 HP2000(Virt2) connected 401 a-full a-100 10/100/1000BaseTX
Gi2/0/2 connected trunk a-full a-1000 10/100/1000BaseTX
Gi2/0/3 ESXI02_ILO connected 401 a-full a-1000 10/100/1000BaseTX
Gi2/0/4 connected trunk a-full a-1000 10/100/1000BaseTX
Gi2/0/5 ESXI03 connected trunk a-full a-1000 10/100/1000BaseTX
Gi2/0/6 connected trunk a-full a-1000 10/100/1000BaseTX
Gi2/0/7 ESXI04 connected trunk a-full a-1000 10/100/1000BaseTX
Gi2/0/8 connected trunk a-full a-1000 10/100/1000BaseTX
Gi2/0/9 ESXI04 connected trunk a-full a-1000 10/100/1000BaseTX
Gi2/0/10 HP2000(Virt1) connected 401 a-full a-100 10/100/1000BaseTX
Gi2/0/11 Zeus connected trunk a-full a-1000 10/100/1000BaseTX
Gi2/0/12 Zeus_iLO connected 401 a-full a-1000 10/100/1000BaseTX
Gi2/0/13 USB-Concentrator notconnect 401 auto auto 10/100/1000BaseTX
Gi2/0/14 CLUSTER-EXPERIMENT notconnect 401 auto auto 10/100/1000BaseTX
Gi2/0/15 CLUSTER-EXPERIMENT notconnect 401 auto auto 10/100/1000BaseTX
Gi2/0/16 ESXI03 connected trunk a-full a-1000 10/100/1000BaseTX
Gi2/0/17 notconnect 1 auto auto 10/100/1000BaseTX
Gi2/0/18 notconnect 1 auto auto 10/100/1000BaseTX
Gi2/0/19 USB-Concentrator notconnect 401 auto auto 10/100/1000BaseTX
Gi2/0/20 notconnect 1 auto auto 10/100/1000BaseTX
Gi2/0/21 WitSrv connected 401 a-full a-1000 10/100/1000BaseTX
Gi2/0/22 WitSrv-Replication connected 401 a-full a-1000 10/100/1000BaseTX
Gi2/0/23 FC-Switches connected 401 a-full a-100 10/100/1000BaseTX
Gi2/0/24 CrownePlazaStorage connected 401 a-half a-10 10/100/1000BaseTX
Gi2/1/1 notconnect 1 auto auto Not Present
Gi2/1/2 notconnect 1 auto auto Not Present
Gi2/1/3 notconnect 1 auto auto Not Present
Gi2/1/4 connected trunk a-full a-1000 1000BaseSX SFP
Po1 C6506E-cmt1-lev3-3 connected trunk a-full a-1000
Po2 QNArch-bond connected 401 a-full a-1000
Fa0 notconnect routed auto auto 10/100BaseTX
C3750XST-itc-hp6#

But I see only two PortChannel interfaces and I suspect that L2-loop caused by this biconnected devices. 

Dut why STP can't block excess ports if L2 loop exist?

[paul driver]

Hello

Is this a L3 switch ?
CPU utilization for five seconds: 66%/8% <--- interrupt % is still relatively low , make sure you have stp portfast enabled all all access ports to negate unnecessary STP tcn

Also - Do you have a default route and if so do you specify a next-hop ip or interface? - if the latter change the next-hop to specify a ip address

res
Paul

[Pepelatz]

No, this is L2 switch.