3850 4 stack switch CPU high utilization with dhcp snooping enabled

[MAHLESSC]

Hello,

I have an issue with a 4 stack 3650, in all of our devices we use dhcp snooping.

Vlan 1 is for clients and the issue is only with that Vlan and I cannot migrate it to a different vlan.

#sh run | i ip dhcp snooping
ip dhcp snooping vlan 1,21,56,601,603-604
ip dhcp snooping

A#sh processes cpu | e 0.00%
CPU utilization for five seconds: 92%/21%; one minute: 54%; five minutes: 26%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
37 421801 1564281 269 0.15% 0.15% 0.16% 0 ARP Input
52 1054595 2199181 479 0.39% 0.23% 0.12% 0 ARP Snoop
63 6266 671 9338 0.23% 0.10% 0.63% 2 SSH Process
74 1020830 477597 2137 0.31% 0.22% 0.21% 0 Net Background
81 17745149 2928948 6058 1.27% 0.81% 0.81% 0 IOSD ipc task
99 1399844 3303225 423 0.31% 0.22% 0.18% 0 cpf_process_tpQ
114 5611815 158540 35396 0.95% 2.10% 2.84% 0 Crimson flush tr
119 143212 2532616 56 0.07% 0.05% 0.06% 0 100ms check
126 4437280 3849041 1152 1.35% 0.73% 0.31% 0 IOSXE-RP Punt Se
153 3497540 473854 7381 0.87% 0.80% 0.78% 0 PLFM-MGR IPC pro
154 508142 102303 4967 0.15% 0.12% 0.11% 0 FEP background p
158 150289 102303 1469 0.07% 0.03% 0.02% 0 Thermal backgrou
165 171434 333560 513 0.15% 0.03% 0.02% 0 Inline Power
174 3097153 2226579 1390 1.03% 0.49% 0.40% 0 MATM RP Shim Pro
204 220530 2521591 87 0.07% 0.07% 0.09% 0 VRRS Main thread
219 558817 358090 1560 0.07% 0.12% 0.13% 0 CDP Protocol
247 7308139 2858195 2556 2.07% 2.28% 2.50% 0 Spanning Tree
249 1365095 4071413 335 0.39% 0.32% 0.29% 0 UDLD
333 135158982 2202348 61370 36.31% 18.74% 5.31% 0 SISF Switcher Th
334 94969888 2302925 41239 23.17% 12.57% 3.66% 0 SISF Main Thread
349 2549485 78674 32405 1.27% 0.64% 0.62% 0 QoS stats proces
354 219135 2521452 86 0.07% 0.08% 0.09% 0 MMA DB TIMER
384 221116 2521507 87 0.07% 0.08% 0.10% 0 MMA DP TIMER
427 250030 1435317 174 0.07% 0.07% 0.07% 0 PM Callback
552 744418 642787 1158 0.39% 0.20% 0.16% 0 LLDP Protocol

If I remove the Vlan 1 from the snooping the cpu goes to normal

sh run | i ip dhcp snooping
ip dhcp snooping vlan 21,56,601,603-604

A#sh processes cpu | e 0.00%
CPU utilization for five seconds: 15%/1%; one minute: 69%; five minutes: 39%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
37 421904 1564734 269 0.23% 0.13% 0.15% 0 ARP Input
63 6386 863 7399 0.63% 0.15% 0.51% 2 SSH Process
74 1021028 477656 2137 0.31% 0.24% 0.22% 0 Net Background
81 17745858 2929788 6057 1.27% 0.86% 0.82% 0 IOSD ipc task
99 1400088 3303890 423 0.23% 0.27% 0.21% 0 cpf_process_tpQ
114 5613449 158585 35397 0.47% 2.31% 2.66% 0 Crimson flush tr
126 4438123 3849982 1152 0.23% 0.91% 0.49% 0 IOSXE-RP Punt Se
130 204019 3554781 57 0.39% 0.12% 0.19% 0 L2 LISP Punt Pro
153 3498290 473956 7381 0.87% 0.87% 0.81% 0 PLFM-MGR IPC pro
158 150317 102323 1469 0.07% 0.03% 0.02% 0 Thermal backgrou
165 171463 333594 513 0.07% 0.03% 0.02% 0 Inline Power
174 3097761 2227114 1390 0.71% 0.68% 0.49% 0 MATM RP Shim Pro
204 220571 2522249 87 0.07% 0.05% 0.07% 0 VRRS Main thread
219 558924 358163 1560 0.15% 0.12% 0.13% 0 CDP Protocol
225 165194 2834200 58 0.23% 0.08% 0.11% 0 IPAM Manager
229 189058 1441719 131 0.07% 0.04% 0.05% 0 IP Input
247 7309774 2859265 2556 2.40% 2.08% 2.37% 0 Spanning Tree
249 1365372 4072234 335 0.07% 0.32% 0.30% 0 UDLD
333 135183329 2202755 61370 0.07% 25.49% 11.24% 0 SISF Switcher Th
334 94986617 2303400 41237 1.27% 17.38% 7.72% 0 SISF Main Thread
349 2550009 78690 32405 1.20% 0.66% 0.63% 0 QoS stats proces
354 219201 2522109 86 0.15% 0.08% 0.08% 0 MMA DB TIMER
384 221174 2522164 87 0.07% 0.06% 0.08% 0 MMA DP TIMER
385 153873 6519255 23 0.07% 0.06% 0.22% 0 MMON MENG
545 313056 586877 533 0.23% 0.12% 0.10% 0 IP SNMP
546 172651 173885 992 0.15% 0.07% 0.06% 0 PDU DISPATCHER
547 531071 175800 3020 0.31% 0.52% 0.28% 0 SNMP ENGINE
552 744567 642917 1158 0.15% 0.18% 0.16% 0 LLDP Protocol

IOS version is Fuji 19.9.4

And this is the config for the access ports

interface GigabitEthernet1/0/1
switchport mode access
switchport nonegotiate
switchport voice vlan 21
switchport port-security maximum 2
switchport port-security violation restrict
switchport port-security aging time 2
switchport port-security aging type inactivity
switchport port-security
no logging event link-status
no snmp trap link-status
auto qos trust dscp
no mdix auto
spanning-tree portfast
spanning-tree bpduguard enable
service-policy input AutoQos-4.0-Trust-Dscp-Input-Policy
service-policy output AutoQos-4.0-Output-Policy
ip dhcp snooping limit rate 20

I dont know why the issue is only on this stack, as I mentioned before we had several switches using the same config and all of the are working good

Even shutting down all the ports on the stack the issue continues

Thanks

[Leo Laohoo]

Post the complete output to the following output:

show processes cpu platform sorted location switch <1 to 16> r0

Post only the 1st page.

[MAHLESSC]

, this is the output for the first switch, on the others the cpu utilization shows a s 8 to 14 %

show processes cpu platform sorted location switch 1 r0
CPU utilization for five seconds: 56%, one minute: 57%, five minutes: 58%
Core 0: CPU utilization for five seconds: 55%, one minute: 57%, five minutes: 57%
Core 1: CPU utilization for five seconds: 60%, one minute: 58%, five minutes: 58%
Core 2: CPU utilization for five seconds: 59%, one minute: 57%, five minutes: 58%
Core 3: CPU utilization for five seconds: 61%, one minute: 57%, five minutes: 57%
Pid PPid 5Sec 1Min 5Min Status Size Name
--------------------------------------------------------------------------------
12426 10463 125% 123% 123% S 2031489024 fed main event
12219 8478 89% 90% 90% R 2179035136 linux_iosd-imag
4450 2 8% 7% 7% S 0 lsmpi-xmit
4451 2 7% 7% 7% S 0 lsmpi-rx
12155 11069 4% 3% 3% S 974524416 sif_mgr
28469 27809 2% 1% 1% S 321224704 fman_rp
4449 2 2% 2% 2% S 0 lsmpi-refill
19201 17876 1% 0% 0% S 304893952 fman_fp_image
8113 7693 1% 0% 0% R 52936704 hman
32122 32061 0% 0% 0% S 150339584 nginx
32061 31931 0% 0% 0% S 142942208 nginx
31979 1 0% 0% 0% S 9502720 rotee
31931 1 0% 0% 0% S 5931008 pman.sh
29707 29705 0% 0% 0% S 313520128 python2.7
29705 29579 0% 0% 0% S 5136384 rdope.sh
29627 1 0% 0% 0% S 9502720 rotee
29579 1 0% 0% 0% S 5931008 pman.sh
29434 29273 0% 0% 0% S 53633024 tms
29352 1 0% 0% 0% S 9502720 rotee
29273 6444 0% 0% 0% S 5931008 pman.sh
29055 28757 0% 0% 0% S 242810880 smand
28865 1 0% 0% 0% S 9502720 rotee
28757 6444 0% 0% 0% S 5931008 pman.sh
28688 28371 0% 0% 0% S 55521280 psd
28494 1 0% 0% 0% S 9502720 rotee
28371 6444 0% 0% 0% S 5931008 pman.sh
27941 27541 0% 0% 0% S 249683968 dbm
27918 1 0% 0% 0% S 9502720 rotee
27809 6444 0% 0% 0% S 5931008 pman.sh
27679 1 0% 0% 0% S 9502720 rotee
27633 27288 0% 0% 0% S 42512384 cmm
27541 6444 0% 0% 0% S 5931008 pman.sh
27507 27107 0% 0% 0% S 430022656 cli_agent
27437 1 0% 0% 0% S 9502720 rotee

[Leo Laohoo]

Raise a TAC Case.
This could be CSCvi87866.