C3850 high cpu with strange log

[tianwen.zhao]

Hello guys,

Our C3850 switches' CPU high frequent. 

#show version
Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 16 WS-C3850-12XS 16.6.5 CAT3K_CAA-UNIVERSALK9 BUNDLE
2 32 WS-C3850-24S 16.6.5 CAT3K_CAA-UNIVERSALK9 BUNDLE
3 56 WS-C3850-48T 16.6.5 CAT3K_CAA-UNIVERSALK9 BUNDLE

C3850#sho processes cpu sorted
CPU utilization for five seconds: 97%/39%; one minute: 79%; five minutes: 48%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
320 19034620 84249648 225 37.65% 27.86% 9.94% 0 SISF Switcher Th
441 17494771 132071721 132 9.67% 7.32% 3.33% 0 SISF Main Thread
233 107421511 156391654 686 7.92% 7.65% 7.43% 0 Spanning Tree
117 35755203 719432257 49 2.29% 2.25% 1.97% 0 IOSXE-RP Punt Se
35 13504890 53040175 254 1.98% 1.30% 1.21% 0 ARP Input
53 11100011 65588362 169 1.42% 0.93% 0.91% 0 ARP Snoop
209 4940552 27356792 180 0.47% 0.45% 0.53% 0 IP Input

There is a phenomenon that When the log "%PLATFORM_MATM-4-QUEUE_OVERLIMIT: MATM dropped learn cache batch messages as queue limit has reached."  show up, The CPU will high suddenly last about few minutes then recover to normal

I saw the process of  "SISF Switcher Th " occupy a part of cpu ,and I search a bug CSCvk32439  similar to this

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvk32439/?rfs=iqvred

But the platform is C2960-X not C3850,How can I fix this ?

Thank you for any answers

[balaji.bandi]

Do you have any recent upgrades done ? what is your QoS configuration.

as per the resolved issue, your version should have resolved as per release document.

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3850/software/release/16-6/release_notes/ol-16-6-3850.html

[tianwen.zhao]

Thank you for your reply.
Yeah , We have upgraded the switches from 16.3.6 to 16.6.5 recently.
Our QOS configuration is system default QOS .
Here are our COPP conf:
control-plane
service-policy input system-cpp-policy
!
policy-map system-cpp-policy
class system-cpp-police-data
police rate 200 pps
class system-cpp-police-routing-control
police rate 1800 pps
class system-cpp-police-control-low-priority
class system-cpp-police-wireless-priority1
class system-cpp-police-wireless-priority2
class system-cpp-police-wireless-priority3-4-5
!

class-map match-any system-cpp-police-data
description ICMP redirect, ICMP_GEN and BROADCAST
!
class-map match-any system-cpp-police-routing-control
description Routing control
!

Thanks

[Leo Laohoo]

CSCvg26012, CSCvk32439

[tianwen.zhao]

Thank you for your reply.
As I said, CSCvk32439 is for C2960-X platform not C3850.
And CSCvg26012 ,I didn't see any policy about DT-PROGRAMMATIC or WL-DEV-TRACK-DHCP . As the SISF is used implicitly by DHCP snooping , How about to disable the DHCP snooping ? Is it get better?

Thanks

[Mark Malone]

no ip dhcp snooping is how you disable it globally
I would run the show tech through Cisco analyzer it may give you exact bug id , if its unknown for your version you should TAC it
If you dont want to TAC just upgrade , you could upgrade to .7 its been stable for us on our 3850s , that may fix it too

[CoreyH]

We're having same issue on a 3850-12XS running 16.12.08 - I attempted disabling dhcp snooping but cpu still hovering 88-99% - we are dropping pings as this thing either can't handle the load or something is buggy in this OS.

CPU utilization for five seconds: 97%/11%; one minute: 90%; five minutes: 89%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
338 1862090940 1467058749 1269 37.60% 33.99% 33.11% 0 SISF Switcher Th
339 1864621743 1405946580 1326 30.25% 33.23% 33.01% 0 SISF Main Thread

[Leo Laohoo]

Post the complete output to the following commands: 

sh platform resource
sh platform software status con brief