Cisco 3650 High CPU utilization

[Andrey Snytko]

Hi All

Help me

Switch 3650 (WS-C3650-24TD)

Cisco IOS Software, IOS-XE Software (CAT3K_CAA-UNIVERSALK9-M), Version 03.07.02E RELEASE SOFTWARE (fc1)

show proc cpu sorted | ex 0.0
Core 0: CPU utilization for five seconds: 12%; one minute: 12%;  five minutes: 12%

Core 1: CPU utilization for five seconds: 3%; one minute: 7%;  five minutes: 39%
Core 2: CPU utilization for five seconds: 7%; one minute: 6%;  five minutes: 6%
Core 3: CPU utilization for five seconds: 94%; one minute: 94%;  five minutes: 60%

PID    Runtime(ms) Invoked  uSecs  5Sec     1Min     5Min     TTY   Process
6032   889844      24282930 618    25.66    26.01    26.02    1088  fed
10871  1151380     39533956 153    3.50     3.11     2.92     34816 iosd
6608   2509855     21312494 35     0.10     0.10     0.10     0     pdsd
10866  1109726     39768503 16     0.10     0.13     0.13     0     wcm

sh processes cpu detailed process fed sorted

Core 0: CPU utilization for five seconds: 14%; one minute: 11%; five minutes: 11%
Core 1: CPU utilization for five seconds: 4%; one minute: 6%; five minutes: 14%
Core 2: CPU utilization for five seconds: 3%; one minute: 4%; five minutes: 5%
Core 3: CPU utilization for five seconds: 97%; one minute: 96%; five minutes: 87%
PID    T C  TID    Runtime(ms) Invoked uSecs  5Sec      1Min     5Min     TTY   Process
                                               (%)       (%)      (%)
6032   L           1336054     2428831 618    25.80     25.39   25.79   1088  fed
6032   L 3  11587  2735180     1654110 0      24.39     23.96   24.26   0     PunjectRx
6032   L 2  11589  975019      4247567 0      0.49      0.51    0.57    0     PunjectTx
6032   L 0  6482   2806155     5034345 0      0.24      0.24    0.24    0     fed-ots-nfl
6032   L 2  6505   2824495     1959187 0      0.24      0.14    0.14    0     IntrDrv
6032   L 0  10444  1676726     2162644 0      0.19      0.25    0.27    0     Xcvr
6032   L 1  6479   17477       7201840 0      0.15      0.21    0.21    0     fed-ots-main
6032   L 0  6473   1354789     5567928 0      0.05      0.03    0.05    1088  fed
6032   L 0  6486   748590      6656099 0      0.05      0.02    0.04    0     Timer
6032   L 3  6032   22970       16952   0      0.00      0.00    0.00    1088  fed
6032   L 0  6474   44680       4877204 0      0.00      0.00    0.00    1088  fed
6032   L 1  6475   2270819     4577663 0      0.00      0.01    0.00    1088  CMI default xdm
6032   L 0  6476   37690       3255183 0      0.00      0.00    0.00    0     fed
6032   L 0  6480   11630       143166  0      0.00      0.00    0.00    0     fed-ots-mo
6032   L 3  6483   0           1270    0      0.00      0.00    0.00    0     fed
6032   L 2  6484   1753360     3005153 0      0.00      0.00    0.00    0     fed
6032   L 0  6485   2378920     4881080 0      0.00      0.04    0.05    0     pdslib thread
6032   L 1  6487   0           11173   0      0.00      0.00    0.00    0     fed
6032   L 1  6544   1700352     4875202 0      0.00      0.00    0.03    0     LED

[Seb Rupik]

Hi there,

Could be related to:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuz57493

Plus the version of IOS-XE you are using has been deferred. I'd recommend at the very least upgrading to 03.07.05E

cheers,

Seb.

[Andrey Snytko]

Thanks.
I try (when will time) upgrade to 03.07.05E.
What do you think about version 16.03.06?

Can be the reason of high CPU that the network goes down and when launch the wireshark I see ip packets which are not addressed to me?

[Seb Rupik]

I suggested 3.7.5 as that is the latest release of the software train you are currently using. The current recommended release for that platform are 16.3.6 and 3.6.8 .

Not sure I understand your second question. Are you running wireshark on a device connected to the switch. Is the switchport a trunk or access port? Configured as a SPAN port?

cheers,

Seb.

[Andrey Snytko]

Device connect to the switch swichport is a access port. But other devices conneted to the other switch (trunk port) also see this trafic. Ports no Configured as a SPAN port

[Georg Pauwen]

Hello,

on a side note, what is the uptime of the switch ('show version' will tell you that) ? If the switch has not bee power cycled for > 1 year, you might want to try and reload it, that might remedy the problem as well...

[Andrey Snytko]

Thanks.

Uptime so big.

sh ver

Switch 02

---------

Switch uptime                      : 1 year, 28 weeks, 3 days, 20 hours, 57 minutes

This switch is a core network. I can't often reload.

[Georg Pauwen]

Hello,

--> Switch uptime : 1 year, 28 weeks, 3 days, 20 hours, 57 minutes

If the problem hasn't occurred before and has just recently begin to pop up, a reload might help...

I would schedule a reload window. Is this actually affecting user network traffic ? 

[Andrey Snytko]

It is very difficult =(