it could be a bug theres a

radams123 · ‎12-05-2015

I have a network without around 40 switches (3560, 2960, 3750x) with a 4510 as my core. Plan is to replace all of the 3560 and 2960 with 3850x. I noticed after I installed a few of the 3850's that I started having some minor, unexplainable network issues. This week, I installed another 3850 in a closet and tied it back to the core via 1GB fiber. About twice a day, the users would call stating the network was down. After investigation, I found that the 3850 was running very high cpu and a reboot would fix it for about 6 hours and it would start again. I replaced the 3850 in question with another 3850 and the same problem came back this morning. I decided to start looking at the other 3850's cpu history aand it appears that they are spiking CPU at the same times. The software version is 03.02.03. Anyone got any ideas. Should I try upgrading the software as they are not mixing well in my network.

Arjun Sarkar · ‎12-05-2015

Hello ,

Please check what is the top talkers at the time og high CPU on 3850. Please execute follwing commands to check the status,

- show process cpu sorted | ex 0.00

- show platform punt client - This is to check if there is any queue is stuck.

PLeasse do refer the troubleshooting guide which is available on CCO.

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/117594-technote-hicpu3850-00.html

Best Regards
Arjun

Please rate if it helps.

Mark Malone · ‎12-08-2015

it could be a bug theres a few for this fed process it runs high on certain ios-xe

https://tools.cisco.com/quickview/bug/CSCuo14511

Heres how you can check certain issues with fed process for 38s

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/117594-technote-hicpu3850-00.html

Mark Malone · ‎12-05-2015

yes the proc cpu sorted willl tell what happening live as Arjun has stated really need that , if you do suspect buggy software move to 3.6.3 thats the safe harbour version least amount of caveats fo that platform

if your not around to capture it apply this it will do it for you and send the results to flash, just chnage it to your local file system below then you can export it

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/116141-trouble-eem-scripts-00.html

event manager scheduler script thread class default number 1 
event manager applet High_CPU
event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.3.1 get-type exact entry-op ge  
    entry-val 50 poll-interval 0.5
  action 0.0 syslog msg "High CPU DETECTED. Please wait - logging Information  
    to file system:high_cpu.txt"
  action 0.1 cli command "enable"
  action 0.2 cli command "show clock | append file system:high_cpu.txt"
  action 1.2 cli command "term length 0"
  action 1.3 cli command "show process cpu sorted | append file system:high_cpu.txt"
  action 1.4 cli command "show log | append file system:high_cpu.txt"
  action 1.5 cli command "show interfaces | append file system:high_cpu.txt"
  action 1.6 cli command "term length 24"

radams123 · ‎12-07-2015

It looks like the fed and iosd process is running high. As stated earlier, I plan to upgrade the software to 3.6.3 to see if it helps but any suggestions would be helpful.

PT-3850#show proc cpu sorted

Core 0: CPU utilization for five seconds: 66%; one minute: 74%; five minutes: 63%

Core 1: CPU utilization for five seconds: 68%; one minute: 68%; five minutes: 56%

Core 2: CPU utilization for five seconds: 74%; one minute: 71%; five minutes: 57%

Core 3: CPU utilization for five seconds: 74%; one minute: 71%; five minutes: 57%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

5716 2820903 27848925 179 41.04 41.02 32.83 1088 fed

9400 2379157 10020564 362 23.70 23.90 19.64 0 iosd

5718 2595175 74753447 1011 3.43 3.48 3.51 0 stack-mgr

6396 3134705 75308221 78 1.97 2.01 1.74 0 pdsd

5717 3869422 34627272 23 0.28 0.15 0.14 0 platform_mgr

6407 357565 7652782 1169 0.14 0.14 0.14 0 cpumemd

9396 2358425 36879904 29 0.09 0.12 0.12 0 wcm

6387 104800 148013 708 0.05 0.01 0.05 0 oom_poll.sh

1 1650 1051 1569 0.00 0.00 0.00 0 init

2 0 125 0 0.00 0.00 0.00 0 kthreadd

3 11840 1012588 11 0.00 0.00 0.00 0 migration/0

4 0 3 0 0.00 0.00 0.00 0 sirq-high/0

5 85792 15095491 29 0.00 0.00 0.00 0 sirq-timer/0

6 0 8 0 0.00 0.00 0.00 0 sirq-net-tx/0

7 20 839 23 0.00 0.00 0.00 0 sirq-net-rx/0

8 720 85614 8 0.00 0.00 0.00 0 sirq-block/0

9 0 8 0 0.00 0.00 0.00 0 sirq-block-iopo

10 0 1152 0 0.00 0.00 0.00 0 sirq-tasklet/0

11 1381980 58561207 23 0.00 0.00 0.00 0 sirq-sched/0

12 2860 59135 48 0.00 0.00 0.00 0 sirq-hrtimer/0

13 2326115 13158949 82 0.00 0.00 0.00 0 sirq-rcu/0

14 12650 1979489 6 0.00 0.00 0.00 0 desched/0

15 42950 3768793 11 0.00 0.00 0.00 0 migration/1

16 0 15 0 0.00 0.00 0.00 0 sirq-high/1

17 284440 14394066 1 0.00 0.02 0.02 0 sirq-timer/1

18 0 19 0 0.00 0.00 0.00 0 sirq-net-tx/1

19 0 6297 0 0.00 0.00 0.00 0 sirq-net-rx/1

20 420 28829 14 0.00 0.00 0.00 0 sirq-block/1

21 0 20 0 0.00 0.00 0.00 0 sirq-block-iopo

22 0 21 0 0.00 0.00 0.00 0 sirq-tasklet/1

23 161430 74587634 2 0.00 0.00 0.00 0 sirq-sched/1

24 0 53383 0 0.00 0.00 0.00 0 sirq-hrtimer/1

25 368810 11908040 3 0.00 0.00 0.00 0 sirq-rcu/1

26 10470 1445360 7 0.00 0.00 0.00 0 desched/1

27 70960 4718895 15 0.00 0.00 0.01 0 migration/2

28 0 27 0 0.00 0.00 0.00 0 sirq-high/2

29 21990 11499905 0 0.00 0.00 0.03 0 sirq-timer/2

30 0 33 0 0.00 0.00 0.00 0 sirq-net-tx/2

31 20 8559 2 0.00 0.00 0.00 0 sirq-net-rx/2

32 120 7306 16 0.00 0.00 0.00 0 sirq-block/2

33 0 32 0 0.00 0.00 0.00 0 sirq-block-iopo

34 0 33 0 0.00 0.00 0.00 0 sirq-tasklet/2

35 11450 69001545 0 0.00 0.00 0.00 0 sirq-sched/2

36 0 43691 0 0.00 0.00 0.00 0 sirq-hrtimer/2

37 33540 10685313 0 0.00 0.00 0.00 0 sirq-rcu/2

38 19090 3277703 5 0.00 0.00 0.00 0 desched/2

39 44670 2636657 16 0.00 0.00 0.00 0 migration/3

40 0 39 0 0.00 0.00 0.00 0 sirq-high/3

41 0 10755851 0 0.00 0.03 0.04 0 sirq-timer/3

42 0 42 0 0.00 0.00 0.00 0 sirq-net-tx/3

43 20 9673 2 0.00 0.00 0.00 0 sirq-net-rx/3

44 30 3411 8 0.00 0.00 0.00 0 sirq-block/3

45 0 44 0 0.00 0.00 0.00 0 sirq-block-iopo

46 0 45 0 0.00 0.00 0.00 0 sirq-tasklet/3

3850 with high cpu