- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-22-2018 07:47 AM - edited 03-08-2019 01:59 PM
Hello,
I need help on understanding the problem we are facing with one of our 3850 stack. The stack is currently supporting 4 switches.
We have a high CPU utilization for the past one week and I am trying to find the reason. I couldn't get any solution so far. The best one I got was rebooting the entire stack.
I have provided below the CPU utilization for all 4 cores its running on. And also the logs I have.
Feb 21 10:51:43.385 Central: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/45, changed state to down
Feb 21 10:53:05.266 Central: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/45, changed state to up
Feb 21 10:53:06.264 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/45, changed state to up
Feb 21 10:53:27.444 Central: %DOT1X-5-FAIL: Authentication failed for client (009c.0206.13a9) on Interface Gi3/0/45 AuditSessionID AC14FFA10000483281A4389C
Feb 21 10:54:20.956 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/45, changed state to down
Feb 21 10:54:21.966 Central: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/45, changed state to down
Feb 21 10:55:38.908 Central: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/45, changed state to up
Feb 21 10:55:39.906 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/45, changed state to up
Feb 21 10:56:00.998 Central: %DOT1X-5-FAIL: Authentication failed for client (009c.0206.13a9) on Interface Gi3/0/45 AuditSessionID AC14FFA10000483381A69074
Feb 21 11:11:13.485 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to down
Feb 21 11:11:14.499 Central: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to down
Feb 21 11:11:21.740 Central: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to up
Feb 21 11:11:22.740 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to up
Feb 21 11:11:35.017 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to down
Feb 21 11:11:36.015 Central: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to down
Feb 21 11:11:39.109 Central: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to up
Feb 21 11:11:40.111 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to up
Feb 21 11:11:57.764 Central: %DOT1X-5-FAIL: Authentication failed for client (34e6.d70a.745f) on Interface Gi1/0/45 AuditSessionID AC14FFA10000483481B529E0
Feb 21 11:11:58.178 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 21 11:11:58.178 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 21 11:11:58.178 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 21 11:11:58.178 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 21 11:11:58.178 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 21 11:11:58.178 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 21 11:11:58.178 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 21 11:11:58.178 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 21 11:11:58.178 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 21 12:04:38.105 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/15, changed state to down
Feb 21 12:04:39.109 Central: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/15, changed state to down
Feb 21 12:23:54.454 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/45, changed state to down
Feb 21 12:23:55.465 Central: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/45, changed state to down
Feb 21 12:25:20.854 Central: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/45, changed state to up
Feb 21 12:25:21.860 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/45, changed state to up
Feb 21 12:25:40.468 Central: %DOT1X-5-FAIL: Authentication failed for client (009c.0206.13a9) on Interface Gi3/0/45 AuditSessionID AC14FFA10000483581F8A616
Feb 21 12:58:17.604 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/40, changed state to down
Feb 21 12:58:18.608 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/40, changed state to down
Feb 21 12:58:26.195 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/40, changed state to up
Feb 21 12:58:27.194 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/40, changed state to up
Feb 21 12:58:34.974 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/40, changed state to down
Feb 21 12:58:36.976 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/40, changed state to up
Feb 21 12:58:57.157 Central: %DOT1X-5-FAIL: Authentication failed for client (0030.c15e.b980) on Interface Gi2/0/40 AuditSessionID AC14FFA10000483782171D9E
Feb 21 14:56:03.675 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/19, changed state to down
Feb 21 14:56:04.676 Central: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/19, changed state to down
Feb 21 16:19:34.198 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/7, changed state to down
Feb 21 16:19:35.208 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/7, changed state to down
Feb 21 16:19:39.377 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/7, changed state to up
Feb 21 16:19:40.379 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/7, changed state to up
Feb 21 16:25:07.438 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/38, changed state to down
Feb 21 16:25:08.439 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/38, changed state to down
Feb 21 16:25:11.127 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/38, changed state to up
Feb 21 16:25:12.126 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/38, changed state to up
Feb 21 16:25:45.796 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/38, changed state to down
Feb 21 16:25:46.798 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/38, changed state to down
Feb 21 16:51:31.824 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/33, changed state to down
Feb 21 16:51:32.825 Central: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/33, changed state to down
Feb 22 03:04:53.506 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/14, changed state to down
Feb 22 03:04:54.518 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/14, changed state to down
Feb 22 03:05:01.208 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/14, changed state to up
Feb 22 03:05:02.210 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/14, changed state to up
Feb 22 03:17:25.455 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/14, changed state to down
Feb 22 03:17:26.454 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/14, changed state to down
Feb 22 03:17:33.189 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/14, changed state to up
Feb 22 03:17:34.189 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/14, changed state to up
Feb 22 03:19:18.285 Central: %DOT1X-5-FAIL: Authentication failed for client (9c93.4e5c.9aa2) on Interface Gi2/0/14 AuditSessionID AC14FFA100004846852ACA9E
Feb 22 06:25:53.567 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/41, changed state to down
Feb 22 06:25:56.685 Central: %LINK-3-UPDOWN: Interface GigabitEthernet4/0/41, changed state to down
Feb 22 06:25:59.887 Central: %LINK-3-UPDOWN: Interface GigabitEthernet4/0/41, changed state to up
Feb 22 06:26:00.888 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/41, changed state to up
Feb 22 06:26:24.076 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/41, changed state to down
Feb 22 06:26:25.086 Central: %LINK-3-UPDOWN: Interface GigabitEthernet4/0/41, changed state to down
Feb 22 06:26:28.258 Central: %LINK-3-UPDOWN: Interface GigabitEthernet4/0/41, changed state to up
Feb 22 06:26:29.256 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/41, changed state to up
Feb 22 06:50:30.516 Central: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/19, changed state to up
Feb 22 06:50:33.067 Central: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/19, changed state to down
Feb 22 06:50:36.216 Central: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/19, changed state to up
Feb 22 06:50:37.214 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/19, changed state to up
Feb 22 06:51:13.149 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/19, changed state to down
Feb 22 06:51:14.144 Central: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/19, changed state to down
Feb 22 06:51:17.276 Central: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/19, changed state to up
Feb 22 06:51:18.277 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/19, changed state to up
Feb 22 07:05:48.257 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/7, changed state to down
Feb 22 07:05:51.385 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/7, changed state to down
Feb 22 07:05:54.680 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/7, changed state to up
Feb 22 07:05:55.678 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/7, changed state to up
Feb 22 07:06:33.215 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/7, changed state to down
Feb 22 07:06:34.225 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/7, changed state to down
Feb 22 07:06:37.387 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/7, changed state to up
Feb 22 07:06:38.384 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/7, changed state to up
Feb 22 07:06:56.236 Central: %DOT1X-5-FAIL: Authentication failed for client (ecf4.bb4d.aebc) on Interface Gi2/0/7 AuditSessionID AC14FFA10000484A85FB3206
Feb 22 07:06:56.506 Central: %IOSXE-5-PLATFORM: 2 process fed: Logging is not supported for Client/Group ACL
Feb 22 07:06:56.506 Central: %IOSXE-5-PLATFORM: 2 process fed: Logging is not supported for Client/Group ACL
Feb 22 07:06:56.506 Central: %IOSXE-5-PLATFORM: 2 process fed: Logging is not supported for Client/Group ACL
Feb 22 07:06:56.506 Central: %IOSXE-5-PLATFORM: 2 process fed: Logging is not supported for Client/Group ACL
Feb 22 07:06:56.506 Central: %IOSXE-5-PLATFORM: 2 process fed: Logging is not supported for Client/Group ACL
Feb 22 07:06:56.506 Central: %IOSXE-5-PLATFORM: 2 process fed: Logging is not supported for Client/Group ACL
Feb 22 07:06:56.506 Central: %IOSXE-5-PLATFORM: 2 process fed: Logging is not supported for Client/Group ACL
Feb 22 07:06:56.506 Central: %IOSXE-5-PLATFORM: 2 process fed: Logging is not supported for Client/Group ACL
Feb 22 07:06:56.506 Central: %IOSXE-5-PLATFORM: 2 process fed: Logging is not supported for Client/Group ACL
Feb 22 07:20:25.847 Central: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/15, changed state to up
Feb 22 07:20:26.846 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/15, changed state to up
Feb 22 07:20:43.426 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/15, changed state to down
Feb 22 07:20:44.428 Central: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/15, changed state to down
Feb 22 07:20:47.505 Central: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/15, changed state to up
Feb 22 07:20:48.510 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/15, changed state to up
Feb 22 08:26:52.275 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/38, changed state to up
Feb 22 08:26:53.280 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/38, changed state to up
Feb 22 08:27:26.394 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/38, changed state to down
Feb 22 08:27:27.398 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/38, changed state to down
Feb 22 08:27:30.644 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/38, changed state to up
Feb 22 08:27:31.648 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/38, changed state to up
Feb 22 09:01:49.866 Central: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/33, changed state to up
Feb 22 09:01:50.867 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/33, changed state to up
Feb 22 09:02:11.567 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/33, changed state to down
Feb 22 09:02:12.565 Central: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/33, changed state to down
Feb 22 09:02:19.786 Central: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/33, changed state to up
Feb 22 09:02:20.788 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/33, changed state to up
Feb 22 09:02:22.016 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/33, changed state to down
Feb 22 09:02:23.015 Central: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/33, changed state to down
Feb 22 09:02:26.194 Central: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/33, changed state to up
Feb 22 09:02:27.198 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/33, changed state to up
Feb 22 09:03:32.608 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/33, changed state to down
Feb 22 09:03:33.608 Central: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/33, changed state to down
Feb 22 09:03:36.767 Central: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/33, changed state to up
Feb 22 09:03:37.773 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/33, changed state to up
Feb 22 09:03:55.747 Central: %DOT1X-5-FAIL: Authentication failed for client (f01f.af21.905e) on Interface Gi1/0/33 AuditSessionID AC14FFA10000485186664DF2
Feb 22 09:03:55.914 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 22 09:03:55.914 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 22 09:03:55.914 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 22 09:03:55.914 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 22 09:03:55.915 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 22 09:03:55.915 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 22 09:03:55.915 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 22 09:03:55.915 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 22 09:03:55.915 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Some of the logs are repeating at irregular interval. is this something to do with this high CPU utilization. Couple the cores are at 100% almost all the time.
Feb 22 09:03:55.914 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 22 09:03:55.914 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 22 09:03:55.914 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 22 09:03:55.914 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 22 09:03:55.915 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 22 09:03:55.915 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 22 09:03:55.915 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 22 09:03:55.915 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 22 09:03:55.915 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Any help on this issue is much appreciated.
Solved! Go to Solution.
- Labels:
-
Other Switching
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-22-2018 08:09 AM
actually there in the link i posted on how to tshoot high cpu on a 3850 , and also stk manager
- Forwarding Engine Driver (FED): This is the heart of the Cisco Catalyst 3850 Series Switch and is responsible for all hardware programming/forwarding.
- IOSd: This is the Cisco IOS daemon that runs on the Linux kernel. It is run as a software process within the kernel.
This is what i would do , take the show tech run it through Cisco tool cli analyzer and see if you get a bug id back or if your completely over utilizing the switch
I would also roll the switch back rather than upgrade to the more stable ios-xe version 3.6.6E and see if that fixes the issue
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-22-2018 11:20 AM
@avinash.bolla wrote:
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
5881 3303778 46124849 472 49.41 49.35 49.26 0 stack-mgr
13631 3774917 28728769 167 2.01 2.66 2.69 34817 iosd
There are several bugs involving high CPUs and "stack-mgr" and "iosd" processes and CSCuo14511 is one of them.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-22-2018 07:52 AM
sh proc cpu sorted | exec 0.00
https://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/117594-technote-hicpu3850-00.html
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-22-2018 07:56 AM
This is the output for "sh proc cpu sorted | exclude 0.00"
Core 0: CPU utilization for five seconds: 10%; one minute: 45%; five minutes: 46%
Core 1: CPU utilization for five seconds: 99%; one minute: 66%; five minutes: 56%
Core 2: CPU utilization for five seconds: 99%; one minute: 95%; five minutes: 97%
Core 3: CPU utilization for five seconds: 6%; one minute: 15%; five minutes: 21%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
5881 3303778 46124849 472 49.41 49.35 49.26 0 stack-mgr
13631 3774917 28728769 167 2.01 2.66 2.69 34817 iosd
5879 1258585 29330092 62 1.62 2.06 2.09 1088 fed
6484 2399245 18480507 238 0.54 0.59 0.59 0 ffm
13626 4118936 23572236 12 0.20 0.14 0.12 0 wcm
5880 2146464 30879302 14 0.15 0.08 0.09 0 platform_mgr
5608 1464328 90388706 158 0.05 0.01 0.04 0 system_mgr
6469 28782 20308219 21 0.05 0.02 0.05 0 obfld
6487 2055026 94832542 972 0.05 0.05 0.05 0 cpumemd
As you can see couple of cores are always at near 100%.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-22-2018 08:00 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-22-2018 08:05 AM
Currently, the version its running is "IOS-XE Software, Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 03.07.01E".
Can you please describe, which one are the platform and forwarder processes again?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-22-2018 08:09 AM
actually there in the link i posted on how to tshoot high cpu on a 3850 , and also stk manager
- Forwarding Engine Driver (FED): This is the heart of the Cisco Catalyst 3850 Series Switch and is responsible for all hardware programming/forwarding.
- IOSd: This is the Cisco IOS daemon that runs on the Linux kernel. It is run as a software process within the kernel.
This is what i would do , take the show tech run it through Cisco tool cli analyzer and see if you get a bug id back or if your completely over utilizing the switch
I would also roll the switch back rather than upgrade to the more stable ios-xe version 3.6.6E and see if that fixes the issue
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-22-2018 08:13 AM
Thank you very much for your quick response. Will try the those options and will update here.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-22-2018 11:20 AM
@avinash.bolla wrote:
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
5881 3303778 46124849 472 49.41 49.35 49.26 0 stack-mgr
13631 3774917 28728769 167 2.01 2.66 2.69 34817 iosd
There are several bugs involving high CPUs and "stack-mgr" and "iosd" processes and CSCuo14511 is one of them.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-22-2018 11:27 AM
right, major problem here is that this switch stack is the distribution switch for most of the main building. But luckily this building is not 24/7.
So a reboot or os upgrade or downgrade would be possible.
It's interesting, the only real problem i've had with 3850s before is that crazy ip device-tracking bug that messes up VMware (or that VMware messes up, depending on who you ask!)
otherwise they have been pretty rock solid.
Hi Avi.
"It's probably NOT the network."
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-22-2018 11:28 AM
Do you have IPv6 enabled?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-22-2018 12:03 PM
No, we didn't enable IPV6. And also, I analysed the "Show Tech-support" in Cisco CLI analyzer and the result was a bug "CSCuo14511" for "stack-mgr" high CPU utilization.
We are going to upgrade the stack to rectify this.
