3850 High CPU

p0wershell · ‎11-25-2014

Hello

don't know why stacked 3850 have trouble with high cpu

Core 0: CPU utilization for five seconds: 53%; one minute: 44%; five minutes: 44%
Core 1: CPU utilization for five seconds: 45%; one minute: 46%; five minutes: 46%
Core 2: CPU utilization for five seconds: 57%; one minute: 49%; five minutes: 50%
Core 3: CPU utilization for five seconds: 30%; one minute: 50%; five minutes: 50%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
10165 948492 3761400 1394 23.92 24.20 24.03 0 iosd
5686 3884630 11025912 352 20.83 21.05 21.22 1088 fed
6222 327240 5317217 61 1.47 1.44 1.42 0 pdsd

flap FED

Core 0: CPU utilization for five seconds: 50%; one minute: 45%; five minutes: 44%
Core 1: CPU utilization for five seconds: 52%; one minute: 47%; five minutes: 46%
Core 2: CPU utilization for five seconds: 48%; one minute: 48%; five minutes: 50%
Core 3: CPU utilization for five seconds: 54%; one minute: 50%; five minutes: 50%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
10165 1009232 3769705 1407 24.13 24.19 24.06 0 iosd
6222 330950 5371545 61 1.42 1.42 1.42 0 pdsd
5688 62210 1524214 40 0.19 0.14 0.14 0 stack-mgr

mb this affected? all show platform punt statistics port-asic 0 cpuq -1 direction rx normal only this have trouble

RX (ASIC2CPU) Stats (asic 0 qn 20 lqn 20):
RXQ 20: CPU_Q_MCAST_END_STATION_SERVICE
----------------------------------------
Packets received from ASIC : 3024821
Send to IOSd total attempts : 3024821
Send to IOSd failed count : 16979
RX suspend count : 16979
RX unsuspend count : 16978
RX unsuspend send count : 16978
RX unsuspend send failed count : 0
RX dropped count : 0
RX conversion failure dropped : 0
RX pkt_hdr allocation failure : 0
RX INTACK count : 188654
RX packets dq'd after intack : 2870
Active RxQ event : 549316
RX spurious interrupt : 11643

Leo Laohoo · ‎11-25-2014

Can you please post the output to the command "sh proc cpu sorted | ex 0.00"?

p0wershell · ‎11-25-2014

thx for you reply

sure

Core 0: CPU utilization for five seconds: 40%; one minute: 57%; five minutes: 54%
Core 1: CPU utilization for five seconds: 63%; one minute: 43%; five minutes: 45%
Core 2: CPU utilization for five seconds: 41%; one minute: 48%; five minutes: 50%
Core 3: CPU utilization for five seconds: 52%; one minute: 51%; five minutes: 50%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
5686 2784332 17452456 405 23.95 23.98 23.99 1088 fed
10165 600685 4387950 2094 23.63 24.04 24.02 0 iosd
6222 561880 8848159 63 1.13 1.54 1.56 0 pdsd
5688 80090 1940508 41 0.38 0.20 0.16 0 stack-mgr
10161 135540 7600595 17 0.19 0.16 0.16 0 wcm
6233 29460 44386 663 0.05 0.04 0.05 0 cpumemd

Leo Laohoo · ‎11-25-2014

IOS daemon?

What entries do you see in your "sh logs"?

p0wershell · ‎11-25-2014

Cisco IOS Software, IOS-XE Software, Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 03.03.03SE RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.
Compiled Sun 27-Apr-14 18:33 by prod_rel_team

sh log

igabitEthernet1/0/6, changed state to down
*Nov 24 21:57:06.736: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/6, changed state to up
*Nov 24 21:57:07.735: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/6, changed state to up
*Nov 24 23:05:15.098: %SW_MATM-4-MACFLAP_NOTIF: Host 90fd.613c.4a0a in vlan 777 is flapping between port Po24 and port Po23
*Nov 24 23:05:15.194: %SW_MATM-4-MACFLAP_NOTIF: Host 90fd.613c.4a0a in vlan 777 is flapping between port Po24 and port Po23
*Nov 24 23:16:38.746: %SW_MATM-4-MACFLAP_NOTIF: Host dc2b.61b1.5df0 in vlan 777 is flapping between port Gi1/0/12 and port Gi1/0/19
*Nov 24 23:16:38.845: %SW_MATM-4-MACFLAP_NOTIF: Host dc2b.61b1.5df0 in vlan 777 is flapping between port Gi1/0/12 and port Gi1/0/19
*Nov 24 23:20:54.328: %SW_MATM-4-MACFLAP_NOTIF: Host f025.b74e.f34d in vlan 777 is flapping between port Po24 and port Gi1/0/19
*Nov 24 23:20:54.424: %SW_MATM-4-MACFLAP_NOTIF: Host f025.b74e.f34d in vlan 777 is flapping between port Po24 and port Gi1/0/19
*Nov 24 23:24:20.350: %SW_MATM-4-MACFLAP_NOTIF: Host 6809.2782.17d8 in vlan 777 is flapping between port Gi1/0/19 and port Gi1/0/12
*Nov 24 23:24:20.445: %SW_MATM-4-MACFLAP_NOTIF: Host 6809.2782.17d8 in vlan 777 is flapping between port Gi1/0/19 and port Gi1/0/12
*Nov 24 23:44:13.038: %SW_MATM-4-MACFLAP_NOTIF: Host d8bb.2c63.5714 in vlan 777 is flapping between port Gi1/0/12 and port Gi1/0/19
*Nov 24 23:44:13.133: %SW_MATM-4-MACFLAP_NOTIF: Host d8bb.2c63.5714 in vlan 777 is flapping between port Gi1/0/12 and port Gi1/0/19
*Nov 24 23:46:03.320: %SW_MATM-4-MACFLAP_NOTIF: Host 0008.2246.010e in vlan 777 is flapping between port Gi1/0/12 and port Gi1/0/19
*Nov 24 23:46:03.412: %SW_MATM-4-MACFLAP_NOTIF: Host 0008.2246.010e in vlan 777 is flapping between port Gi1/0/12 and port Gi1/0/19
*Nov 24 23:49:06.799: Configuring snmpv3 USM user, persisting snmpEngineBoots. Please Wait...

*Nov 24 23:49:06.860: Configuring snmpv3 USM user, persisting snmpEngineBoots. Please Wait...
*Nov 24 23:49:08.002: %SYS-5-CONFIG_I: Configured from console by supportcisco on vty0 (10.49.1.10)
*Nov 24 23:55:39.243: %SYS-5-CONFIG_I: Configured from console by supportcisco on vty0 (10.49.1.10)
*Nov 24 23:57:00.564: %SYS-5-CONFIG_I: Configured from console by supportcisco on vty0 (10.49.1.10)
*Nov 24 23:59:50.557: %SYS-5-CONFIG_I: Configured from console by supportcisco on vty0 (10.49.1.10)
*Nov 25 00:00:22.956: %SYS-5-CONFIG_I: Configured from console by supportcisco on vty0 (10.49.1.10)
*Nov 25 00:05:06.968: %SYS-5-CONFIG_I: Configured from console by supportcisco on vty0 (10.49.1.10)
*Nov 25 00:05:21.097: %SYS-5-CONFIG_I: Configured from console by supportcisco on vty0 (10.49.1.10)
*Nov 25 00:06:00.452: %SYS-5-CONFIG_I: Configured from console by supportcisco on vty0 (10.49.1.10)
*Nov 25 00:09:28.830: %SYS-5-CONFIG_I: Configured from console by supportcisco on vty0 (10.49.1.10)
*Nov 25 00:09:59.965: %SW_MATM-4-MACFLAP_NOTIF: Host 6809.2782.17d8 in vlan 777 is flapping between port Po23 and port Po24
*Nov 25 00:10:00.140: %SW_MATM-4-MACFLAP_NOTIF: Host 6809.2782.17d8 in vlan 777 is flapping between port Po23 and port Po24
*Nov 25 00:11:11.712: %SYS-5-CONFIG_I: Configured from console by supportcisco on vty0 (10.49.1.10)
*Nov 25 00:11:15.417: %SYS-5-CONFIG_I: Configured from console by supportcisco on vty0 (10.49.1.10)
*Nov 25 00:15:13.519: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/7, changed state to down
*Nov 25 00:15:13.611: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/13, changed state to down
*Nov 25 00:15:14.534: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/7, changed state to down
*Nov 25 00:15:15.677: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/13, changed state to up
*Nov 25 00:15:16.602: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/7, changed state to up
*Nov 25 00:15:17.528: %SW_MATM-4-MACFLAP_NOTIF: Host dc9b.9c40.250e in vlan 777 is flapping between port Po24 and port Po23
*Nov 25 00:15:17.629: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/7, changed state to up
*Nov 25 00:15:17.629: %SW_MATM-4-MACFLAP_NOTIF: Host dc9b.9c40.250e in vlan 777 is flapping between port Po24 and port Po23
*Nov 25 00:16:02.099: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/7, changed state to down
*Nov 25 00:16:02.190: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/13, changed state to down
*Nov 25 00:16:03.034: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/7, changed state to down
*Nov 25 00:16:03.147: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/13, changed state to down
*Nov 25 00:16:05.583: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/7, changed state to up
*Nov 25 00:16:05.772: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/13, changed state to up
*Nov 25 00:16:06.616: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/7, changed state to up
*Nov 25 00:16:06.800: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/13, changed state to up
*Nov 25 00:16:15.792: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/7, changed state to down
*Nov 25 00:16:16.801: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/7, changed state to down
*Nov 25 00:16:22.902: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/7, changed state to up
*Nov 25 00:16:22.992: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/13, changed state to down
*Nov 25 00:16:23.956: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/7, changed state to up
*Nov 25 00:16:24.047: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/13, changed state to down
*Nov 25 00:16:29.979: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/13, changed state to up
*Nov 25 00:16:30.980: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/13, changed state to up
*Nov 25 00:16:51.019: %SYS-5-CONFIG_I: Configured from console by supportcisco on vty0 (10.49.1.10)
*Nov 25 00:18:09.539: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/7, changed state to down
*Nov 25 00:18:10.006: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/13, changed state to down
*Nov 25 00:18:10.564: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/7, changed state to down
*Nov 25 00:18:11.048: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/13, changed state to down
*Nov 25 00:18:13.232: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/7, changed state to up
*Nov 25 00:18:13.615: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/13, changed state to up
*Nov 25 00:18:14.276: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/7, changed state to up
*Nov 25 00:18:14.640: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/13, changed state to up
*Nov 25 00:19:16.852: %SYS-5-CONFIG_I: Configured from console by supportcisco on vty0 (10.49.1.10)
*Nov 25 00:23:35.895: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/7, changed state to down
*Nov 25 00:23:35.997: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/13, changed state to down
*Nov 25 00:23:36.912: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/7, changed state to down
*Nov 25 00:23:37.000: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/13, changed state to down
*Nov 25 00:23:39.551: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/7, changed state to up
*Nov 25 00:23:39.664: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/13, changed state to up
*Nov 25 00:23:40.576: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/7, changed state to up
*Nov 25 00:23:40.684: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/13, changed state to up
*Nov 25 00:26:12.079: %SYS-5-CONFIG_I: Configured from console by supportcisco on vty0 (10.49.1.10)
*Nov 25 00:28:31.414: %SYS-5-CONFIG_I: Configured from console by supportcisco on vty0 (10.49.1.10)
*Nov 25 00:34:39.049: %SW_MATM-4-MACFLAP_NOTIF: Host b418.d144.e169 in vlan 777 is flapping between port Gi1/0/12 and port Po23
*Nov 25 00:34:39.223: %SW_MATM-4-MACFLAP_NOTIF: Host b418.d144.e169 in vlan 777 is flapping between port Gi1/0/12 and port Po23
*Nov 25 00:39:20.071: %SYS-5-CONFIG_I: Configured from console by supportcisco on vty0 (10.49.1.10)
*Nov 25 00:39:23.376: %SW_MATM-4-MACFLAP_NOTIF: Host c063.9408.03da in vlan 777 is flapping between port Po23 and port Gi1/0/19
*Nov 25 00:39:23.475: %SW_MATM-4-MACFLAP_NOTIF: Host c063.9408.03da in vlan 777 is flapping between port Po23 and port Gi1/0/19
*Nov 25 00:42:37.420: %SW_MATM-4-MACFLAP_NOTIF: Host f025.b74e.f34d in vlan 777 is flapping between port Gi1/0/12 and port Gi1/0/19
*Nov 25 00:42:37.596: %SW_MATM-4-MACFLAP_NOTIF: Host f025.b74e.f34d in vlan 777 is flapping between port Gi1/0/12 and port Gi1/0/19
*Nov 25 00:44:35.993: %SYS-5-CONFIG_I: Configured from console by supportcisco on vty0 (10.49.1.10)
*Nov 25 00:50:41.172: %SW_MATM-4-MACFLAP_NOTIF: Host 0008.2246.010e in vlan 777 is flapping between port Gi1/0/12 and port Gi1/0/19
*Nov 25 00:50:41.361: %SW_MATM-4-MACFLAP_NOTIF: Host 0008.2246.010e in vlan 777 is flapping between port Gi1/0/12 and port Gi1/0/19
*Nov 25 01:01:27.767: %SYS-5-CONFIG_I: Configured from console by supportcisco on vty0 (10.49.1.10)
*Nov 25 01:12:16.560: %SW_MATM-4-MACFLAP_NOTIF: Host bc3b.af06.8f95 in vlan 777 is flapping between port Gi1/0/12 and port Gi1/0/19
*Nov 25 01:12:16.660: %SW_MATM-4-MACFLAP_NOTIF: Host bc3b.af06.8f95 in vlan 777 is flapping between port Gi1/0/12 and port Gi1/0/19
*Nov 25 01:14:10.944: %SW_MATM-4-MACFLAP_NOTIF: Host 7cc5.373b.bfae in vlan 777 is flapping between port Po24 and port Po23
*Nov 25 01:14:11.120: %SW_MATM-4-MACFLAP_NOTIF: Host 7cc5.373b.bfae in vlan 777 is flapping between port Po24 and port Po23
*Nov 25 01:29:07.558: %SW_MATM-4-MACFLAP_NOTIF: Host b418.d144.e169 in vlan 777 is flapping between port Gi1/0/12 and port Gi1/0/19
*Nov 25 01:29:07.655: %SW_MATM-4-MACFLAP_NOTIF: Host b418.d144.e169 in vlan 777 is flapping between port Gi1/0/12 and port Gi1/0/19
*Nov 25 01:33:40.911: %SW_MATM-4-MACFLAP_NOTIF: Host a0ed.cdb9.3dac in vlan 777 is flapping between port Po24 and port Po23
*Nov 25 01:33:41.014: %SW_MATM-4-MACFLAP_NOTIF: Host a0ed.cdb9.3dac in vlan 777 is flapping between port Po24 and port Po23
*Nov 25 01:38:23.135: %SW_MATM-4-MACFLAP_NOTIF: Host acfd.ec67.65b4 in vlan 777 is flapping between port Po23 and port Po24
*Nov 25 01:38:23.323: %SW_MATM-4-MACFLAP_NOTIF: Host acfd.ec67.65b4 in vlan 777 is flapping between port Po23 and port Po24
*Nov 25 01:41:25.962: %SW_MATM-4-MACFLAP_NOTIF: Host 80be.0574.4ceb in vlan 777 is flapping between port Gi1/0/12 and port Gi1/0/19
*Nov 25 01:41:26.055: %SW_MATM-4-MACFLAP_NOTIF: Host 80be.0574.4ceb in vlan 777 is flapping between port Gi1/0/12 and port Gi1/0/19
*Nov 25 01:43:18.622: %SW_MATM-4-MACFLAP_NOTIF: Host d8bb.2c64.47c2 in vlan 777 is flapping between port Gi1/0/12 and port Gi1/0/19
*Nov 25 01:43:18.794: %SW_MATM-4-MACFLAP_NOTIF: Host d8bb.2c64.47c2 in vlan 777 is flapping between port Gi1/0/12 and port Gi1/0/19
*Nov 25 01:43:37.430: %SW_MATM-4-MACFLAP_NOTIF: Host 001d.0791.388a in vlan 777 is flapping between port Gi1/0/12 and port Gi1/0/19
*Nov 25 01:43:37.535: %SW_MATM-4-MACFLAP_NOTIF: Host 001d.0791.388a in vlan 777 is flapping between port Gi1/0/12 and port Gi1/0/19
*Nov 25 01:47:10.184: %SW_MATM-4-MACFLAP_NOTIF: Host c885.50b5.7475 in vlan 777 is flapping between port Gi1/0/19 and port Po24
*Nov 25 01:47:10.358: %SW_MATM-4-MACFLAP_NOTIF: Host c885.50b5.7475 in vlan 777 is flapping between port Gi1/0/19 and port Po24
*Nov 25 01:48:34.352: %SW_MATM-4-MACFLAP_NOTIF: Host b878.2e8a.6a82 in vlan 777 is flapping between port Po23 and port Po24
*Nov 25 01:48:34.532: %SW_MATM-4-MACFLAP_NOTIF: Host b878.2e8a.6a82 in vlan 777 is flapping between port Po23 and port Po24
*Nov 25 01:54:27.250: %SW_MATM-4-MACFLAP_NOTIF: Host b452.7e74.5e95 in vlan 777 is flapping between port Gi1/0/12 and port Gi1/0/19
*Nov 25 01:54:27.355: %SW_MATM-4-MACFLAP_NOTIF: Host b452.7e74.5e95 in vlan 777 is flapping between port Gi1/0/12 and port Gi1/0/19
*Nov 25 02:01:36.639: %SW_MATM-4-MACFLAP_NOTIF: Host f0f6.1c2e.5756 in vlan 777 is flapping between port Gi1/0/12 and port Gi1/0/19
*Nov 25 02:01:36.816: %SW_MATM-4-MACFLAP_NOTIF: Host f0f6.1c2e.5756 in vlan 777 is flapping between port Gi1/0/12 and port Gi1/0/19
*Nov 25 02:08:09.903: %SW_MATM-4-MACFLAP_NOTIF: Host 786c.1c98.b5ad in vlan 777 is flapping between port Po23 and port Po24
*Nov 25 02:08:10.090: %SW_MATM-4-MACFLAP_NOTIF: Host 786c.1c98.b5ad in vlan 777 is flapping between port Po23 and port Po24
*Nov 25 02:10:28.574: %SW_MATM-4-MACFLAP_NOTIF: Host c09f.426d.d7d6 in vlan 777 is flapping between port Gi1/0/19 and port Gi1/0/12
*Nov 25 02:10:28.745: %SW_MATM-4-MACFLAP_NOTIF: Host c09f.426d.d7d6 in vlan 777 is flapping between port Gi1/0/19 and port Gi1/0/12
*Nov 25 02:21:39.802: %SW_MATM-4-MACFLAP_NOTIF: Host 80ea.9607.7765 in vlan 777 is flapping between port Gi1/0/19 and port Gi1/0/12
*Nov 25 02:21:39.992: %SW_MATM-4-MACFLAP_NOTIF: Host 80ea.9607.7765 in vlan 777 is flapping between port Gi1/0/19 and port Gi1/0/12
*Nov 25 02:23:16.912: %SW_MATM-4-MACFLAP_NOTIF: Host 6809.2782.17d8 in vlan 777 is flapping between port Gi1/0/12 and port Po23
*Nov 25 02:23:17.102: %SW_MATM-4-MACFLAP_NOTIF: Host 6809.2782.17d8 in vlan 777 is flapping between port Gi1/0/12 and port Po23
*Nov 25 02:26:44.606: %SW_MATM-4-MACFLAP_NOTIF: Host dc9b.9c40.250e in vlan 777 is flapping between port Po24 and port Po23
*Nov 25 02:26:44.794: %SW_MATM-4-MACFLAP_NOTIF: Host dc9b.9c40.250e in vlan 777 is flapping between port Po24 and port Po23
*Nov 25 02:34:58.241: %SW_MATM-4-MACFLAP_NOTIF: Host d8bb.2c64.47c2 in vlan 777 is flapping between port Gi1/0/19 and port Gi1/0/12
*Nov 25 02:34:58.335: %SW_MATM-4-MACFLAP_NOTIF: Host d8bb.2c64.47c2 in vlan 777 is flapping between port Gi1/0/19 and port Gi1/0/12
*Nov 25 02:51:13.408: %SW_MATM-4-MACFLAP_NOTIF: Host 40b3.9570.f1fd in vlan 777 is flapping between port Gi1/0/19 and port Po23
*Nov 25 02:51:13.505: %SW_MATM-4-MACFLAP_NOTIF: Host 40b3.9570.f1fd in vlan 777 is flapping between port Gi1/0/19 and port Po23
*Nov 25 03:05:02.808: %SW_MATM-4-MACFLAP_NOTIF: Host a888.085b.c866 in vlan 777 is flapping between port Gi1/0/12 and port Gi1/0/19
*Nov 25 03:05:02.903: %SW_MATM-4-MACFLAP_NOTIF: Host a888.085b.c866 in vlan 777 is flapping between port Gi1/0/12 and port Gi1/0/19
*Nov 25 03:09:06.387: %SW_MATM-4-MACFLAP_NOTIF: Host 54ae.278b.556f in vlan 777 is flapping between port Po24 and port Po23
*Nov 25 03:09:06.483: %SW_MATM-4-MACFLAP_NOTIF: Host 54ae.278b.556f in vlan 777 is flapping between port Po24 and port Po23
*Nov 25 03:14:31.833: %SW_MATM-4-MACFLAP_NOTIF: Host 60d9.c725.7006 in vlan 777 is flapping between port Po23 and port Gi1/0/19
*Nov 25 03:14:31.933: %SW_MATM-4-MACFLAP_NOTIF: Host 60d9.c725.7006 in vlan 777 is flapping between port Po23 and port Gi1/0/19
*Nov 25 03:34:47.090: %SW_MATM-4-MACFLAP_NOTIF: Host acfd.ec67.65b4 in vlan 777 is flapping between port Po24 and port Po23
*Nov 25 03:34:47.182: %SW_MATM-4-MACFLAP_NOTIF: Host acfd.ec67.65b4 in vlan 777 is flapping between port Po24 and port Po23
*Nov 25 03:43:36.015: %SW_MATM-4-MACFLAP_NOTIF: Host b4f0.abd1.37c2 in vlan 777 is flapping between port Po23 and port Po24
*Nov 25 03:43:36.114: %SW_MATM-4-MACFLAP_NOTIF: Host b4f0.abd1.37c2 in vlan 777 is flapping between port Po23 and port Po24
*Nov 25 03:49:53.583: %SW_MATM-4-MACFLAP_NOTIF: Host d8bb.2cb3.bdcc in vlan 777 is flapping between port Gi1/0/12 and port Gi1/0/19
*Nov 25 03:49:53.681: %SW_MATM-4-MACFLAP_NOTIF: Host d8bb.2cb3.bdcc in vlan 777 is flapping between port Gi1/0/12 and port Gi1/0/19
*Nov 25 03:49:58.128: %SW_MATM-4-MACFLAP_NOTIF: Host 74e2.f5af.3f38 in vlan 777 is flapping between port Gi1/0/19 and port Po23

InayathUlla Sharieff · ‎11-25-2014

Like to add additional help with leo on this:

Based on the logs I could see MAC flap occuring between the ports G1/0/19 and Po23.

Kindly check what is connected to these ports and why we are seeing duplication.

Second thing Please find way to troubleshoot high cpu on 3850:

Catalyst 3850 Series Switch High CPU Usage Troubleshoot
http://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/117594-te
chnote-hicpu3850-00.html

<snip>

This document describes how to troubleshoot CPU usage concerns, primarily due to
interrupts, on the new Cisco IOS(r)-XE platform. Additionally, the document introduces
several new commands on this platform that are integral in order to troubleshoot such
problems.

HTH

p0wershell · ‎11-25-2014

thx for your reply

before posting there i am read this guide

i am determinate

Step 2: Determine the CPU Queue that Causes the High CPU Usage Condition

RX (ASIC2CPU) Stats (asic 0 qn 20 lqn 20):
RXQ 20: CPU_Q_MCAST_END_STATION_SERVICE
----------------------------------------
Packets received from ASIC : 3024821
Send to IOSd total attempts : 3024821
Send to IOSd failed count : 16979
RX suspend count : 16979
RX unsuspend count : 16978
RX unsuspend send count : 16978
RX unsuspend send failed count : 0
RX dropped count : 0
RX conversion failure dropped : 0
RX pkt_hdr allocation failure : 0
RX INTACK count : 188654
RX packets dq'd after intack : 2870
Active RxQ event : 549316
RX spurious interrupt : 11643

but what i can do ?

now all normal, this is flap trouble.... when trouble come again, post addition information

Ismael Almandos · ‎09-24-2015

Hi p0wershell@mail.ru, I am having the exact same problem that you had, in very similar conditions. Could you please tell me if you have solved the issue?, and if you have indeed solved it, could tell me how did you do it?

Thank you in advance.

Leo Laohoo · ‎11-26-2014

I going to agree with Inayath.

I am suspecting your IOS daemon CPU spike is caused by the incorrect configuration of Etherchannel.

p0wershell · ‎12-16-2014

hello

"flap" wifi connect to 2960 when human moves across building he change wifi AP, and change PORTCHANNEL too, i think this is not to bad.

trouble in Multicast!

RX (ASIC2CPU) Stats (asic 0 qn 20 lqn 20):
RXQ 20: CPU_Q_MCAST_END_STATION_SERVICE
----------------------------------------
Packets received from ASIC : 535765512
Send to IOSd total attempts : 535764862
Send to IOSd failed count : 2624599
RX suspend count : 2624599
RX unsuspend count : 2624599
RX unsuspend send count : 2624600
RX unsuspend send failed count : 1
RX dropped count : 649
RX conversion failure dropped : 649
RX pkt_hdr allocation failure : 0
RX INTACK count : 55806795
RX packets dq'd after intack : 1055207
Active RxQ event : 116603097
RX spurious interrupt : 5312201

----------------------------------------------------------------------------------

tag buffer jumbo fallback packets received failures
alloc free bytes conv buf
27 0/1024/2048 0/5 0/5 0 0 0 0 0
65536 0/1024/1600 0/0 0/512 31284566 31284566 2002207193 0 0
65537 0/ 512/1600 0/0 0/512 922238 922238 195398227 0 0
65538 0/ 5/5 0/0 0/5 0 0 0 0 0
65539 2/2048/1600 0/16 0/512 23840004 23842875 3002779496 0 0
65540 0/ 128/1600 0/8 0/0 357463 357463 41172730 0 0
65541 2/ 128/1600 0/16 0/32 10778028 15229705 1298773735 0 0
s65542 733/ 768/1600 0/4 0/0 535850867 4201994200 3127366840 0 2451867
65544 0/ 96/1600 0/4 0/0 0 0 0 0 0
65545 0/ 96/1600 0/8 0/32 0 0 0 0 0
65546 0/ 512/1600 0/32 0/512 7903366 8056563 755109739 0 0
65547 0/ 96/1600 0/8 0/32 0 0 0 0 0
65548 0/ 512/1600 0/32 0/256 318 318 19080 0 0
65551 0/ 512/1600 0/0 0/256 31934 31934 4630871 0 0
65556 0/ 16/1600 0/4 0/0 0 0 0 0 0
65557 0/ 16/1600 0/4 0/0 0 0 0 0 0
65558 0/ 16/1600 0/4 0/0 291538 291538 16428606 0 0
65559 0/ 16/1600 0/4 0/0 614 614 142707 0 0
65560 0/ 16/1600 0/4 0/0 434 434 36881 0 0
65561 5/ 512/1600 0/0 0/128 36863055 59397924 2221294360 0 179

----------------------------------------------------------------------------------

show pds tag all | in Active|Tags|65542
Active Client Client
Tags Handle Name TDA SDA FDA TBufD TBytD
65542 6853728 Punt Rx Mcast Snoop 536047248 536047248 0 536047248 993436420

-------------------------------------------------------------------------------------

show pds tag all
65542 6853728 Punt Rx Mcast Snoop 536076471 536076471 0 536076471 996066238

Dumping Packet(245376) # 0 of Length 90
-----------------------------------------

Meta-data

0000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0010 00 00 16 36 00 00 00 00 00 00 00 00 55 73 f1 f0 ...6........Us..
0020 00 00 00 00 fd 01 10 df 00 5b 70 00 00 11 44 00 .........[p...D.
0030 00 11 44 00 00 42 de 40 00 42 de 40 00 00 00 00 ..D..B.@.B.@....
0040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0050 00 00 00 5a 00 00 00 00 00 01 00 06 00 00 00 00 ...Z............
0060 01 02 bd c0 00 00 00 ab 00 00 00 00 00 00 00 00 ................
0070 01 07 46 80 00 00 00 75 01 08 87 80 00 00 00 9f ..F....u........
0080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0090 00 31 00 00 00 00 00 00 00 00 00 00 00 00 00 00 .1..............
00a0 00 00 00 00 00 00 00 03 00 00 00 00 00 00 00 00 ................
Data

0000 33 33 ff d5 c0 4d 40 a8 f0 45 ec 49 81 00 00 31 33...M@..E.I...1
0010 86 dd 60 00 00 00 00 20 00 01 fe 80 00 00 00 00 ..`.... ........
0020 00 00 42 a8 f0 ff fe 45 ec 49 ff 02 00 00 00 00 ..B....E.I......
0030 00 00 00 00 00 01 ff d5 c0 4d 3a 00 05 02 00 00 .........M:.....
0040 01 00 83 00 e1 a4 00 00 00 00 ff 02 00 00 00 00 ................
0050 00 00 00 00 00 01 ff d5 c0 4d .........M

0000 33 33 ff d5 c0 4d 40 a8 f0 45 ec 49 81 00 00 31 33...M@..E.I...1

33 33 ff d5 c0 4d - Multicast IPv6

--------------------------------------------------

FED Tracing

[12/16/14 01:52:48.781 UTC 6c409d24 5686]
00 00 00 00 00 4e 00 5e 3a 00 06 db 00 00 51 40
00 00 02 00 00 01 00 00 07 00 00 00 00 00 00 31
00 00 00 00 20 00 00 13 00 00 00 00 00 01 00 79
00 00 00 04 00 ee 01 07 00 00 00 00 00 00 00 00

[12/16/14 01:52:48.781 UTC 6c409d25 5686]
33 33 ff d5 c0 4d 40 a8 f0 43 62 f0 81 00 00 31
86 dd 60 00 00 00 00 20 00 01 fe 80 00 00 00 00
00 00 42 a8 f0 ff fe 43 62 f0 ff 02 00 00 00 00
00 00 00 00 00 01 ff d5 c0 4d 3a 00 05 02 00 00

[12/16/14 01:52:48.781 UTC 6c409d26 5686]
01 00 83 00 6b 00 00 00 00 00 ff 02 00 00 00 00
00 00 00 00 00 01 ff d5 c0 4d d2 ed ff 87
[12/16/14 01:52:48.781 UTC 6c409d27 5686] Frame descriptors:
[12/16/14 01:52:48.781 UTC 6c409d28 5686]
=========
fdFormat=0x4 systemTtl=0xe
loadBalHash1=0x1b loadBalHash2=0x1b
spanSessionMap=0x0 forwardingMode=0x0
destModIndex=0x0 skipIdIndex=0x4
srcGpn=0x2ee qosLabel=0x1
srcCos=0x0 ingressTranslatedVlan=0x7
bpdu=0x0 spanHistory=0x0
sgt=0x0 fpeFirstHeaderType=0x0
srcVlan=0x31 rcpServiceId=0x2
wccpSkip=0x0 srcPortLeIndex=0x13
cryptoProtocol=0x0 debugTagId=0x0
vrfId=0x0 saIndex=0x0
pendingAfdLabel=0x0 destClient=0x1
appId=0x0 finalStationIndex=0x79
decryptSuccess=0x0 encryptSuccess=0x0
rcpMiscResults=0x0 stackedFdPresent=0x0
spanDirection=0x0 egressRedirect=0x0
redirectIndex=0x0 exceptionLabel=0x0
destGpn=0x0 inlineFd=0x0
suppressRefPtrUpdate=0x0 suppressRewriteSideEfects=0x0
cmi2=0x0 currentRi=0x1
currentDi=0x5140 dropIpUnreachable=0x0
srcZoneId=0x0 srcAsicId=0x0
originalDi=0x0 originalRi=0x0
srcL3IfIndex=0x7 dstL3IfIndex=0x0
dstVlan=0x0 frameLength=0x5e
fdCrc=0x3a tunnelSpokeId=0x0

=========

[12/16/14 01:52:48.781 UTC 6c409d29 5686]

[12/16/14 01:52:48.781 UTC 6c409d2a 5686] PUNT PATH (fed_punject_rx_process_packet:830):RX: Q: 20, Tag: 65542 указывает, что мы смотрим то что нужно

[12/16/14 01:52:48.781 UTC 6c409d2b 5686] PUNT PATH (fed_punject_get_physical_iif:583):RX: Physical IIF-id 0x105c2c00000007b

[12/16/14 01:52:48.781 UTC 6c409d2c 5686] PUNT PATH (fed_punject_get_src_l3if_index:438):RX: L3 IIF-id 0x102bdc0000000ab

[12/16/14 01:52:48.781 UTC 6c409d2d 5686] PUNT PATH (fed_punject_get_logical_iif:629):RX: Logical IIF-id 0x105f0c000000099

[12/16/14 01:52:48.781 UTC 6c409d2e 5686] PUNT PATH (fed_punject_get_source_cos:642):RX: Source Cos 0

[12/16/14 01:52:48.781 UTC 6c409d2f 5686] PUNT PATH (fed_punject_get_vrf_id:657):RX: VRF-id 0

[12/16/14 01:52:48.781 UTC 6c409d30 5686] PUNT PATH (fed_punject_get_src_zoneid:671):RX: Zone-id 0

[12/16/14 01:52:48.781 UTC 6c409d31 5686] PUNT PATH (fed_punject_fd_2_pds_md:518):RX: get_src_zoneid failed

[12/16/14 01:52:48.781 UTC 6c409d32 5686] PUNT PATH (fed_punject_get_acl_log_direction:699):RX: : Invalid CMI2

[12/16/14 01:52:48.781 UTC 6c409d33 5686] PUNT PATH (fed_punject_fd_2_pds_md:541):RX: get_acl_log_direction failed

[12/16/14 01:52:48.781 UTC 6c409d34 5686] PUNT PATH (fed_punject_get_acl_full_direction:728):RX: DI 0x5140 ACL Full Direction 1

[12/16/14 01:52:48.781 UTC 6c409d35 5686] PUNT PATH (fed_punject_get_source_sgt:450):RX: Source SGT 0

[12/16/14 01:52:48.781 UTC 6c409d36 5686] PUNT PATH (fed_punject_get_first_header_type:684):RX: FirstHeaderType 0

[12/16/14 01:52:48.781 UTC 6c409d37 5686] PUNT PATH (fed_punject_rx_process_packet:916):RX: fed_punject_pds_send packet 0x3cac0 to IOSd with tag 65542

[12/16/14 01:52:48.781 UTC 6c409d38 5686] PUNT PATH (fed_punject_rx_process_packet:744):RX: **** RX packet 0x2a760 on qn 20, len 158 ****

IPv6 Multicast - 33 33 ff d5 c0 4d