Hi All,
I have a issue with a 3650 constantly running at 100% cpu and I have been unable to resolve this.
CTER3-3650#CTER3-3650#show proc cpu sorted
CTER3-3650#sho proc
CTER3-3650#sho processes cpu sort
CTER3-3650#sho processes cpu sorted | ex 0.0
Core 0: CPU utilization for five seconds: 100%; one minute: 99%; five minutes: 99%
Core 1: CPU utilization for five seconds: 99%; one minute: 99%; five minutes: 99%
Core 2: CPU utilization for five seconds: 100%; one minute: 99%; five minutes: 99%
Core 3: CPU utilization for five seconds: 100%; one minute: 99%; five minutes: 99%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
5674 1321924 16101262 499 35.12 43.42 43.05 0 stack-mgr
5672 3937135 19674900 651 33.53 41.51 42.26 1088 fed
6212 3026809 86451857 1211 19.94 3.74 3.38 0 eicored
6203 3546957 25284505 1189 7.89 4.08 3.47 0 pdsd
6230 2704854 87149186 2235 0.10 1.38 1.35 0 snmp_subagent
CTER3-3650#sho proc cpu detailed process iosd sort | ex 0.0
Core 0: CPU utilization for five seconds: 99%; one minute: 99%; five minutes: 99%
Core 1: CPU utilization for five seconds: 97%; one minute: 99%; five minutes: 99%
Core 2: CPU utilization for five seconds: 100%; one minute: 99%; five minutes: 99%
Core 3: CPU utilization for five seconds: 99%; one minute: 99%; five minutes: 99%
PID T C TID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
(%) (%) (%)
160 I 982983 1649447 0 16.88 10.11 11.99 0 NGWC L2M
7 I 1317998 1584002 0 4.00 0.66 0.66 0 Check heaps
289 I 1387923 2667234 0 2.99 3.22 3.00 0 SNMP ENGINE
22 I 4178407 3173413 0 0.88 0.99 0.88 2 CMI IOSd task
98 I 3714961 1523229 0 0.44 0.11 0.11 0 PLATFORM_MGR SPI in
241 I 3635527 1758079 0 0.44 0.22 0.22 0 Spanning Tree
59 I 3890449 4269676 0 0.22 0.44 0.44 0 ARP Snoop
CTER3-3650#sho proc cpu detailed process fed sort | ex 0.0
Core 0: CPU utilization for five seconds: 95%; one minute: 99%; five minutes: 99%
Core 1: CPU utilization for five seconds: 99%; one minute: 99%; five minutes: 99%
Core 2: CPU utilization for five seconds: 99%; one minute: 99%; five minutes: 99%
Core 3: CPU utilization for five seconds: 99%; one minute: 99%; five minutes: 99%
PID T C TID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
(%) (%) (%)
5672 L 4032535 1967512 651 47.37 41.43 41.98 1088 fed
5672 L 0 6111 1993435 1323326 0 22.72 20.64 21.27 0 fed-ots-main
5672 L 3 10699 361576 1910269 0 16.42 18.90 18.99 0 PunjectRx
5672 L 0 6114 2992853 5320315 0 0.50 0.51 0.50 0 fed-ots-nfl
CTER3-3650#sho platform punt client
tag buffer jumbo fallback packets received failures
alloc free bytes conv buf
27 0/1024/2048 0/5 0/5 0 0 0 0 0
65536 0/1024/1600 0/0 0/512 123114809 123114809 2553938877 0 0
65537 0/ 512/1600 0/0 0/512 26090873 26090873 1515823137 0 0
65538 0/ 5/5 0/0 0/5 0 0 0 0 0
65539 2/2048/1600 0/16 0/512 117665987 117665985 3733224833 0 0
65540 0/ 128/1600 0/8 0/0 0 0 0 0 0
65541 0/ 128/1600 0/16 0/32 3502140980 3502140969 1422369189 0 18
65542 0/ 768/1600 0/4 0/0 1516477614 3928395657 691655428 0 3254417
65544 0/ 96/1600 0/4 0/0 0 0 0 0 0
65545 0/ 96/1600 0/8 0/32 0 0 0 0 0
65546 52/ 512/1600 0/32 0/512 2491828554 2499039986 727260577 0 0
65547 0/ 96/1600 0/8 0/32 0 0 0 0 0
65548 0/ 512/1600 0/32 0/256 41815602 41815416 3844127882 0 2337
65551 0/ 512/1600 0/0 0/256 177382 177382 10646576 0 19
65556 0/ 16/1600 0/4 0/0 0 0 0 0 0
65557 0/ 16/1600 0/4 0/0 0 0 0 0 0
65558 0/ 16/1600 0/4 0/0 0 0 0 0 0
65559 0/ 16/1600 0/4 0/0 0 0 0 0 0
65560 0/ 16/1600 0/4 0/0 0 0 0 0 0
65561 0/ 512/1600 0/0 0/128 2081090233 2225069567 2209607374 0 38
65563 0/ 512/1600 0/16 0/256 0 0 0 0 0
65564 0/ 512/1600 0/16 0/256 0 0 0 0 0
65565 0/ 512/1600 0/16 0/256 0 0 0 0 0
65566 0/ 512/1600 0/16 0/256 0 0 0 0 0
65581 0/ 1/1 0/0 0/0 0 0 0 0 0
131071 0/ 96/1600 0/4 0/0 0 0 0 0 0
fallback pool: 0/1500/1600
jumbo pool: 0/128/9300
CTER3-3650#
CTER3-3650#sho mgmt-infra trace messages fed-punject-detail
[02/12/16 13:38:50.487 fcbfe0a 5672] PUNT PATH (fed_punject_get_physical_iif:583):RX: Physical IIF-id 0x10629c00000002f
[02/12/16 13:38:50.487 fcbfe0b 5672] PUNT PATH (fed_punject_get_src_l3if_index:410):RX: Invalid SRC_L3IF_INDEX: 0
[02/12/16 13:38:50.487 fcbfe0c 5672] PUNT PATH (fed_punject_fd_2_pds_md:466):RX: l3_if = 0x0
[02/12/16 13:38:50.487 fcbfe0d 5672] PUNT PATH (fed_punject_fd_2_pds_md:478):RX: l2_logical_if = 0x0
[02/12/16 13:38:50.487 fcbfe0e 5672] PUNT PATH (fed_punject_get_source_cos:642):RX: Source Cos 0
[02/12/16 13:38:50.487 fcbfe0f 5672] PUNT PATH (fed_punject_get_vrf_id:657):RX: VRF-id 0
[02/12/16 13:38:50.487 fcbfe10 5672] PUNT PATH (fed_punject_get_src_zoneid:671):RX: Zone-id 0
[02/12/16 13:38:50.487 fcbfe11 5672] PUNT PATH (fed_punject_fd_2_pds_md:518):RX: get_src_zoneid failed
[02/12/16 13:38:50.487 fcbfe12 5672] PUNT PATH (fed_punject_get_acl_log_direction:699):RX: : Invalid CMI2
[02/12/16 13:38:50.487 fcbfe13 5672] PUNT PATH (fed_punject_fd_2_pds_md:541):RX: get_acl_log_direction failed
[02/12/16 13:38:50.487 fcbfe14 5672] PUNT PATH (fed_punject_get_acl_full_direction:728):RX: DI 0x5140 ACL Full Direction 1
[02/12/16 13:38:50.487 fcbfe15 5672] PUNT PATH (fed_punject_get_source_sgt:450):RX: Source SGT 0
[02/12/16 13:38:50.487 fcbfe16 5672] PUNT PATH (fed_punject_get_first_header_type:684):RX: FirstHeaderType 0
[02/12/16 13:38:50.487 fcbfe17 5672] PUNT PATH (fed_punject_rx_process_packet:916):RX: fed_punject_pds_send packet 0x4ba0 to IOSd with tag 65542
[02/12/16 13:38:50.488 fcbfe18 5672] PUNT PATH (fed_punject_rx_process_packet:744):RX: **** RX packet 0x26e80 on qn 20, len 158 ****
[02/12/16 13:38:50.488 fcbfe19 5672]
buf_no 0 buf_len 158
[02/12/16 13:38:50.488 fcbfe1a 5672]
00 00 00 00 00 4e 00 5e ea 80 0e 59 00 00 51 40
00 00 00 00 00 01 00 60 75 00 00 00 00 00 00 fa
00 00 00 00 30 00 00 0c 00 00 00 00 00 01 00 79
00 00 00 04 00 12 01 00 00 00 00 00 00 00 00 00
[02/12/16 13:38:50.488 fcbfe1b 5672]
33 33 88 00 00 01 c8 cb b8 27 17 49 81 00 e0 fa
86 dd 60 00 00 00 00 20 00 01 fe 80 00 00 00 00
00 00 ca cb b8 ff fe 27 17 49 ff 12 00 00 00 00
00 00 00 00 00 80 88 00 00 01 3a 00 05 02 00 00
[02/12/16 13:38:50.488 fcbfe1c 5672]
01 00 83 00 d5 c6 00 00 00 00 ff 12 00 00 00 00
00 00 00 00 00 80 88 00 00 01 9e 75 4a f1
[02/12/16 13:38:50.488 fcbfe1d 5672] Frame descriptors:
[02/12/16 13:38:50.488 fcbfe1e 5672]
=========
fdFormat=0x4 systemTtl=0xe
loadBalHash1=0x39 loadBalHash2=0x39
spanSessionMap=0x0 forwardingMode=0x0
destModIndex=0x0 skipIdIndex=0x4
srcGpn=0x12 qosLabel=0x1
srcCos=0x6 ingressTranslatedVlan=0x75
bpdu=0x0 spanHistory=0x0
sgt=0x0 fpeFirstHeaderType=0x0
srcVlan=0xfa rcpServiceId=0x3
wccpSkip=0x0 srcPortLeIndex=0xc
cryptoProtocol=0x0 debugTagId=0x0
vrfId=0x0 saIndex=0x0
pendingAfdLabel=0x0 destClient=0x1
appId=0x0 finalStationIndex=0x79
decryptSuccess=0x0 encryptSuccess=0x0
rcpMiscResults=0x0 stackedFdPresent=0x0
spanDirection=0x0 egressRedirect=0x0
redirectIndex=0x0 exceptionLabel=0x0
destGpn=0x0 inlineFd=0x0
suppressRefPtrUpdate=0x0 suppressRewriteSideEfects=0x0
cmi2=0x0 currentRi=0x1
currentDi=0x5140 dropIpUnreachable=0x0
srcZoneId=0x0 srcAsicId=0x0
originalDi=0x0 originalRi=0x0
srcL3IfIndex=0x0 dstL3IfIndex=0x0
dstVlan=0x0 frameLength=0x5e
fdCrc=0xea tunnelSpokeId=0x0
=========
CTER3-3650#sho ipv6 mld snooping vlan 250
Global MLD Snooping configuration:
-------------------------------------------
MLD snooping : Enabled
MLDv2 snooping (minimal) : Enabled
Listener message suppression : Enabled
TCN solicit query : Disabled
TCN flood query count : 2
Robustness variable : 2
Last listener query count : 2
Last listener query interval : 1000
Vlan 250:
--------
MLD snooping : Enabled
MLD immediate leave : Disabled
Robustness variable : 2
Last listener query count : 2
Last listener query interval : 1000
CTER3-3650#
I have also reduce the multicast storm level to 0.5
Any ideas as a way forward to reduce the CPU will be appreciated.
Regards
Ian
