[案例分享]ARP导致3850交换机CPU中断高 - 第2页

xupeng · ‎04-09-2015

本帖最后由 xupeng 于 2015-4-10 09:12 编辑

案例分享:ARP导致3850交换机CPU中断高

第一步: 找到CPU占用率高的进程

3850系列交换机使用4核CPU, 使用show process cpu 命令将会看到:

3850-2#show processes cpu sort | exclude 0.0

Core 0: CPU utilization for five seconds: 53%; one minute: 39%; five minutes: 41%

Core 1: CPU utilization for five seconds: 43%; one minute: 57%; five minutes: 54%

Core 2: CPU utilization for five seconds: 95%; one minute: 60%; five minutes: 58%

Core 3: CPU utilization for five seconds: 32%; one minute: 31%; five minutes: 29%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

8525 472560 2345554 7525 31.37 30.84 30.83 0 iosd

5661 2157452 9234031 698 13.17 12.56 12.54 1088 fed

6206 19630 74895 262 1.83 0.43 0.10 0 eicored

6197 725760 11967089 60 1.41 1.38 1.47 0 pdsd

根据以上输出,可以看到iosd和fed进程占用大量CPU.

*Forwarding Engine Driver (FED): This is the heart of the Cisco Catalyst 3850 Series Switch and is responsible for all hardware programming/forwarding.

当由于中断(interrupts)导致CPU利用率高,将会看到IOSd和FED进程占用大量CPU,以下子进程将占用CPU:

FED Punject TX
FED Punject RX
FED Punject replenish
FED Punject TX complete

我们可以使用 show process cpu detailed 命令去获得进程的更详细内容.

在这个案例中,我们看看这2个进程:

3850-2#show processes cpu detailed process iosd sort | ex 0.0

Core 0: CPU utilization for five seconds: 36%; one minute: 39%; five minutes: 40%

Core 1: CPU utilization for five seconds: 73%; one minute: 52%; five minutes: 53%

Core 2: CPU utilization for five seconds: 22%; one minute: 56%; five minutes: 58%

Core 3: CPU utilization for five seconds: 46%; one minute: 40%; five minutes: 31%

PID T C TID Runtime(ms)Invoked uSecs 5Sec 1Min 5Min TTY Process

(%) (%) (%)

8525 L 556160 2356540 7526 30.42 30.77 30.83 0 iosd

8525 L 1 8525 712558 284117 0 23.14 23.33 23.38 0 iosd

59 I 1115452 4168181 0 42.22 39.55 39.33 0 ARP Snoop

198 I 3442960 4168186 0 25.33 24.22 24.77 0 IP Host Track Proce

30 I 3802130 4168183 0 24.66 27.88 27.66 0 ARP Input

283 I 574800 3225649 0 4.33 4.00 4.11 0 DAI Packet Process

3850-2#show processes cpu detailed process fed sorted | ex 0.0

Core 0: CPU utilization for five seconds: 45%; one minute: 44%; five minutes: 44%

Core 1: CPU utilization for five seconds: 38%; one minute: 44%; five minutes: 45%

Core 2: CPU utilization for five seconds: 42%; one minute: 41%; five minutes: 40%

Core 3: CPU utilization for five seconds: 32%; one minute: 30%; five minutes: 31%

PID T C TID Runtime(ms)Invoked uSecs 5Sec 1Min 5Min TTY Process

(%) (%) (%)

5638 L 612840 1143306 536 13.22 12.90 12.93 1088 fed

5638 L 3 8998 396500 602433 0 9.87 9.63 9.61 0 PunjectTx

5638 L 3 8997 159890 66051 0 2.70 2.70 2.74 0 PunjectRx

在IOSd的输出中,看到ARP Snoop, IP Host Track Process, 和 ARP Input 比较高. 比较常见的是由于ARP包造成CPU中断高.

第二步:确定导致CPU利用率高的CPU队列

思科3850系列交换机为不同类型的包提供了不同的队列(FED维护32个RX CPU Queue,这些queue会直接上CPU).

监视这些Queue可以发现哪些包被送上了CPU,哪些被IOSd进程处理. 这些Queue是基于每个Port-ASIC的. 例如,有2个Port-ASIC, 接口1到24属于Port-ASIC 0.

可以使用showplatform punt statistics port-asic cpuq direction 命令去看.

在showplatform punt statistics port-asic 0 cpuq -1 direction rx 命令中, -1参数代理所有queue, 引起,这条命令将显示Port-ASIC 0上所有receivequeue.

现在，必须确定哪个Queue有大量的数据包在以高速率进入queue. 在这个案例中, 检查发现了名叫 CPU_Q_PROTO_SNOOPING的第16个queue有异常.

RX (ASIC2CPU) Stats (asic 0 qn 16lqn 16):

RXQ 16: CPU_Q_PROTO_SNOOPING

----------------------------------------

Packets received from ASIC : 79099152

Send to IOSd total attempts : 79099152

Send to IOSd failed count : 1240331

RX suspend count : 1240331

RX unsuspend count : 1240330

RX unsuspend send count : 1240330

RX unsuspend send failed count : 0

RX dropped count : 0

RX conversion failure dropped : 0

RX pkt_hdr allocation failure : 0

RX INTACK count : 0

RX packets dq'd after intack : 0

Active RxQ event : 9906280

RX spurious interrupt : 0

另一种方法是使用show platform punt client 命令去发现:

3850-2#show platform punt client

tag buffer jumbo fallback packets received failures

65559 0/ 16/1600 0/4 0/0 0 0 0 0 0

65560 0/ 16/1600 0/4 0/0 0 0 0 0 0

s65561 421/ 512/1600 0/0 0/128 79565859131644697 478984244 0 37467

65563 0/ 512/1600 0/16 0/256 0 0 0 0 0

65564 0/ 512/1600 0/16 0/256 0 0 0 0 0

找到最多数据包这一行对应的tag,在这个案例中, tag为65561.

*Handle:A handle can be thought of as a pointer. It is a means to discover moredetailed information about specific variables that are used in the outputs thatthe box produces. This is similar to the concept of Local Target Logic (LTL)indices on the Cisco Catalyst 6500 Series Switch.

*PacketDelivery System (PDS): This is thearchitecture and process of how packets aredelivered to and from varioussubsystem. As an example, it controls how packetsare delivered from the FED tothe IOSd and vice versa.

输入以下命令,可以看到queue名是Rx Proto Snoop.

3850-2#show pds tag all | in Active|Tags|65561

Active Client Client

Tags HandleName TDA SDA FDA TBufD TBytD

65561 7296672 Punt Rx ProtoSnoop 79821397 79821397 0 79821397 494316524

在65561前的s表示大量入向数据包导致FED handle被暂停或者被压垮(FEDhandle is suspended and overwhelmed). 如果s不消失,代表queue永久性的卡住了.

第三步:转储发送到CPU的数据包

在show pds tag all 的输出中看到 handle是7296672.

在使用show pds client packet last sink命令之前,需要开启debug pds pktbuf-last.否则将看到以下输出:

3850-2#show pds client 7296672 packet last sink

% switch-2:pdsd:This command works in debug mode only. Enable debug using

"debug pds pktbuf-last" command

开启debug pds pktbuf-last后,将看到以下输出:

3850-2#show pds client 7296672 packet last sink

Dumping Packet(54528) # 0 of Length 60

-----------------------------------------

Meta-data

0000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................

0010 00 00 16 1d 00 00 00 00 00 00 00 00 55 5a 57 f0 ............UZW.

0020 00 00 00 00 fd 01 10 df 00 5b 70 00 00 10 43 00 .........[p...C.

0030 00 10 43 00 00 41 fd 00 00 41 fd 00 00 00 00 00 ..C..A...A......

0040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................

0050 00 00 00 3c 00 00 00 00 00 01 00 19 00 00 00 00 ...<............

0060 01 01 b6 80 00 00 00 4f 00 00 00 00 00 00 00 00 .......O........

0070 01 04 d8 80 00 00 00 33 00 00 00 00 00 00 00 00 .......3........

0080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................

0090 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................

00a0 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 00 ................

Data

0000 ff ff ff ff ff ff aa bb cc dd 00 00 08 06 00 01 ................

0010 08 00 06 04 00 01 aa bb cc dd 00 00 c0 a8 01 0a ................

0020 ff ff ff ff ff ff c0 a8 01 14 00 01 02 03 04 05 ................

0030 06 07 08 09 0a 0b 0c 0d 0e 0f 10 11 ............

Meta-data是被系统内部使用的,Data输出是真实数据包的信息. 以下命令将使用Meta-data 0070开始的前16bits.

* IIF=Interface ID Factory

3850-2#show platform port-asic ifm iif-id 0x0104d88000000033

Interface Table

Interface IIF-ID : 0x0104d88000000033

Interface Name : Gi2/0/20

Interface Block Pointer : 0x514d2f70

Interface State : READY

Interface Stauts : IFM-ADD-RCVD, FFM-ADD-RCVD

Interface Ref-Cnt : 6

Interface Epoch : 0

Interface Type : ETHER

Port Type : SWITCH PORT

Port Location : LOCAL

Slot : 2

Unit : 20

Slot Unit : 20

Acitve : Y

SNMP IF Index : 22

GPN : 84

EC Channel : 0

EC Index : 0

ASIC : 0

ASIC Port : 14

Port LE Handle : 0x514cd990

Non Zero Feature Ref Counts

FID : 48(AL_FID_L2_PM), Ref Count : 1

FID : 77(AL_FID_STATS), Ref Count : 1

FID : 51(AL_FID_L2_MATM), Ref Count : 1

FID : 13(AL_FID_SC), Ref Count : 1

FID : 26(AL_FID_QOS), Ref Count : 1

Sub block information

FID : 48(AL_FID_L2_PM), Private Data : 0x54072618

FID : 26(AL_FID_QOS), Private Data : 0x514d31b8

从以上输出可以看到ARP数据包来源于接口Gi2/0/20. 此时,如果关闭这个接口,将解决这个问题并降低CPU利用率.

第四步:使用FED tracing

在第3步讨论的方法的缺点是只可以解析列队中最后一个包,也许并不是真正导致CPU高的数据包.

有一个更好的排错方法是使用 FED tracing, 可以将FED送上CPU的数据包进行抓包.

=====

步骤:

=====

1/ 需要enable detail tracking, 才能抓包

3850-2#set trace control fed-punject-detail enable

2/ 调整抓包buffer. 按需调整detail tracing的buffer.默认buffer大小是32768.

3850-2#show mgmt-infra trace settings fed-punject-detail

One shot Trace Settings:

Buffer Name: fed-punject-detail
Default Size: 32768
Current Size: 32768
Traces Dropped due to internal error: No
Total Entries Written: 0
One shot mode: No
One shot and full: No
Disabled: False

可以通过以下命令修改buffer大小

3850-2#set trace control fed-punject-detail buffer-size

3850-2#set trace control fed-punject-detail buffer-size ?
<8192-67108864> The new desired buffer size, in bytes
default Reset trace buffer size to default

3/ 添加抓包filter. 你可以添加不同的filter,并且使用match all 或者match any来混合使用这些filter进行抓包.

3850-2#set trace fed-punject-detail direction rx filter_add

3850-2#set trace fed-punject-detail direction rx filter_add ?
cpu-queue rxq 0..31
field field
offset offset

3850-2#set trace fed-punject-detail direction rx ?

Clear_all Clear all debug configured

Dump_all Dump all debug info: on

Dump_all_off Dump all debug info: off

Filter_add Add debug filter condition

Filter_clear Clear a debug filter configured

Filter_disable Disable configured filter condition(s)

Filter_enable Enable configured filter condition(s)

Match_all Match all configured filter conditions

Match_any Match any configured filter condition

Show_all Show all debug configured

在第二步案例分析中, 我们看到Queue16有大量数据包送上CPU, 所以需要查看这个queue,看看哪些数据包被送上CPU了.

使用以下命令去设置查看哪个cpu queue

3850-2#set trace fed-punject-detail direction rx filter_add cpu-queue

例如:
3850-2#set trace fed-punject-detail direction rx filter_add cpu-queue 16 16

使用match all 或者match any来混合使用已经定义了的filter来进行抓包

3850-2#set trace fed-punject-detail direction rx match_all
3850-2#set trace fed-punject-detail direction rx filter_enable

4/显示抓到的数据包

使用show mgmt-infra trace messages fed-punject-detail命令查看抓到的包.

3850-2#show mgmt-infra trace messages fed-punject-detail
[11/25/13 07:05:53.814 UTC 2eb0c9 5661]
00 00 00 00 00 4e 00 40 07 00 02 08 00 00 51 3b
00 00 00 00 00 01 00 00 03 00 00 00 00 00 00 01
00 00 00 00 20 00 00 0e 00 00 00 00 00 01 00 74
00 00 00 04 00 54 41 02 00 00 00 00 00 00 00 00

[11/25/13 07:05:53.814 UTC 2eb0ca 5661]
ff ff ff ff ff ff aa bb cc dd 00 00 08 06 00 01
08 00 06 04 00 01 aa bb cc dd 00 00 c0 a8 01 0a
ff ff ff ff ff ff c0 a8 01 14 00 01 02 03 04 05
06 07 08 09 0a 0b 0c 0d 0e 0f 10 11 f6 b9 10 32
[11/25/13 07:05:53.814 UTC 2eb0cb 5661] Frame descriptors:
[11/25/13 07:05:53.814 UTC 2eb0cc 5661]
=========
fdFormat=0x4 systemTtl=0xe
loadBalHash1=0x8 loadBalHash2=0x8
spanSessionMap=0x0 forwardingMode=0x0
destModIndex=0x0 skipIdIndex=0x4
srcGpn=0x54 qosLabel=0x41
srcCos=0x0 ingressTranslatedVlan=0x3
bpdu=0x0 spanHistory=0x0
sgt=0x0 fpeFirstHeaderType=0x0
srcVlan=0x1 rcpServiceId=0x2
wccpSkip=0x0 srcPortLeIndex=0xe
cryptoProtocol=0x0 debugTagId=0x0
vrfId=0x0 saIndex=0x0
pendingAfdLabel=0x0 destClient=0x1
appId=0x0 finalStationIndex=0x74
decryptSuccess=0x0 encryptSuccess=0x0
rcpMiscResults=0x0 stackedFdPresent=0x0
spanDirection=0x0 egressRedirect=0x0
redirectIndex=0x0 exceptionLabel=0x0
destGpn=0x0 inlineFd=0x0
suppressRefPtrUpdate=0x0 suppressRewriteSideEfects=0x0
cmi2=0x0 currentRi=0x1
currentDi=0x513b dropIpUnreachable=0x0
srcZoneId=0x0 srcAsicId=0x0
originalDi=0x0 originalRi=0x0
srcL3IfIndex=0x2 dstL3IfIndex=0x0
dstVlan=0x0 frameLength=0x40
fdCrc=0x7 tunnelSpokeId=0x0

=========
[11/25/13 07:05:53.814 UTC 2eb0cd 5661]
[11/25/13 07:05:53.814 UTC 2eb0ce 5661] PUNT PATH (fed_punject_rx_process_packet:
830):RX: Q: 16, Tag: 65561
[11/25/13 07:05:53.814 UTC 2eb0cf 5661] PUNT PATH (fed_punject_get_physical_iif:
579):RX: Physical IIF-id 0x104d88000000033
[11/25/13 07:05:53.814 UTC 2eb0d0 5661] PUNT PATH (fed_punject_get_src_l3if_index:
434):RX: L3 IIF-id 0x101b6800000004f
[11/25/13 07:05:53.814 UTC 2eb0d1 5661] PUNT PATH (fed_punject_fd_2_pds_md:478):
RX: l2_logical_if = 0x0
[11/25/13 07:05:53.814 UTC 2eb0d2 5661] PUNT PATH (fed_punject_get_source_cos:638):
RX: Source Cos 0
[11/25/13 07:05:53.814 UTC 2eb0d3 5661] PUNT PATH (fed_punject_get_vrf_id:653):
RX: VRF-id 0
[11/25/13 07:05:53.814 UTC 2eb0d4 5661] PUNT PATH (fed_punject_get_src_zoneid:667):
RX: Zone-id 0
[11/25/13 07:05:53.814 UTC 2eb0d5 5661] PUNT PATH (fed_punject_fd_2_pds_md:518):
RX: get_src_zoneid failed
[11/25/13 07:05:53.814 UTC 2eb0d6 5661] PUNT PATH (fed_punject_get_acl_log_direction:
695): RX: : Invalid CMI2
[11/25/13 07:05:53.814 UTC 2eb0d7 5661] PUNT PATH (fed_punject_fd_2_pds_md:541):RX:
get_acl_log_direction failed
[11/25/13 07:05:53.814 UTC 2eb0d8 5661] PUNT PATH (fed_punject_get_acl_full_direction:
724):RX: DI 0x513b ACL Full Direction 1
[11/25/13 07:05:53.814 UTC 2eb0d9 5661] PUNT PATH (fed_punject_get_source_sgt:446):
RX: Source SGT 0
[11/25/13 07:05:53.814 UTC 2eb0da 5661] PUNT PATH (fed_punject_get_first_header_type:680):
RX: FirstHeaderType 0
[11/25/13 07:05:53.814 UTC 2eb0db 5661] PUNT PATH (fed_punject_rx_process_packet:916):
RX: fed_punject_pds_send packet 0x1f00 to IOSd with tag 65561
[11/25/13 07:05:53.814 UTC 2eb0dc 5661] PUNT PATH (fed_punject_rx_process_packet:744):
RX: **** RX packet 0x2360 on qn 16, len 128 ****
[11/25/13 07:05:53.814 UTC 2eb0dd 5661]
buf_no 0 buf_len 128

以上输出提供了足够信息,让我们知道数据包从哪里来 ,以及含有什么信息.

ff ff ff ff ff ff - 目的 MAC地址

aa bb cc dd 00 00 - 源 MAC地址

现在可以根据源MAC地址来找到相应的接口.

在log中, 也包含了非常有用的信息:

[11/25/13 07:05:53.814 UTC 2eb0ce 5661] PUNT PATH (fed_punject_rx_process_packet:
830):RX: Q: 16, Tag: 65561
[11/25/13 07:05:53.814 UTC 2eb0cf 5661] PUNT PATH (fed_punject_get_physical_iif:
579):RX: Physical IIF-id 0x104d88000000033

通过第一条log很容易看到数据包来自哪个queue和tag.

第二条log更有用, 因为它包含了源接口的IIF-ID, 可以使用以下命令找到源接口.

3850-2#show platform port-asic ifm iif-id 0x0104d88000000033
Interface Table
Interface IIF-ID : 0x0104d88000000033
Interface Name : Gi2/0/20
Interface Block Pointer : 0x514d2f70
Interface State : READY
Interface Stauts : IFM-ADD-RCVD, FFM-ADD-RCVD
Interface Ref-Cnt : 6
Interface Epoch : 0
Interface Type : ETHER
Port Type : SWITCH PORT
Port Location : LOCAL
Slot : 2
Unit : 20
Slot Unit : 20
Acitve : Y
SNMP IF Index : 22
GPN : 84
EC Channel : 0
EC Index : 0
ASIC : 0
ASIC Port : 14
Port LE Handle : 0x514cd990
Non Zero Feature Ref Counts
FID : 48(AL_FID_L2_PM), Ref Count : 1
FID : 77(AL_FID_STATS), Ref Count : 1
FID : 51(AL_FID_L2_MATM), Ref Count : 1
FID : 13(AL_FID_SC), Ref Count : 1
FID : 26(AL_FID_QOS), Ref Count : 1
Sub block information
FID : 48(AL_FID_L2_PM), Private Data : 0x54072618
FID : 26(AL_FID_QOS), Private Data : 0x514d31b8

Fisheryu · ‎04-09-2015

what a helpful tips, thank you so much!

Fisheryu · ‎04-09-2015

what a helpful tips, thank you so much!

jing zhang · ‎04-11-2015

绝对实用，赞一个！

suzhouxiaoniu · ‎04-19-2015

CPU排错的东西，一个都不能放过，谢谢分享

jiangygu · ‎07-23-2015

赞赞赞~

cpmld-199 · ‎07-24-2015

学习了，谢谢楼主分享。

ke hua · ‎07-28-2015

学习了，谢谢楼主分享。:)

cuilingfeng · ‎07-30-2015

厉害~ 没用过的排错方法:)

Pengfei Yu · ‎03-29-2016

:o:o:o:o:o:o:o:o我终于找到这篇文章的源头的了！！！
我只想说，真的是“精华”啊，牛。
另，我想问问这些详细具体的命令，与一下命令输出内容是否一样的？这样即可简化排查难度
“
ter len 0
debug platform packet all receive buffer
-------5s-------
undebug all
show platform cpu packet buffered
”

siyxiao · ‎03-29-2016

顶！d=====(￣▽￣*)b！

jm.nie · ‎07-08-2019

今天早上遇到3850的CPU100%的情况，:L:L:L:L:L:L

牛W牛 · ‎07-14-2019

一大早過來學習下，謝謝分享:D

jackyduys · ‎09-28-2020

今天遇到个纯粹就是fed高：
Core 0: CPU utilization for five seconds: 61%; one minute: 75%; five minutes: 77%
Core 1: CPU utilization for five seconds: 73%; one minute: 80%; five minutes: 80%
Core 2: CPU utilization for five seconds: 95%; one minute: 76%; five minutes: 76%
Core 3: CPU utilization for five seconds: 95%; one minute: 95%; five minutes: 92%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 1490 1055 1412 0.00 0.00 0.00 0 init
2 0 124 0 0.00 0.00 0.00 0 kthreadd
3 3530 405190 8 0.00 0.00 0.00 0 migration/0
4 0 3 0 0.00 0.00 0.00 0 sirq-high/0
5 4211285 245477568 5 0.00 0.00 0.00 0 sirq-timer/0
6 0 8 0 0.00 0.00 0.00 0 sirq-net-tx/0
7 0 17216 0 0.00 0.00 0.00 0 sirq-net-rx/0
8 560 757818 0 0.00 0.00 0.00 0 sirq-block/0
9 0 8 0 0.00 0.00 0.00 0 sirq-block-iopo
10 40 10861 3 0.00 0.00 0.00 0 sirq-tasklet/0
11 635802 920556573 5 0.00 0.00 0.00 0 sirq-sched/0
12 64300 1166897 55 0.00 0.00 0.00 0 sirq-hrtimer/0
13 4001943 224533536 11 0.00 0.00 0.00 0 sirq-rcu/0
14 8900 1100416 8 0.00 0.00 0.00 0 desched/0
15 1820 197746 9 0.00 0.00 0.00 0 migration/1
16 0 15 0 0.00 0.00 0.00 0 sirq-high/1
17 1704820 235022513 0 0.00 0.01 0.03 0 sirq-timer/1
18 0 20 0 0.00 0.00 0.00 0 sirq-net-tx/1
19 10 56968 0 0.00 0.00 0.00 0 sirq-net-rx/1
20 1170 301835 3 0.00 0.00 0.00 0 sirq-block/1
21 0 20 0 0.00 0.00 0.00 0 sirq-block-iopo
22 0 21 0 0.00 0.00 0.00 0 sirq-tasklet/1
23 790360 113697071 0 0.00 0.00 0.00 0 sirq-sched/1
24 0 699286 0 0.00 0.00 0.00 0 sirq-hrtimer/1
25 2052040 200150870 1 0.00 0.00 0.00 0 sirq-rcu/1
26 6040 552790 10 0.00 0.00 0.00 0 desched/1
27 4540 464919 9 0.00 0.00 0.00 0 migration/2
28 0 27 0 0.00 0.00 0.00 0 sirq-high/2
29 133730 151828093 0 0.00 0.00 0.01 0 sirq-timer/2
30 0 31 0 0.00 0.00 0.00 0 sirq-net-tx/2
31 40 149638 0 0.00 0.00 0.00 0 sirq-net-rx/2
32 330 91140 3 0.00 0.00 0.00 0 sirq-block/2
33 0 32 0 0.00 0.00 0.00 0 sirq-block-iopo
34 0 33 0 0.00 0.00 0.00 0 sirq-tasklet/2
35 55820 753711334 0 0.00 0.00 0.00 0 sirq-sched/2
36 0 536992 0 0.00 0.00 0.00 0 sirq-hrtimer/2
37 189710 138523848 0 0.00 0.00 0.00 0 sirq-rcu/2
38 4020 583101 6 0.00 0.00 0.00 0 desched/2
39 4950 578558 8 0.00 0.00 0.00 0 migration/3
40 0 39 0 0.00 0.00 0.00 0 sirq-high/3
41 280 118155725 0 0.00 0.00 0.01 0 sirq-timer/3
42 0 45 0 0.00 0.00 0.00 0 sirq-net-tx/3
43 0 232698 0 0.00 0.00 0.00 0 sirq-net-rx/3
44 280 60795 4 0.00 0.00 0.00 0 sirq-block/3
45 0 44 0 0.00 0.00 0.00 0 sirq-block-iopo
46 0 45 0 0.00 0.00 0.00 0 sirq-tasklet/3
47 150 607173814 0 0.00 0.00 0.00 0 sirq-sched/3
48 0 430129 0 0.00 0.00 0.00 0 sirq-hrtimer/3
49 50 111716049 0 0.00 0.00 0.00 0 sirq-rcu/3
50 5810 796565 7 0.00 0.00 0.00 0 desched/3
51 711540 57398656 12 0.00 0.00 0.00 0 events/0
52 37540 27380537 1 0.00 0.00 0.00 0 events/1
53 4680 27380140 0 0.00 0.00 0.00 0 events/2
54 290 27301010 0 0.00 0.00 0.00 0 events/3
55 50 2954 16 0.00 0.00 0.00 0 khelper
58 0 55 0 0.00 0.00 0.00 0 netns
60 0 60 0 0.00 0.00 0.00 0 async/mgr
158 102410 4563397 22 0.00 0.00 0.00 0 sync_supers
160 24640 5476740 4 0.00 0.00 0.00 0 bdi-default
162 3570 598153 5 0.00 0.00 0.00 0 kblockd/0
163 240 237655 1 0.00 0.00 0.00 0 kblockd/1
164 50 73357 0 0.00 0.00 0.00 0 kblockd/2
165 50 49239 1 0.00 0.00 0.00 0 kblockd/3
171 0 67 0 0.00 0.00 0.00 0 ata/0
172 0 66 0 0.00 0.00 0.00 0 ata/1
173 0 65 0 0.00 0.00 0.00 0 ata/2
174 0 66 0 0.00 0.00 0.00 0 ata/3
175 0 67 0 0.00 0.00 0.00 0 ata_aux
183 0 70 0 0.00 0.00 0.00 0 khubd
208 0 88 0 0.00 0.00 0.00 0 rpciod/0
209 0 88 0 0.00 0.00 0.00 0 rpciod/1
210 0 77 0 0.00 0.00 0.00 0 rpciod/2
211 0 76 0 0.00 0.00 0.00 0 rpciod/3
227 0 77 0 0.00 0.00 0.00 0 kswapd0
228 0 78 0 0.00 0.00 0.00 0 aio/0
229 0 79 0 0.00 0.00 0.00 0 aio/1
230 0 80 0 0.00 0.00 0.00 0 aio/2
231 0 81 0 0.00 0.00 0.00 0 aio/3
232 0 82 0 0.00 0.00 0.00 0 nfsiod
234 0 83 0 0.00 0.00 0.00 0 crypto/0
235 0 84 0 0.00 0.00 0.00 0 crypto/1
236 0 85 0 0.00 0.00 0.00 0 crypto/2
237 0 86 0 0.00 0.00 0.00 0 crypto/3
787 1890 1311820 1 0.00 0.00 0.00 0 pata_octeon_cf
788 10 97 103 0.00 0.00 0.00 0 scsi_eh_0
794 3500 514 6809 0.00 0.00 0.00 0 mtdblockd
820 359700 55952381 6 0.00 0.02 0.04 0 octeon-ethernet
854 0 96 0 0.00 0.00 0.00 0 kstriped
856 0 97 0 0.00 0.00 0.00 0 ksnapd
934 390 1211 322 0.00 0.00 0.00 0 udevd
1229 31230 5476765 5 0.00 0.01 0.00 0 flush-8:0
1536 210 4404 47 0.00 0.00 0.00 0 loop1
1561 70 4254 16 0.00 0.00 0.00 0 loop2
1586 50 2136 23 0.00 0.00 0.00 0 loop3
1611 90 2886 31 0.00 0.00 0.00 0 loop4
1636 100 2454 40 0.00 0.00 0.00 0 loop5
1681 10 559 17 0.00 0.00 0.00 0 loop6
2375 10 1003 9 0.00 0.00 0.00 0 udevd
2376 10 978 10 0.00 0.00 0.00 0 udevd
3275 120 1106 108 0.00 0.00 0.00 0 dbus-daemon
3653 120 1169 102 0.00 0.00 0.00 0 klogd
3665 20 972 20 0.00 0.00 0.00 0 portmap
3734 0 953 0 0.00 0.00 0.00 0 sshd
3746 0 944 0 0.00 0.00 0.00 0 xinetd
3750 0 945 0 0.00 0.00 0.00 0 xinetd
3784 0 112 0 0.00 0.00 0.00 0 lockd
3785 80 7720 10 0.00 0.00 0.00 0 nfsd
3786 80 7724 10 0.00 0.00 0.00 0 nfsd
3787 70 7721 9 0.00 0.00 0.00 0 nfsd
3788 70 7725 9 0.00 0.00 0.00 0 nfsd
3789 50 7725 6 0.00 0.00 0.00 0 nfsd
3790 110 7728 14 0.00 0.00 0.00 0 nfsd
3791 80 7726 10 0.00 0.00 0.00 0 nfsd
3792 30 7726 3 0.00 0.00 0.00 0 nfsd
3798 0 975 0 0.00 0.00 0.00 0 rpc.mountd
5348 3494715 135571482 89 0.00 0.02 0.03 0 system_mgr
5577 533870 30127585 17 0.00 0.00 0.00 1088 slproc
5653 3301954 855449408 575 75.3 74.9 74.9 1088 fed
5654 3578347 774328973 12 0.10 0.09 0.10 0 platform_mgr
5655 568688 970860191 31 0.00 0.04 0.05 0 stack-mgr
6148 601022 54692026 89 0.00 0.00 0.01 0 idope.py
6150 1781340 110948330 16 0.00 0.02 0.04 0 ns_oir_proxy
6151 1776680 110956820 16 0.00 0.02 0.04 0 profiled
6152 2270890 2628487 863 0.00 0.00 0.03 0 oom_poll.sh
6153 3161220 165283670 19 0.05 0.01 0.03 0 obfld
6154 1782900 110956882 16 0.00 0.00 0.01 0 consoled
6155 1821482 138695468 44 0.00 0.00 0.00 0 console_relay
6156 1765740 110956909 15 0.00 0.00 0.00 0 csprovider
6157 1768560 111230689 15 0.00 0.00 0.00 0 system_mgr_cli
6159 3316412 222398560 34 0.00 0.00 0.02 0 plogd
6160 1771600 110957162 15 0.00 0.02 0.00 0 psdprov
6161 748205 364778884 46 0.63 0.35 0.30 0 pdsd
6162 2264630 117251466 19 0.00 0.00 0.00 0 gold_slave
6163 3537260 115694679 30 0.00 0.01 0.03 0 osinfo-provider
6164 1764850 110956841 15 0.00 0.00 0.00 0 oscore_p
6166 2324530 113842185 20 0.00 0.01 0.03 0 netd
6168 4231245 135561048 94 0.00 0.04 0.02 0 mem_mgmt
6169 2537170 138321977 18 0.00 0.00 0.00 0 mgmte_tap
6172 1781750 110982119 16 0.00 0.00 0.00 0 installer
6173 1438401 247568902 144 0.00 0.27 0.04 0 ffm
6175 212803 159053618 1432 0.00 0.81 0.93 0 eicored
6177 1796900 111874335 16 0.00 0.02 0.04 0 dtmgr
6186 891812 138165281 1187 0.14 0.16 0.14 0 cpumemd
6193 2519800 138778479 18 0.00 0.00 0.01 0 liin_tap
6198 1798680 111071727 16 0.00 0.02 0.04 0 cli_agent
6202 189915 378149245 23 0.00 0.04 0.04 0 snmp_subagent
6205 2353840 138330764 17 0.00 0.00 0.00 0 erm_p
6212 2226390 138331120 16 0.00 0.00 0.00 0 ngdumper_provid
6214 2221920 138331056 16 0.00 0.00 0.02 0 sysinfo_p
6248 531300 30113517 17 0.00 0.00 0.02 0 fs_cmd
8488 470 1368 343 0.00 0.00 0.00 0 rollback_timer.
8687 0 1358 0 0.00 0.00 0.00 0 rollback_timer.
8689 160 1358 117 0.00 0.00 0.00 0 xlogger
8697 0 1362 0 0.00 0.00 0.00 0 inotifywait
8892 130 1177 110 0.00 0.00 0.00 0 nova_sntp.sh
9022 0 1175 0 0.00 0.00 0.00 0 nova_sntp.sh
9023 160 1175 136 0.00 0.00 0.00 0 xlogger
9026 0 1180 0 0.00 0.00 0.00 0 sntp
9241 3927962 240756311 24 0.19 0.09 0.09 0 wcm
9242 1414292 912254334 6 0.00 0.02 0.04 0 table_mgr
9243 80 1378 58 0.00 0.00 0.00 0 wnweb.sh
9244 1799090 110959717 16 0.05 0.05 0.01 0 iifd
9245 3182537 315168539 160 5.23 5.07 4.75 0 iosd
9246 1792450 111237374 16 0.00 0.00 0.00 0 ha_mgr
10200 2614732 135622672 50 0.00 0.01 0.04 0 wnweb_paster.py
10201 340 1378 246 0.00 0.00 0.00 0 xlogger
11787 0 2628487 0 0.00 0.00 0.00 0 sleep

xingzhetan · ‎11-12-2020

很详细很实用

bo chen · ‎11-28-2020

很实用的文章，收藏了。