2010-08-06 09:41 PM
今回紹介する事例は、ACE module で FT の設定を行い、Traffic を流すと、ACE module を搭載している Switch (supervisor) の CPU 使用率が高くなるという現象です。
Switch 2 台にそれぞれ ACE module を挿入し、使用している user が多いですが、機器の都合上、今回は 1 台の Switch 上に 2 枚の ACE module を挿入し確認します。
# 構成
# Switch 設定 (今回の試験に関係のある部分のみ抜粋)
sup720#sh run | i svclc svclc autostate svclc multiple-vlan-interfaces svclc module 4 vlan-group 1 svclc module 6 vlan-group 1 svclc vlan-group 1 771-773 sup720# sup720#sh run int gi 1/31 Building configuration... Current configuration : 162 bytes ! interface GigabitEthernet1/31 switchport switchport access vlan 771 switchport mode access no ip address load-interval 30 end sup720#sh run int gi 1/35 Building configuration... Current configuration : 144 bytes ! interface GigabitEthernet1/35 switchport switchport access vlan 772 switchport mode access no ip address end sup720#sh run int vl 773 Building configuration... Current configuration : 66 bytes ! interface Vlan773 ip address 192.168.73.250 255.255.255.0 end sup720# |
# ACE 設定(active)
ACE20a/Admin# sh run Generating configuration.... peer hostname ACE20b hostname ACE20a boot system image:c6ace-t1k9-mz.A2_3_1.bin resource-class sticky limit-resource all minimum 0.00 maximum unlimited limit-resource sticky minimum 1.00 maximum unlimited context Admin member sticky access-list all line 8 extended permit ip any any rserver host sv1 ip address 192.168.72.50 inservice serverfarm host sf rserver sv1 inservice sticky http-cookie ace ace_cookie serverfarm sf class-map match-all vip 2 match virtual-address 192.168.71.100 any policy-map type loadbalance first-match lb class class-default sticky-serverfarm ace_cookie policy-map multi-match client-vips class vip loadbalance vip inservice loadbalance policy lb loadbalance vip icmp-reply access-group input all interface vlan 771 ip address 192.168.71.251 255.255.255.0 alias 192.168.71.250 255.255.255.0 peer ip address 192.168.71.252 255.255.255.0 service-policy input client-vips no shutdown interface vlan 772 ip address 192.168.72.251 255.255.255.0 alias 192.168.72.250 255.255.255.0 peer ip address 192.168.72.252 255.255.255.0 no shutdown ft interface vlan 773 ip address 192.168.73.251 255.255.255.0 peer ip address 192.168.73.252 255.255.255.0 no shutdown ft peer 1 heartbeat interval 300 heartbeat count 10 ft-interface vlan 773 ft group 1 peer 1 priority 110 associate-context Admin inservice |
# ACE 設定(standby)
ACE20b/Admin# sh run Generating configuration.... peer hostname ACE20a hostname ACE20b boot system image:c6ace-t1k9-mz.A2_3_1.bin resource-class sticky limit-resource all minimum 0.00 maximum unlimited limit-resource sticky minimum 1.00 maximum unlimited context Admin member sticky access-list all line 8 extended permit ip any any rserver host sv1 ip address 192.168.72.50 inservice serverfarm host sf rserver sv1 inservice sticky http-cookie ace ace_cookie serverfarm sf class-map match-all vip 2 match virtual-address 192.168.71.100 any policy-map type loadbalance first-match lb class class-default sticky-serverfarm ace_cookie policy-map multi-match client-vips class vip loadbalance vip inservice loadbalance policy lb loadbalance vip icmp-reply access-group input all interface vlan 771 ip address 192.168.71.252 255.255.255.0 alias 192.168.71.250 255.255.255.0 peer ip address 192.168.71.251 255.255.255.0 service-policy input client-vips no shutdown interface vlan 772 ip address 192.168.72.252 255.255.255.0 alias 192.168.72.250 255.255.255.0 peer ip address 192.168.72.251 255.255.255.0 no shutdown ft interface vlan 773 ip address 192.168.73.252 255.255.255.0 peer ip address 192.168.73.251 255.255.255.0 no shutdown ft peer 1 heartbeat interval 300 heartbeat count 10 ft-interface vlan 773 ft group 1 peer 1 peer priority 110 associate-context Admin inservice |
# client からの Traffic
# httperf --server 192.168.71.100 --port 80 --uri /index.html --rate 700 --num-conn 20000 --num-call 10 |
# Switch の rate と cpu 使用率
sup720#sh int gi 1/31 | i rate Queueing strategy: fifo 30 second input rate 4850000 bits/sec, 5401 packets/sec 30 second output rate 11670000 bits/sec, 4687 packets/sec sup720#sh proc c sort 5sec CPU utilization for five seconds: 19%/14%; one minute: 6%; five minutes: 3% !___ 5Mbps 程度の Traffic が入ってきただけで、CPU 使用率が 20% 程度まで上昇しています。 !___ また、割り込み処理で 14% の CPU 使用率を消費しています。 PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 123 838468 12765193 65 4.07% 1.20% 0.56% 0 IP Input 9 1960588 19609098 99 0.15% 0.12% 0.14% 0 ARP Input 3 324 566 572 0.07% 0.09% 0.06% 0 Exec 118 2662528 2728514 975 0.07% 0.04% 0.02% 0 CDP Protocol 176 320172 281876 1135 0.07% 0.00% 0.00% 0 SNMP ENGINE sup720#sh int vl 773 Vlan773 is up, line protocol is up Hardware is EtherSVI, address is 0016.9c6d.e580 (bia 0016.9c6d.e580) Internet address is 192.168.73.250/24 MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive not supported ARP type: ARPA, ARP Timeout 04:00:00 Last input 00:00:00, output 00:00:23, output hang never Last clearing of "show interface" counters never Input queue: 0/75/2628/2005 (size/max/drops/flushes); Total output drops: 0 !___ vlan 773 で input queue drop が発生 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 482000 bits/sec, 254 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec L2 Switched: ucast: 6928 pkt, 1221580 bytes - mcast: 1295873 pkt, 293490902 bytes L3 in Switched: ucast: 0 pkt, 0 bytes - mcast: 0 pkt, 0 bytes mcast L3 out Switched: ucast: 0 pkt, 0 bytes mcast: 0 pkt, 0 bytes 2609114 packets input, 519657046 bytes, 0 no buffer Received 2609114 broadcasts (0 IP multicasts) 0 runts, 0 giants, 9 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 141 packets output, 11302 bytes, 0 underruns 0 output errors, 4 interface resets 0 output buffer failures, 0 output buffers swapped out |
vlan 773 で Input queue drop が発生していることから、vlan 773 に流れ込む Traffic が Process switching (CPU 処理) されている可能性が疑われます。 そこで、buffer を確認してみると、下記出力のように dest mac がすべて F の broadcast packet が FT interface 間で流れています。 これは、ACE 間でやり取りを行っている heartbeat packet や connection 情報を同期するための packet になります。 FT の設定を行うと自動的に UDP 9 connection と TCP 1 connection が connection table に登録され、ACE 間の情報のやり取りに使用されます。
# Switch buffer 情報
sup720#show buffers input-interface vl 773 packet Buffer information for Small buffer at 0x44CB77F8 data_area 0x8048DC4, refcount 1, next 0x44D8B898, flags 0x280 linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1 if_input 0x47288B08 (Vlan773), if_output 0x0 (None) inputtime 2d17h (elapsed 12w1d) outputtime 9w4d (elapsed 3w0d), oqnumber 65535 datagramstart 0x8048E3A, datagramsize 60, maximum size 308 mac_start 0x8048E3A, addr_start 0x8048E3A, info_start 0x0 network_start 0x8048E48, transport_start 0x8048E5C, caller_pc 0x41876F04 source: 192.168.73.251, destination: 192.168.73.252, id: 0x1D62, ttl: 254, TOS: 0 prot: 17, source port 50002, destination port 50000 08048E30: FFFF FFFFFFFF ...... 08048E40: 00070E0F 2CA10800 4500002C 1D620000 ....,!..E..,.b.. 08048E50: FE118A16 C0A849FB C0A849FC C352C350 ~...@(I{@(I|CRCP 08048E60: 00180000 000100C0 00000008 00000000 .......@........ 08048E70: 0000012C 000043 ...,..C Buffer information for Medium buffer at 0x44DD3480 data_area 0x81B8384, refcount 1, next 0x44E21A00, flags 0x280 linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1 if_input 0x47288B08 (Vlan773), if_output 0x0 (None) inputtime 00:00:00.000 (elapsed never) outputtime 00:00:00.000 (elapsed never), oqnumber 65535 datagramstart 0x81B83FA, datagramsize 230, maximum size 460 mac_start 0x81B83FA, addr_start 0x81B83FA, info_start 0x0 network_start 0x81B8408, transport_start 0x81B841C, caller_pc 0x41876F04 source: 192.168.73.251, destination: 192.168.73.252, id: 0x3911, ttl: 254, TOS: 0 prot: 17, source port 50302, destination port 50300 081B83F0: FFFF FFFFFFFF ...... 081B8400: 00070E0F 2CA10800 450000D8 39110000 ....,!..E..X9... 081B8410: FE116DBB C0A849FB C0A849FC C47EC47C ~.m;@(I{@(I|D~D| 081B8420: 00C40000 02020280 000100BC 00000000 .D.........<.... 081B8430: 0303000F F7759C39 000FF775 91EB0800 ....wu.9..wu.k.. 081B8440: 45000028 00000000 00060000 C0A84732 E..(........@(G2 081B8450: C0A84764 E6360050 00000000 02F1B52B @(Gdf6.P.....q5+ 081B8460: 50020000 00000000 00000008 05B40206 P............4.. 081B8470: 00080480 24450000 19200030 00000000 ....$E... .0.... 081B8480: 00000000 E488920D 00000000 C0A84832 ....d.......@(H2 081B8490: C0A84732 06010009 0050EA21 7B8CB6F3 @(G2.....Pj!{.6s 081B84A0: 00110000 05B40000 00080480 24450005 .....4......$E.. 081B84B0: 16D00030 00000000 00000000 7B8CB6F3 .P.0........{.6s 081B84C0: 00000000 00000000 00050000 00270000 .............'.. 081B84D0: 30203139 3A32313A 31372047 4D540D0A 0 19:21:17 GMT.. 081B84E0: FF . Buffer information for Medium buffer at 0x501BBD20 data_area 0x821C7E4, refcount 1, next 0x0, flags 0x280 linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1 if_input 0x47288B08 (Vlan773), if_output 0x0 (None) inputtime 00:00:00.000 (elapsed never) outputtime 00:00:00.000 (elapsed never), oqnumber 65535 datagramstart 0x821C85A, datagramsize 230, maximum size 460 mac_start 0x821C85A, addr_start 0x821C85A, info_start 0x0 network_start 0x821C868, transport_start 0x821C87C, caller_pc 0x41876F04 source: 192.168.73.251, destination: 192.168.73.252, id: 0x60AB, ttl: 254, TOS: 0 prot: 17, source port 50302, destination port 50300 0821C850: FFFF FFFFFFFF ...... 0821C860: 00070E0F 2CA10800 450000D8 60AB0000 ....,!..E..X`+.. 0821C870: FE114621 C0A849FB C0A849FC C47EC47C ~.F!@(I{@(I|D~D| 0821C880: 00C40000 02020280 000100BC 00000000 .D.........<.... 0821C890: 0303000F F7759C39 000FF775 91EB0800 ....wu.9..wu.k.. 0821C8A0: 45000028 00000000 00060000 C0A84732 E..(........@(G2 0821C8B0: C0A84764 EE200050 00000000 957C8582 @(Gdn .P.....|.. 0821C8C0: 50020000 00000000 00000008 05B40206 P............4.. 0821C8D0: 00082480 24450000 3ED0004F 00000000 ..$.$E..>P.O.... 0821C8E0: 00000000 7636FBD2 00000000 C0A84832 ....v6{R....@(H2 0821C8F0: C0A84732 06010009 005083BC A57646A5 @(G2.....P.<%vF% 0821C900: 00110000 05B40000 00080480 24450005 .....4......$E.. 0821C910: 16D0004E 00000000 00000000 A57646A5 .P.N........%vF% 0821C920: 204D6F6E 2C203037 204A756E 20323031 Mon, 07 Jun 201 0821C930: 30203139 3A32313A 31372047 4D540D0A 0 19:21:17 GMT.. 0821C940: 57 W |
# FT 設定時の ACE connection 情報
ACE20a/Admin# show conn total current connections : 11 conn-id np dir proto vlan source destination state ----------+--+---+-----+----+---------------------+---------------------+------+ 99 1 in UDP 773 192.168.73.252:50000 192.168.73.251:50002 -- -- - - -- -- -- -- -- 100 1 in UDP 773 192.168.73.252:50100 192.168.73.251:50102 -- -- - - -- -- -- -- -- 101 1 in UDP 773 192.168.73.252:50300 192.168.73.251:50302 -- -- - - -- -- -- -- -- 102 1 in UDP 773 192.168.73.252:50300 192.168.73.251:50301 -- -- - - -- -- -- -- -- 103 1 in UDP 773 192.168.73.252:50200 192.168.73.251:50202 -- -- - - -- -- -- -- -- 97 2 in UDP 773 192.168.73.252:50100 192.168.73.251:50101 -- -- - - -- -- -- -- -- 98 2 in UDP 773 192.168.73.252:50300 192.168.73.251:50302 -- -- - - -- -- -- -- -- 99 2 in UDP 773 192.168.73.252:50300 192.168.73.251:50301 -- -- - - -- -- -- -- -- 100 2 in UDP 773 192.168.73.252:50200 192.168.73.251:50201 -- -- - - -- -- -- -- -- 101 2 in TCP 773 192.168.73.251:32801 192.168.73.252:2000 ESTAB 102 2 out TCP 773 192.168.73.252:2000 192.168.73.251:32801 ESTAB ACE20a/Admin# ACE20a/Admin# |
今回の事象は、ACE 間でやり取りされているFT packet が broadcast であるため、それを受信した vlan 773 で process switching され、CPU 使用率が上昇していました。 これは、vlan 773 を shutdown するか、設定を削除することで回避できます。
# SVI 設定削除後の CPU 使用率
sup720#conf t Enter configuration commands, one per line. End with CNTL/Z. sup720(config)#no int vl 773 sup720(config)#end sup720# sup720#sh int gi 1/31 | i rate Queueing strategy: fifo 30 second input rate 6101000 bits/sec, 6793 packets/sec 30 second output rate 14680000 bits/sec, 5888 packets/sec sup720#sh proc c | i second CPU utilization for five seconds: 2%/1%; one minute: 1%; five minutes: 1% !___ 流れている traffic rate は先ほどより多いにもかかわらず CPU 使用率は低いまま sup720# |
検索バーにキーワード、フレーズ、または質問を入力し、お探しのものを見つけましょう
シスコ コミュニティをいち早く使いこなしていただけるよう役立つリンクをまとめました。みなさんのジャーニーがより良いものとなるようお手伝いします