high CPU utilization issues

benedict dcunha · ‎01-14-2013

Dear All,

I have a 2 cisco core (cisco WS-C6509-E (R7000) processor) and been working for quite sometime.

they are conneted with HSRP with active standby config with a 10 g module for redundancy

just today I see that the cpu utilization went to about 50% and its the same on both cores

its been always between 3 to 5 %

googling arround I found that netdr help in troubleshooting but could not really get something positive in tracing out the cause of the issue

below is the output of sh proc cpu of core B and also core A shows the same

---------

Core-B#sh proc cpu SORted | exclude 0.00

CPU utilization for five seconds: 50%/36%; one minute: 48%; five minutes: 48%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

23 15187040 150730722 100 0.95% 0.29% 0.23% 0 IPC Seat Manager

462 18924316 122501511 154 0.55% 0.31% 0.29% 0 Port manager per

260 16893808 22686974 744 0.47% 0.46% 0.44% 0 CDP Protocol

266 8945852 204119832 43 0.47% 0.42% 0.44% 0 IP Input

459 31188 372100659 0 0.39% 0.37% 0.37% 0 HSRP Common

11 8350916 37153191 224 0.39% 0.43% 0.41% 0 ARP Input

289 8604 764068581 0 0.15% 0.17% 0.15% 0 Ethernet Msec Ti

336 2035844 10887101 186 0.07% 0.08% 0.07% 0 CEF: IPv4 proces

217 244560 2458425 99 0.07% 0.02% 0.02% 0 Compute load avg

51 148740 7014706 21 0.07% 0.07% 0.07% 0 Per-Second Jobs

517 3199180 82983066 38 0.07% 0.08% 0.07% 0 HSRP IPv4

Core-B#sh proc cpu SORted | exclude 0.00

CPU utilization for five seconds: 47%/34%; one minute: 48%; five minutes: 48%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

64 265906122207696984 0 9.43% 9.35% 9.35% 0 Net Input

260 16893816 22687109 744 0.55% 0.47% 0.44% 0 CDP Protocol

266 8945856 204120050 43 0.47% 0.43% 0.44% 0 IP Input

11 8350928 37153215 224 0.39% 0.43% 0.41% 0 ARP Input

459 31188 372100767 0 0.39% 0.37% 0.37% 0 HSRP Common

289 8604 764068870 0 0.15% 0.16% 0.15% 0 Ethernet Msec Ti

517 3199180 82983095 38 0.15% 0.09% 0.08% 0 HSRP IPv4

357 767456 2985099 257 0.07% 0.04% 0.05% 0 HIDDEN VLAN Proc

51 148740 7014718 21 0.07% 0.07% 0.07% 0 Per-Second Jobs

374 33980 186021477 0 0.07% 0.04% 0.05% 0 RADIUS

305 2250804 1193016 1886 0.07% 0.03% 0.02% 0 QOS Stats Gather

336 2035844 10887116 186 0.07% 0.08% 0.07% 0 CEF: IPv4 proces

Core-B#sh proc cpu SORted | exclude 0.00

CPU utilization for five seconds: 47%/34%; one minute: 47%; five minutes: 48%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

64 266072922209089683 0 9.27% 9.34% 9.35% 0 Net Input

8 17847068 1118918 15950 0.87% 0.36% 0.29% 0 Check heaps

260 16894720 22697971 744 0.55% 0.44% 0.44% 0 CDP Protocol

11 8351768 37155008 224 0.47% 0.40% 0.40% 0 ARP Input

459 31188 372109624 0 0.39% 0.37% 0.37% 0 HSRP Common

266 8946628 204131091 43 0.39% 0.39% 0.43% 0 IP Input

289 8604 764092148 0 0.23% 0.17% 0.16% 0 Ethernet Msec Ti

521 4964604 15243190 325 0.15% 0.05% 0.06% 0 SNMP ENGINE

336 2035900 10888915 186 0.15% 0.08% 0.08% 0 CEF: IPv4 proces

51 148748 7015494 21 0.07% 0.07% 0.07% 0 Per-Second Jobs

517 3199224 82985086 38 0.07% 0.08% 0.07% 0 HSRP IPv4

357 767512 2985170 257 0.07% 0.04% 0.05% 0 HIDDEN VLAN Proc

305 2250868 1193044 1886 0.07% 0.02% 0.02% 0 QOS Stats Gather

-----------

the Net Input is very high

apprecite if someone can help and advice

thanks and regards

simon

Arumugam Muthaiah · ‎01-14-2013

Hi,

Based on the output i can see that 34% caused by traffic being interrupt switched by the CPU. High CPU utilization on an interrupt level is primarily caused by packets handled on interrupt level.

Core-B#sh proc cpu SORted | exclude 0.00

CPU utilization for five seconds: 47%/34%; one minute: 47%; five minutes: 48%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

64 266072922209089683 0 9.27% 9.34% 9.35% 0 Net Input

8 17847068 1118918 15950 0.87% 0.36% 0.29% 0 Check heaps

260 16894720 22697971 744 0.55% 0.44% 0.44% 0 CDP Protocol

Inorder to find the packet which is hitting the CPU, we can use the Netdr tool. This is available on the Catalyst 6500 with a Sup720 or Sup32 that allows one to capture packets on the RP or SP inband. The netdr command can be used to capture both Tx and Rx packets in the software switching path.

#debug netdr capture rx

#show netdr capture

Note: This debug command doesnt harm any production. Please take capture the data and share for the investigation.

Refer:

https://supportforums.cisco.com/docs/DOC-15608

Regards,

Aru

*** Please rate if the post is useful ***

Regards, Aru *** Please rate if the post useful ***

benedict dcunha · ‎01-14-2013

Dear Aru,

Really grateful and apprecite your immedite wise reply

by the way here below isa part of the sh netdr capure

-----

------- dump of incoming inband packet -------

interface Vl12, routine mistral_process_rx_packet_inlin, timestamp 14:23:08.485

dbus info: src_vlan 0xC(12), src_indx 0x8A(138), len 0x5A(90)

bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x400C(16396)

E8020400 000C0000 008A0000 5A000000 FE000510 00000008 00000008 400C8075

mistral hdr: req_token 0x0(0), src_index 0x8A(138), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0xC(12)

destmac 59.A3.BC.A1.4D.99, srcmac FE.80.00.00.00.00, protocol 0000

layer 3 data: 4034FCFF 7F204A42 880013C9 60000000 FE800000 00000000

A04C59A3 BCA14D99 0201FFFF 1F500000 00FF0000 00000CFD

00560000 BFEFFFFC 0000FFFF 0000008A 0000400C 1800

------- dump of incoming inband packet -------

interface Vl12, routine mistral_process_rx_packet_inlin, timestamp 14:23:08.485

dbus info: src_vlan 0xC(12), src_indx 0x8A(138), len 0x5A(90)

bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x400C(16396)

F8020400 000C0000 008A0000 5A000000 FE000510 00000008 00000008 400CF075

mistral hdr: req_token 0x0(0), src_index 0x8A(138), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0xC(12)

destmac 59.A3.BC.A1.4D.99, srcmac FE.80.00.00.00.00, protocol 0000

layer 3 data: 4034FCFF 7F204A42 880013C9 60000000 FE800000 00000000

A04C59A3 BCA14D99 0201FFFF 1F500000 00FF0000 00000CFD

00560000 BFEFFFFC 0000FFFF 0000008A 0000400C 1800

------- dump of incoming inband packet -------

interface Vl12, routine mistral_process_rx_packet_inlin, timestamp 14:23:08.485

dbus info: src_vlan 0xC(12), src_indx 0x8A(138), len 0x5A(90)

bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x400C(16396)

08020400 000C0000 008A0000 5A000000 FE000510 00000008 00000008 400CA07A

mistral hdr: req_token 0x0(0), src_index 0x8A(138), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0xC(12)

destmac 59.A3.BC.A1.4D.99, srcmac FE.80.00.00.00.00, protocol 0000

layer 3 data: 4034FCFF 7F204A42 880013C9 60000000 FE800000 00000000

A04C59A3 BCA14D99 0201FFFF 1F500000 00FF0000 00000CFD

00560000 BFEFFFFC 0000FFFF 0000008A 0000400C 1800

------- dump of incoming inband packet -------

interface Vl12, routine mistral_process_rx_packet_inlin, timestamp 14:23:08.485

dbus info: src_vlan 0xC(12), src_indx 0x8A(138), len 0x5A(90)

bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x400C(16396)

30020400 000C0000 008A0000 5A000000 FE000510 00000008 00000008 400C087A

mistral hdr: req_token 0x0(0), src_index 0x8A(138), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0xC(12)

destmac 59.A3.BC.A1.4D.99, srcmac FE.80.00.00.00.00, protocol 0000

layer 3 data: 4034FCFF 7F204A42 880013C9 60000000 FE800000 00000000

A04C59A3 BCA14D99 0201FFFF 1F500000 00FF0000 00000CFD

00560000 BFEFFFFC 0000FFFF 0000008A 0000400C 1800

------- dump of incoming inband packet -------

interface Vl12, routine mistral_process_rx_packet_inlin, timestamp 14:23:08.485

dbus info: src_vlan 0xC(12), src_indx 0x8A(138), len 0x5A(90)

bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x400C(16396)

48020400 000C0000 008A0000 5A000000 FE000510 00000008 00000008 400CE07F

mistral hdr: req_token 0x0(0), src_index 0x8A(138), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0xC(12)

destmac 59.A3.BC.A1.4D.99, srcmac FE.80.00.00.00.00, protocol 0000

layer 3 data: 4034FCFF 7F204A42 880013C9 60000000 FE800000 00000000

A04C59A3 BCA14D99 0201FFFF 1F500000 00FF0000 00000CFD

00560000 BFEFFFFC 0000FFFF 0000008A 0000400C 1800

------- dump of incoming inband packet -------

interface Vl12, routine mistral_process_rx_packet_inlin, timestamp 14:23:08.485

dbus info: src_vlan 0xC(12), src_indx 0x8A(138), len 0x5A(90)

bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x400C(16396)

90020400 000C0000 008A0000 5A000000 FE000510 00000008 00000008 400C6870

mistral hdr: req_token 0x0(0), src_index 0x8A(138), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0xC(12)

destmac 59.A3.BC.A1.4D.99, srcmac FE.80.00.00.00.00, protocol 0000

layer 3 data: 4034FCFF 7F204A42 880013C9 60000000 FE800000 00000000

A04C59A3 BCA14D99 0201FFFF 1F500000 00FF0000 00000CFD

00560000 BFEFFFFC 0000FFFF 0000008A 0000400C 1800

---------------------

is there so packets comming from vlan 12 if i am right

and the src mac is FE:80:00:00:00:00:00

and how does one disable the netdr capture

can I find the source ip from this

im sorry as am a novice in cisco

apprecite your kind help and advice

regards

simon

once again thanks in advance and god bless you

benedict dcunha · ‎01-14-2013

Dear Aru,

As per you advice and te link you mentioned I did manage to tract the inface on the core

its a gibex 1gb fibre port connected to a 48 port access switch

so i did shut the port and the utilization went to normal like a charm

but i am just stuck as to how i would find the user pc or pcs ( i mean the source IP responsible )

would really apprecite your kind help

Thanks and regards

simon

Arumugam Muthaiah · ‎01-15-2013

Hi Simon,

Sorry for the delay. Its good to hear that the problem has been fixed. I have analyzed the captured shared by you and see the detail below,

------- dump of incoming inband packet -------

interface Vl12, routine mistral_process_rx_packet_inlin, timestamp 14:23:08.485

dbus info: src_vlan 0xC(12), src_indx 0x8A(138), len 0x5A(90)

bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x400C(16396)

90020400 000C0000 008A0000 5A000000 FE000510 00000008 00000008 400C6870

mistral hdr: req_token 0x0(0), src_index 0x8A(138), rx_offset 0x76(118)

requeue 0, obl_pkt 0, vlan 0xC(12)

destmac 59.A3.BC.A1.4D.99, srcmac FE.80.00.00.00.00, protocol 0000

layer 3 data: 4034FCFF 7F204A42 880013C9 60000000 FE800000 00000000

A04C59A3 BCA14D99 0201FFFF 1F500000 00FF0000 00000CFD

00560000 BFEFFFFC 0000FFFF 0000008A 0000400C 1800

Interface traffic is coming from --> Vlan12
Ingress vlan for of the traffic --> 0xC(12) vlan 12
Destination index where the traffic is being sent --> 0x400C

You can track down the physical port where this traffic is sourced from by tracking down the mac-address FE.80.00.00.00.00

#show mac-address address FE.80.00.00.00.00 --> this will show the source interface. You can also look at ingress location of this traffic by taking the source index/ltl index

#remote command switch test mcast ltl index 8A --> which will show you source interface for reconfirmation

On our capture, i can see that flood bit is 1, If the packet that is captured has the flood bit set in the netdr capture, this traffic is being flooded to all ports within the vlan.

In order to see where this packet would be sent we need to add 8 to first value in the index (8 + 0x400C = C00C)

6500-2#remote command switch test mcast ltl index C00C --> will show you the interface

Please verify the access switch interfaces error counter / logs / traffic utilization and see if there is any misbehavior.

Regards,

Aru

*** Please rate if the post is useful ***

Regards, Aru *** Please rate if the post useful ***

benedict dcunha · ‎01-17-2013

Dear amu,

First of all you dont need to say sorry for the delay..

u r the one who was so helpful and really so grateful with your wise reply

and also there was no delay

actually after the link which you had mentioned trouble shootin was more ea=

sier

first i found the access switch which was connected and then show mac-addre=

ss address inter.... showed me the interface where the source pc was connec=

ted and goin on the core=20

show arp | include mac id resulted in showing the ip

maybe i feel that maybe not the perfect troubleshooting but you professiona=

ls have to excuse me.

by the way in future i guess I be able to handle such issues.

by the way anu a big favor if i can ask you .

from your expertise do you know of any professional paid software which wou=

ld be able to help us trouble shoot the source ip address .

in our network we dont have dhcp .. only static ips

once again thanks a million and god bless you

regards

simon