Hight CPU utilization 7603 ws-sup32

[Pascal Faucher]

Hi,

Since a couple of months, I have some CPU problem.

there is no hight process :

l#sh processes cpu | exclude 0.00
CPU utilization for five seconds: 39%/30%; one minute: 32%; five minutes: 32%
PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
130    52713208 240359987        219  0.95%  0.91%  1.00%   0 IP Input        
202     1811824   1218190       1487  0.07%  0.06%  0.05%   0 HIDDEN VLAN Proc
338    20873364    219225      95214  7.11%  0.94%  0.71%   0 BGP Scanner  

total amount of traffic on the router around  700Mb-1Gb out /  300Mb In

cisco CISCO7603 (R7000) processor (revision 1.1) with 458752K/65536K bytes of memory.
Processor board ID FOX090308DT
R7000 CPU at 300Mhz, Implementation 0x27, Rev 3.3, 256KB L2, 1024KB L3 Cache
Last reset from power-on
5 Virtual Ethernet interfaces
9 Gigabit Ethernet interfaces
1915K bytes of non-volatile configuration memory.

65536K bytes of Flash internal SIMM (Sector size 512K)

I already check, some conf and process and there seem to have no issue

Any Clue ?

Thanks for your help

Pascal

[phiharri]

Hi Pascal,

The five second CPU utilization after the / indicates the CPU time taken under interrupts, usually software CEF or process switched traffic. That's why you don't see it accounted under any specific process.

There are a number of tools on the 7600 platform which allow you to investigate further what this traffic is and identify why the traffic reaches the switch CPU rather than being handled in hardware.

Take a look at the following CSC document which describes 'debug netdr' https://supportforums.cisco.com/docs/DOC-14086 - following this process when the CPU utilization is increased should give you a head start on isolating the traffic causing the load.

Hope this helps,

/Phil

[Pascal Faucher]

HI Phil,

Thank you very much for this, I will check for this and update my case if everything go well.

Regards,

Pascal

[Pascal Faucher]

Hi, after trying everything on this page.

https://supportforums.cisco.com/docs/DOC-14086

I have 4 - 7604 let call them Net01-02-03-04

Net01 and Net02 work together

Net03-04  work together ( those are ok and have around 2Gb of traffic) normal CPU  (-5%)

--------------------------------------------------

NET02.

sh proc cp | ex 0.00
CPU utilization for five seconds: 30%/26%; one minute: 30%; five minutes: 30%
PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
   5    42748280   2126669      20101  2.31%  0.51%  0.50%   0 Check heaps     
130   124185956 554268138        224  1.67%  1.28%  1.15%   0 IP Input        
202     4238252   2823441       1501  0.15%  0.06%  0.05%   0 HIDDEN VLAN Proc
296     1615916   7281410        221  0.07%  0.03%  0.01%   0 Port manager per

---------------------------------------------------

NEt01

sh proc cp | ex 0.00
CPU utilization for five seconds: 1%/0%; one minute: 2%; five minutes: 2%
PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
  92         776      1055        735  0.31%  0.03%  0.07%   1 Virtual Exec    
130   108507352 661524931        164  0.31%  0.17%  0.15%   0 IP Input

-------------------------------------------------

Net01-02 have a HSRP session

Net01

interface Vlan144
description Public routing
ip address x.x.244.120 255.255.255.128
load-interval 30
standby 244 ip x.x.244.65
standby 244 priority 90
standby 244 preempt
standby 245 ip x.x.244.66
standby 245 priority 120
standby 245 preempt
arp timeout 180

-------------

Net02

interface Vlan144
description Public routing
ip address x.x.244.121  255.255.255.128
load-interval 30
standby 244 ip x.x.244.65
standby 244 priority 120
standby 244 preempt
standby 245 ip x.x.244.66
standby 245 priority 90
standby 245 preempt
arp timeout 180

-------------------------------------------

When I transfert all the traffic on the NEt01 by changing the HSPR priority ( on the net01 I change standby 244 priority 90 by standby 244 priority 150 so he could win the default route) I dont have the CPU problem. So the problem seem to come from traffic overload or something like that, but I could not found where he from.

(Net02)

------- dump of outgoing inband packet -------
interface Vl144, routine draco2_ibc_soutput
dbus info: src_vlan 0x90(144), src_indx 0x380(896), len 0x4A(74)
  bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x0(0)
  00020000 00902800 03800000 4A000000 00000000 00000000 00000000 00000000
mistral hdr: req_token 0x0(0), src_index 0x380(896), rx_offset 0x30(48)
  requeue 0, obl_pkt 0, vlan 0x0(0)
destmac 00.08.A3.7F.D7.80, srcmac 00.13.60.F4.21.00, protocol 0800
layer 3 data: 45000038 7B310000 FF0104BF 459CFFDE 52E7A372 030D8E44
              00000000 45000030 21334000 75067D30 52E7A372 AD00C40A
              062601BD D0CF95FB 00000090 00000008 00007F0A 0807

---------

Interface information:
        Interface IBC0/0(idb 0x5105E658)
        Hardware is Mistral IBC (revision 5)
        5 minute rx rate 73498000 bits/sec, 25000 packets/sec
        5 minute tx rate 73502000 bits/sec, 24986 packets/sec
        311297646 packets input, 114668296595 bytes
        60092 broadcasts received
        311097505 packets output, 114671138442 bytes
        71900 broadcasts sent
        0 Bridge Packet loopback drops
        310055246 Packets CEF Switched, 0 Packets Fast Switched
        0 Packets SLB Switched, 0 Packets CWAN Switched
        Label switched pkts dropped: 0
        Xconnect pkts processed: 0, dropped: 0
        Total paks copied for process level 0
        IBC resets   = 1; last at 05:48:45.168 EST Tue Nov 23 2010

--------------

Net01

------- dump of outgoing inband packet -------
interface Vl144, routine draco2_ibc_soutput
dbus info: src_vlan 0x90(144), src_indx 0x380(896), len 0x42(66)
  bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x0(0)
  06020000 00902800 03800000 42000000 00000000 00000000 00000000 00000000
mistral hdr: req_token 0x0(0), src_index 0x380(896), rx_offset 0x30(48)
  requeue 0, obl_pkt 0, vlan 0x0(0)
destmac 01.00.5E.00.00.02, srcmac 00.08.A3.7F.D7.80, protocol 0800
layer 3 data: 45C00030 00000000 01113872 AC10F478 E0000002 07C107C1
              001C832A 00000803 0A5AF400 63697363 6F000000 AC10F441
              08004BBE 00000000 00000000 00000000 0000

-----------------

Interface information:
        Interface IBC0/0(idb 0x47822D04)
        Hardware is Mistral IBC (revision 5)
        5 minute rx rate 1969000 bits/sec, 483 packets/sec
        5 minute tx rate 1965000 bits/sec, 475 packets/sec
        2365140171 packets input, 1158722691101 bytes
        9439796 broadcasts received
        2339240047 packets output, 1156865111750 bytes
        8145512 broadcasts sent
        0 Bridge Packet loopback drops
        2322662678 Packets CEF Switched, 0 Packets Fast Switched
        0 Packets SLB Switched, 0 Packets CWAN Switched
        Label switched pkts dropped: 0
        Xconnect pkts processed: 0, dropped: 0
        Total paks copied for process level 0
        IBC resets   = 1; last at 13:14:49.751 EDT Thu Jul 2 2009

----------------------

Any idea ?

thanks,

Pascal.

[phiharri]

Hi Pascal,

The two samples you sent before are in outgoing direction so don't tell us much. The idea is to identify if there are some trends in the traffic going to CPU by checking several samples, it's best to just do the capture in rx direction and then apply some filters, try this on switch with high CPU:

debug netdr clear
debug netdr cap rx
(wait a few seconds)
undebug all

sh netdr cap | i int
Is there one interface receiving the majority of traffic? If so - any config/feature on that interface that could require software switching (SLB, PBR, NAT, WCCP, ACL with log etc)?

sh netdr cap | i ttl
Is there an excess of traffic with TTL 1?
Are there some common destination IP/subnets?

sh netdr cap | i dest_indx
How many of the capture lines have 'flood 1' set?
What are the most common dest_indx value?

/Phil

[Pascal Faucher]

Hi Phil,

Thanks for the answer:

there is the result of my test:

sh netdr cap | i int
Is there one interface  receiving the majority of traffic? If so - (: Yes the vl144 receive the majority of the traffic,)

any config/feature on that  interface that could require software switching (SLB, PBR, NAT, WCCP,  ACL with log etc)? ( nothing spécial )

interface Vlan144
description Public-vdl-cax
ip address x.x.244.121 255.255.255.128
load-interval 30
standby 244 ip x.x.244.65
standby 244 priority 120
standby 244 preempt
standby 245 ip x.x.244.66
standby 245 priority 90
standby 245 preempt
arp timeout 180

---------

Vlan144 is up, line protocol is up
  Hardware is EtherSVI, address is 0013.60f4.2100 (bia 0013.60f4.2100)
  Description: Public-vdl-cax
  Internet address is x.x.244.121/25
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 91/255, rxload 36/255
  Encapsulation ARPA, loopback not set
  Keepalive not supported
  ARP type: ARPA, ARP Timeout 00:03:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 3d22h
  Input queue: 2/75/19079/19079 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  30 second input rate 143132000 bits/sec, 44905 packets/sec
  30 second output rate 359022000 bits/sec, 80050 packets/sec
  L2 Switched: ucast: 166337 pkt, 17694266 bytes - mcast: 694677 pkt, 45873303 bytes
  L3 in Switched: ucast: 7909141204 pkt, 3286319992374 bytes - mcast: 0 pkt, 0 bytes mcast
  L3 out Switched: ucast: 15513122309 pkt, 13301093464482 bytes mcast: 0 pkt, 0 bytes
     16743537215 packets input, 6410396741760 bytes, 0 no buffer
     Received 694673 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     33154627851 packets output, 19485616656920 bytes, 0 underruns
     0 output errors, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out

sh netdr cap | i ttl
Is there an excess of traffic  with TTL 1? NO  ( around 1 %)
Are there some common destination IP/subnets? ( no)

sh netdr cap | i dest_indx
How many of the capture  lines have 'flood 1' set? ( no 0%)
What are the most common dest_indx value? ( 0x7F07(32519) )

I will continu to investigate.

Thanks again.

/ Pascal

[phiharri · Accepted solution]

Hi Pascal,

0x7F07 tells us these packets are sent to the CPU to generate an ICMP redirect. Any packet which the forwarding result sends back out the input interface is sent to CPU on this index.

If you check some of the destination IPs with show ip route x.x.x.x I expect you will see Vlan144 as the result (perhaps a default route?). This may indicate some suboptimal routing in your network or just a normal aspect of the design.

You can add to global config mls rate-limit unicast ip icmp redirect 0 prevent these packets being sent to the CPU with no other impact on forwarding.

Hope this helps!

/Phil

[Pascal Faucher]

Hi phil,

This one ( mls rate-limit unicast ip icmp redirect 0) fixe the problem )  thanks you very much for this Phil

Now I will find out if those paquet are  normal.

Thanks again for you help. I owe you one !

Pascal.

[Energy Future Holdings Parent Company]

Hi Phillip,

I am also Observing the Same problem of High CPU on 6500.

Please suggest some Investigations

When i ran Debug as defiend by u . i found two problems

• •-          Many configured ACEs have the “log” command. This means that a copy of the packets will also be sent to the CPU. As you can see below:

ip access-list extended watch110_in

permit ip any host 172.27.192.10 log

permit ip host 172.27.206.2 any log

permit ip host 172.27.206.10 any log

permit ip host 172.27.206.126 any log

• •-          In the netdr capture that I performed, I could see packets with a source IP of any of the above was also sent to the CPU.

2/ Redirects

• •-          There are a number of packets which also have the destination index of 0x7F07

Please suggest

[phiharri]

Greetings,

I'm glad the above debug was helpful in identifying two sources of traffic that are causing increased CPU utilization on your 6500.

1) I would suggest removing the 'log' statements from your ACLs. What is the purpose of logging ACL hits in your configuration? There is a feature called Optimized ACL Logging (OAL) which can perform a similar function to ACL logging with minimal impact to the CPU, but there are several restrictions around that feature. I believe you would be better off looking into Netflow if the log statements are being used to monitor large volumes of data-plane traffic.

2) As discussed above, destination index 0x7F07 indicates that the packets are redirected to the CPU to generate ICMP redirects. This happens when the packet is sent back out on the same Layer 3 interface it arrived on. You can configure 'mls rate-limit unicast ip icmp redirect 0' to prevent redirecting frames to the CPU to generate ICMP redirects, but as this usually indicates suboptimal routing I suggest looking closely at which next-hop device this traffic is coming from and what the routing table shows for that destination address on both devices.

Hope this helps,

/Phil

[Energy Future Holdings Parent Company]

Hi Philip,

Thanks for the reply.....I think ICMP Redirect works when the 2 Ip's source and next hop are in same subnet.

I don’t see the packets entering the leaving the same interface that can cause ICMP redirect.

I am typing some output below , Please suggest in this case .

------- dump of incoming inband packet -------

interface Vl305, routine mistral_process_rx_packet_inlin, timestamp 21:26:11.827

dbus info: src_vlan 0x131(305), src_indx 0xB43(2883), len 0x59E(1438)

  bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x7F07(32519)

  88820400 01310000 0B430005 9E080000 00060565 0E000008 00000000 7F073EE7

mistral hdr: req_token 0x0(0), src_index 0xB43(2883), rx_offset 0x76(118)

  requeue 0, obl_pkt 0, vlan 0x131(305)

destmac 00.26.CA.1F.8B.80, srcmac 00.17.A4.77.50.14, protocol 0800

protocol ip: version 0x04, hlen 0x05, tos 0x00, totlen 1420, identifier 9426

  df 1, mf 0, fo 0, ttl 128, src 172.27.206.30, dst 172.27.216.114

    tcp src 54669, dst 8400, seq 2679885686, ack 930915520, win 64860 off 5 checksum 0x9ECC ack

------- dump of outgoing inband packet -------

interface Vl305, routine draco2_fastsend, timestamp 21:26:11.827

dbus info: src_vlan 0x131(305), src_indx 0xB43(2883), len 0x59E(1438)

  bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x7F07(32519)

  00020000 01312800 0B430005 9E080000 00060565 00000000 00000000 7F073EE7

mistral hdr: req_token 0x0(0), src_index 0xB43(2883), rx_offset 0x76(118)

  requeue 0, obl_pkt 0, vlan 0x131(305)

destmac 70.CA.9B.F0.53.1D, srcmac 00.26.CA.1F.8B.80, protocol 0800

protocol ip: version 0x04, hlen 0x05, tos 0x00, totlen 1420, identifier 9426

  df 1, mf 0, fo 0, ttl 127, src 172.27.206.30, dst 172.27.216.114

    tcp src 54669, dst 8400, seq 2679885686, ack 930915520, win 64860 off 5 checksum 0x9ECC ack

0x7F07 tells us these packets are sent to the CPU to generate an ICMP redirect. Any packet which the forwarding result sends back out the input interface is sent to CPU on this index.

WACDVSS01A#sh ip route 172.27.216.114                            <<< Destination Address>>>

Routing entry for 172.27.216.0/22

  Known via "static", distance 1, metric 0

  Redistributing via eigrp 1965

  Advertised by eigrp 1965 metric 100000 1 255 1 1500 route-map SERVER_FARM

                bgp 64911

  Routing Descriptor Blocks:

  * 172.27.200.10

WACDVSS01A#sh ip arp | in 172.27.206.30                                                     <<< Source Address >>>

Internet  172.27.206.30           4   0017.a477.5016  ARPA   Vlan305

WACDVSS01A#sh ip arp | in 172.27.200.10                                                

Internet  172.27.200.10           0   70ca.9bf0.531c  ARPA   Vlan110

WACDVSS01A#sh run inter

WACDVSS01A#sh run interface vl

WACDVSS01A#sh run interface vlan 110

Building configuration...

Current configuration : 116 bytes

!

interface Vlan110

description " ASA UPLINK VLAN "

ip address 172.27.200.9 255.255.255.248

ip flow ingress                                                           <<< Should be no ip-redirect>>>

end

[phiharri]

Hi,

Does 'sh ip ro 172.27.200.10' show this as a connected destination? Are there any features on the input interface Vlan305 which could cause forwarding other than via the standard routing table? (PBR, NAT, WCCP etc..), can you share config on that interface?

Kind Regards,

/Phil

[bhupen541]

Hi Phil/all,

i am seeing high CPU on 6509 frequently for a very short period of time.

i captured packet by EM script. I see dest Mac-01.00.5E.00.00.02 for many communication..

But i dont see that Mac address any where in my network. Can you /anyone please help me here ?

------- dump of incoming inband packet -------
interface Vl2145, routine draco2_process_rx_packet_inline
dbus info: src_vlan 0x861(2145), src_indx 0x340(832), len 0x42(66)
bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x4861(18529)
EE820400 08610400 03400000 42080000 00110550 000004A0 00000008 4861BA03
mistral hdr: req_token 0x0(0), src_index 0x340(832), rx_offset 0x76(118)
requeue 0, obl_pkt 0, vlan 0x861(2145)
destmac 01.00.5E.00.00.02, srcmac 00.00.0C.07.AC.00, protocol 0800
protocol ip: version 0x04, hlen 0x05, tos 0xC0, totlen 48, identifier 0
df 0, mf 0, fo 0, ttl 1, src 172.21.91.130, dst 224.0.0.2
udp src 1985, dst 1985 len 28 checksum 0xA0C9

------- dump of incoming inband packet -------
interface Vl2384, routine draco2_process_rx_packet_inline
dbus info: src_vlan 0x950(2384), src_indx 0x340(832), len 0x42(66)
bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x4950(18768)
E6820400 09500400 03400100 42080000 00110520 00000450 00000008 4950F056
mistral hdr: req_token 0x0(0), src_index 0x340(832), rx_offset 0x76(118)
requeue 0, obl_pkt 0, vlan 0x950(2384)
destmac 01.00.5E.00.00.02, srcmac 00.00.0C.07.AC.00, protocol 0800
protocol ip: version 0x04, hlen 0x05, tos 0xC0, totlen 48, identifier 0
df 0, mf 0, fo 0, ttl 1, src 172.21.71.130, dst 224.0.0.2
udp src 1985, dst 1985 len 28 checksum 0xC8C9

------- dump of incoming inband packet -------
interface Vl2404, routine draco2_process_rx_packet_inline
dbus info: src_vlan 0x964(2404), src_indx 0x340(832), len 0x42(66)
bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x4964(18788)
6E820400 09640400 03400100 42080000 00110540 00000450 00000008 49649A56
mistral hdr: req_token 0x0(0), src_index 0x340(832), rx_offset 0x76(118)
requeue 0, obl_pkt 0, vlan 0x964(2404)
destmac 01.00.5E.00.00.02, srcmac 00.00.0C.07.AC.00, protocol 0800
protocol ip: version 0x04, hlen 0x05, tos 0xC0, totlen 48, identifier 0
df 0, mf 0, fo 0, ttl 1, src 172.21.76.2, dst 224.0.0.2
udp src 1985, dst 1985 len 28 checksum 0xBFC9

------- dump of incoming inband packet -------
interface Vl2364, routine draco2_process_rx_packet_inline
dbus info: src_vlan 0x93C(2364), src_indx 0x340(832), len 0x42(66)
bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x493C(18748)
3E820400 093C0400 03400000 42080000 00110570 000004A0 00000008 493C7A47
mistral hdr: req_token 0x0(0), src_index 0x340(832), rx_offset 0x76(118)
requeue 0, obl_pkt 0, vlan 0x93C(2364)
destmac 01.00.5E.00.00.02, srcmac 00.00.0C.07.AC.00, protocol 0800
protocol ip: version 0x04, hlen 0x05, tos 0xC0, totlen 48, identifier 0
df 0, mf 0, fo 0, ttl 1, src 172.21.67.2, dst 224.0.0.2
udp src 1985, dst 1985 len 28 checksum 0xD1C9

------- dump of incoming inband packet -------
interface Po4, routine draco2_process_rx_packet_inline
dbus info: src_vlan 0x3F5(1013), src_indx 0x343(835), len 0x40(64)
bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x380(896)
F0020400 03F50000 03430000 40000000 00060428 08000400 00000000 0380C77C
mistral hdr: req_token 0x0(0), src_index 0x343(835), rx_offset 0x76(118)
requeue 0, obl_pkt 0, vlan 0x3F5(1013)
destmac 00.1D.46.D6.E4.00, srcmac 00.1D.46.76.F4.00, protocol 0800
protocol ip: version 0x04, hlen 0x05, tos 0x40, totlen 40, identifier 13297
df 1, mf 0, fo 0, ttl 54, src 7.6.255.25, dst 10.253.8.12
tcp src 47197, dst 22, seq 1923130346, ack 1319359757, win 65535 off 5 checksum 0xA3FC ack

------- dump of incoming inband packet -------
interface Vl2464, routine draco2_process_rx_packet_inline
dbus info: src_vlan 0x9A0(2464), src_indx 0x340(832), len 0x42(66)
bpdu 0, index_dir 0, flood 1, dont_lrn 0, dest_indx 0x49A0(18848)
16820400 09A00400 03400100 42080000 00110520 00000450 00000008 49A0A25D
mistral hdr: req_token 0x0(0), src_index 0x340(832), rx_offset 0x76(118)
requeue 0, obl_pkt 0, vlan 0x9A0(2464)
destmac 01.00.5E.00.00.02, srcmac 00.00.0C.07.AC.00, protocol 0800
protocol ip: version 0x04, hlen 0x05, tos 0xC0, totlen 48, identifier 0
df 0, mf 0, fo 0, ttl 1, src 172.21.116.2, dst 224.0.0.2
udp src 1985, dst 1985 len 28 checksum 0x6FC9