Buy or Renew
Buy or Renew
Cisco + Splunk: It’s a new day for your data. Learn more.
 
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
6683
Views
20
Helpful
32
Replies

7606 Router cpu utilization is hitting 99%!

Go to solution
lonamatiyahoocom
Level 1
Level 1

‎01-17-2012 12:45 AM - edited ‎03-04-2019 02:55 PM

7606-IGW-1#more bootdisk:crashinfo_20100824-055939-UTC

Writing crashinfo to bootdisk:crashinfo_20100824-055939-UTCYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x9469914, alignment 8

Pool: Processor  Free: 4921508  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP Input", ipl= 0, pid= 202

-Traceback= 8553BAC 85542B0 9430154 9439D64 9469918 942F8D4 946B154 942F720 B065610 B065798 8B91528 8B91BC4 8B92338 8B94CFC 8B8F8A8 8B73080

004120: *Aug 23 13:38:22.446 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004121: *Aug 23 13:38:32.450 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004122: *Aug 23 13:38:42.466 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004123: *Aug 23 13:38:44.426 UTC: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x9469914, alignment 8

Pool: Processor  Free: 4683836  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 9430154 9439D64 9469918 942F7E8 942F530 A7D5BE4 A7CB838

004124: *Aug 23 13:38:52.470 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004125: *Aug 23 13:39:02.482 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004126: *Aug 23 13:39:12.490 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004127: *Aug 23 13:39:14.426 UTC: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x9469914, alignment 8

Pool: Processor  Free: 5064576  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP Input", ipl= 0, pid= 202

-Traceback= 8553BAC 85542B0 9430154 9439D64 9469918 942F8D4 946B154 942F720 B0655AC B065798 8B91528 8B91BC4 8B92338 8B94CFC 8B8F8A8 8B73080

004128: *Aug 23 13:39:22.506 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004129: *Aug 23 13:39:32.506 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004130: *Aug 23 13:39:42.506 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004131: *Aug 23 13:39:44.426 UTC: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x9469914, alignment 8

Pool: Processor  Free: 5399612  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP Input", ipl= 0, pid= 202

-Traceback= 8553BAC 85542B0 9430154 9439D64 9469918 942F8D4 946B154 942F720 B0655AC B065798 8B91528 8B91BC4 8B92338 8B94CFC 8B8F8A8 8B73080

004132: *Aug 23 13:39:52.506 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004133: *Aug 23 13:40:02.522 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004134: *Aug 23 13:40:12.522 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004135: *Aug 23 13:40:14.426 UTC: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x9469914, alignment 8

Pool: Processor  Free: 5001240  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP Input", ipl= 0, pid= 202

-Traceback= 8553BAC 85542B0 9430154 9439D64 9469918 942F8D4 946B154 942F720 B0655AC B065798 8B91528 8B91BC4 8B92338 8B94040 8B8F7B4 8B73080

004136: *Aug 23 13:40:22.530 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004137: *Aug 23 13:40:32.554 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004138: *Aug 23 13:40:42.578 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004139: *Aug 23 13:40:44.430 UTC: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x9469914, alignment 8

Pool: Processor  Free: 4847268  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 9430154 9439D64 9469918 942F7E8 942F530 A7D5BE4 A7CB838

004140: *Aug 23 13:40:52.606 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004141: *Aug 23 13:41:02.614 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004142: *Aug 23 13:41:12.626 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004143: *Aug 23 13:41:14.446 UTC: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x9469914, alignment 8

Pool: Processor  Free: 4596772  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP Input", ipl= 0, pid= 202

-Traceback= 8553BAC 85542B0 9430154 9439D64 9469918 942F8D4 946B154 942F720 B0655AC B065798 8B91528 8B91BC4 8B92338 8B94040 8B8F7B4 8B73080

004144: *Aug 23 13:41:22.630 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004145: *Aug 23 13:41:32.638 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004146: *Aug 23 13:41:42.638 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004147: *Aug 23 13:41:44.462 UTC: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x9469914, alignment 8

Pool: Processor  Free: 5020716  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 9430154 9439D64 9469918 942F7E8 942F530 A7D5BE4 A7CB838

004148: *Aug 23 13:41:52.646 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004149: *Aug 23 13:42:02.658 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004150: *Aug 23 13:42:12.674 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004151: *Aug 23 13:42:14.478 UTC: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x9469914, alignment 8

Pool: Processor  Free: 5479568  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 9430154 9439D64 9469918 942F7E8 942F530 A7D5BE4 A7CB838

004152: *Aug 23 13:42:22.690 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004153: *Aug 23 13:42:32.694 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004154: *Aug 23 13:42:42.706 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004155: *Aug 23 13:42:44.478 UTC: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x9469914, alignment 8

Pool: Processor  Free: 5044544  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 9430154 9439D64 9469918 942F7E8 942F530 A7D5BE4 A7CB838

004156: *Aug 23 13:42:52.710 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004157: *Aug 23 13:43:02.730 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004158: *Aug 23 13:43:12.730 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004159: *Aug 23 13:43:14.502 UTC: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x9469914, alignment 8

Pool: Processor  Free: 4698184  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 9430154 9439D64 9469918 942F7E8 942F530 A7D5BE4 A7CB838

004160: *Aug 23 13:43:22.746 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004161: *Aug 23 13:43:32.754 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004162: *Aug 23 13:43:42.778 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

Labels:
0 Helpful
1 Accepted Solution

Accepted Solutions

Go to solution
Peter Paluch
Cisco Employee
Cisco Employee

‎01-17-2012 12:58 AM

Lorna,

According to these messages, your router was not able to allocate more memory for internal IOS processes including NAT. There are various possible reasons for that:

  1. Your router is low on memory with respect to the amount of services and clients it is serving
  2. Your router has been running for an extended period of time without restart and the memory allocation became so fragmented that the IOS is unable to find an unfragmented section of free RAM for the requesting process
  3. Your NAT configuration results in too many translation entries being created and maintained
  4. You have experienced an IOS bug

Does this problem steadily reappear after restarting the device? If yes, I suggest considering upgrading the IOS and verifying the configuration for obvious memory-hungry processes that could be optimized. Extending the RAM may also be recommendable.

Best regards,

Peter

View solution in original post

0 Helpful
32 Replies 32

Go to solution
Peter Paluch
Cisco Employee
Cisco Employee

‎01-17-2012 12:58 AM

Lorna,

According to these messages, your router was not able to allocate more memory for internal IOS processes including NAT. There are various possible reasons for that:

  1. Your router is low on memory with respect to the amount of services and clients it is serving
  2. Your router has been running for an extended period of time without restart and the memory allocation became so fragmented that the IOS is unable to find an unfragmented section of free RAM for the requesting process
  3. Your NAT configuration results in too many translation entries being created and maintained
  4. You have experienced an IOS bug

Does this problem steadily reappear after restarting the device? If yes, I suggest considering upgrading the IOS and verifying the configuration for obvious memory-hungry processes that could be optimized. Extending the RAM may also be recommendable.

Best regards,

Peter

0 Helpful

Go to solution
lonamatiyahoocom
Level 1
Level 1
In response to Peter Paluch

‎01-17-2012 04:16 AM

Thank you Peter.

Most of the cpu process is taken by IP NAT

i guess that is draining the memory.

The problem recurs after restarting.

sh processes cpu | exclude 0.00%

CPU utilization for five seconds: 99%/72%; one minute: 73%; five minutes: 27%

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

   7        1120        71      15774  2.14%  0.30%  0.12%   0 Check heaps

  66         464       317       1463  0.23%  0.13%  0.05%   1 Virtual Exec

206       26760     84203        317  6.36%  6.76%  2.95%   0 IP Input

329        8292      9048        916  2.30%  1.90%  0.82%   0 FM core

340         384       145       2648  0.15%  0.09%  0.05%   0 HIDDEN VLAN Proc

490       28740      1811      15869 14.96%  8.76%  2.68%   0 IP NAT Ager

i have 2 rsp720 cards...

0 Helpful

Go to solution
Peter Paluch
Cisco Employee
Cisco Employee
In response to lonamatiyahoocom

‎01-17-2012 05:08 AM

Hi Lorna,

A couple of questions (actually, lots of )

Your CPU load is surprisingly high even in interrupt context (99%/72%). Please try to check the show cef not-cef-switched and show ip cef switching-statistics commands to see if there is any traffic being punted to CPU inappropriately. Also, are you performing any multicast routing or NetFlow version 7 data collection on this device? Are there any tunnels terminated on this 7600?

Can you also post the output of the show ip nat statistics and the first lines of the show memory free commands?

What is the IOS version you are running?

Best regards,

Peter

0 Helpful

Go to solution
lonamatiyahoocom
Level 1
Level 1
In response to Peter Paluch

‎01-17-2012 11:26 PM

Hi Peter,

Below is the output.

There are no tunnels terminated.

There is no multicast routing or NetFlow version 7 data collection performed.

ios version is:

(c7600rsp72043_rp-ADVIPSERVICESK9-M), Version 12.2(33)SRD2a, RELEASE SOFTWARE (fc2)

#####show ip cef switching statistics

       Reason                          Drop       Punt  Punt2Host

RP LES No route                  17490          0        695

RP LES Packet destined for us   0      21462          0

RP LES TTL expired                  0          0       6923

RP LES Bad IP packet length   19          0          0

RP LES Routed to Null0           4405        0          6

RP LES Features                6088870       0   61618491

RP LES Total                    6110784      21462   61626115

All    Total                        6110784      21462   61626115

########show cef not-cef-switched

% Command accepted but obsolete, see 'show (ip|ipv6) cef switching statistics [feature]'

IPv4 CEF Packets passed on to next switching layer

Slot  No_adj No_encap Unsupp'ted Redirect  Receive  Options   Access     Frag

RP         0       0    61580822        0    21356        0   240042        0

5/0        0       0           0        0        0        0        0        0

#######show ip nat statistics

Total active translations: 30496 (56 static, 30440 dynamic; 30436 extended)

Outside interfaces:

  GigabitEthernet1/1, GigabitEthernet1/2, GigabitEthernet5/1

Inside interfaces:

  GigabitEthernet5/2, GigabitEthernet6/2

Hits: 113323141  Misses: 0

CEF Translated packets: 61102193, CEF Punted packets: 61575188

Expired translations: 9618313

Dynamic mappings:

-- Inside Source

[Id: 1] access-list NAT-ACL pool INTERNET_POOL refcount 29322

pool INTERNET_POOL: netmask 255.255.255.0

#######

#sh memory free

                        Head    Total(b)          Used(b)      Free(b)           Lowest(b)  Largest(b)

Processor   12DCA8F0   690181904   195160932   495020972   471076240   461029160

      I/O         3C000000    67108864    39674068    27434796    27434796    27404988

           Processor memory

0 Helpful

Go to solution
Konstantin Dunaev
Level 4
Level 4
In response to lonamatiyahoocom

‎01-18-2012 03:44 AM

Hello,

if you router has turned  into "processed-switched" mode you can see it if you try those commands:


sh ibc

shows the statistics of CPU processed packets 

sh mls cef exception status

  shows the current "exception" status of CEF process 

sh mls cef 0.0.0.0  (or other prefix)

  shows the "exception" status of CEF and you should see the MAC address and the interface in  adjacency field in normal status and "receive" in "exception" status

Status "exception" for CEF table means that the forwarding is processed by CPU. Only system reboot can recover the system.

0 Helpful

Go to solution
lonamatiyahoocom
Level 1
Level 1
In response to Konstantin Dunaev

‎01-18-2012 04:00 AM

Thanks guys, the issue is sorted.

i had to two

Rsp720-3C-GE.

but the IOS were different. hence they were not load balancing.

i have corrected and the cpu process is at 7%.

best regards,

Lorna

0 Helpful

Go to solution
Konstantin Dunaev
Level 4
Level 4
In response to lonamatiyahoocom

‎01-18-2012 04:21 AM

but if I correctly undestand, the traffic is not processed by CPU on 720's, CPU processes only a few types of packets and doesn't really depend on amount of whole traffic, which is forwarded by CEF in hardware.

Do you mean that because of IOS difference they have build some kind of loop?

Have you rebooted the router with 100% CPU load? Did you have a chance to check a CEF status?

0 Helpful

Go to solution
lonamatiyahoocom
Level 1
Level 1
In response to Konstantin Dunaev

‎01-23-2012 02:22 AM

hi Konstantin,


Unfortunately my problem is not solved:(

the cpu process is still at 99%.

i realized it's related to natting.

71758: *Jan 23 07:22:42.355 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for NAT Port Range. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

but there is enough memory.

#sh processes cpu sOrted | EXCLude 0.00%

CPU utilization for five seconds: 99%/78%; one minute: 99%; five minutes: 99%

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

473     2898728   1197976       2419 11.49% 13.17% 16.45%   0 IP NAT Ager

202     1224836   1092060       1121  7.90%  7.87%  6.93%   0 IP Input

322      416936   1011441        412  0.71%  0.81%  0.91%   0 FM core

   7      567812     60170       9436  0.63%  0.45%  0.36%   0 Check heaps

   1        9820     61826        158  0.07%  0.12%  0.07%   0 Chunk Manager

193        5864  10668454          0  0.07%  0.04%  0.01%   0 ACCT Periodic Pr

-7606-IGW-1#sh memory

                Head    Total(b)     Used(b)     Free(b)   Lowest(b)  Largest(b)

Processor 12DCA8F0   690181904   499364108   190817796   190817796   190299264

      I/O      3C000000    67108864    39675576    27433288    27433288    27267644

any idea how i can sort this?



121105-debug netdr capture rx.txt.zip
0 Helpful

Go to solution
nkarpysh
Cisco Employee
Cisco Employee
In response to lonamatiyahoocom

‎01-23-2012 02:26 AM

Lorna,

Your CPU is primarily driven by interrupts (traffic ) see the second value after /:

99%/78%

So some traffic being processed switched causing you the problems. You need to identify that. There is non-impacting method to sniff CPU:

debug netdr capture rx

term len 0

show netdr capture

u all

This will show you what are the packets sent to CPU instead of being HW switched. You can examine those or attached here for anyone to advise how to mitigate those.

Nik

HTH,
Niko

Go to solution
lonamatiyahoocom
Level 1
Level 1
In response to nkarpysh

‎01-23-2012 03:59 AM

i have attached the output

0 Helpful

Go to solution
Konstantin Dunaev
Level 4
Level 4
In response to lonamatiyahoocom

‎01-23-2012 04:47 AM

could you also  take the ouput from: 

sh ibc

shows the statistics of CPU processed packets 

sh mls cef exception status

  shows the current "exception" status of CEF process 

sh mls cef 0.0.0.0  (or other prefix)

   shows the "exception" status of CEF and you should see the MAC address  and the interface in  adjacency field in normal status and "receive" in "exception" status

it'd show the status of you CEF table

0 Helpful

Go to solution
lonamatiyahoocom
Level 1
Level 1
In response to Konstantin Dunaev

‎01-23-2012 06:38 AM

TREASU-7606-IGW-1#sh ibc

Interface information:

        Interface IBC0/0(idb 0x18AA0F70)

        5 minute rx rate 56486000 bits/sec, 11606 packets/sec

        5 minute tx rate 249060000 bits/sec, 144584 packets/sec

        123081359 packets input, 86825263858 bytes

        123069607 broadcasts received

        359510131 packets output, 115599080308 bytes

        102533 broadcasts sent

        0 Bridge Packet loopback drops

        109480858 Packets CEF Switched, 7634939 Packets Fast Switched

        0 Packets SLB Switched, 0 Packets CWAN Switched

        Label switched pkts dropped: 0    Pkts dropped during dma: 754207

        Invalid pkts dropped: 17641    Pkts dropped(not cwan consumed): 2727

        Xconnect pkts processed: 0, dropped: 0

        Xconnect pkt reflection drops: 0

        Total paks copied for process level 0

        Total short paks sent in route cache 28838452

        Total throttle drops 46125    Input queue drops 2031

        total spd packets classified (5781168 low, 13018 medium, 4 high)

        total spd packets dropped (753377 low, 830 medium, 0 high)

        spd prio pkts allowed in due to selective throttling (0 med, 0 high)

        IBC resets   = 1; last at 11:17:19.374 UTC Mon Jan 23 2012

Driver Level Counters: (Cumulative, Zeroed only at Reset)

          Frames          Bytes

  Rx(0)   164754273       3854708031

  Rx(1)   114163593       1178997874

  Tx(0)   513776342       1547263353

Input Drop Frame Count

     Rx0 = 279787           Rx1 = 351777

Per Queue Receive Errors:

     FRME   OFLW   BUFE   NOENP  DISCRD DISABLE

Rx0 0      0      0      14133  0        0

Rx1 0      0      0      10491  0        0

  Tx Errors/State:

   One Collision Error   = 0            More Collisions       = 0

   No Encap Error        = 0            Deferred Error        = 0

   Loss Carrier Error    = 0            Late Collision Error  = 0

   Excessive Collisions  = 0            Buffer Error          = 0

   Tx Freeze Count       = 0            Tx Intrpt Serv timeout= 1

  Counters collected at Idb:

   Is input throttled    = 0            Throttle Count        = 0

   Rx Resource Errors    = 0            Input Drops           = 654695

   Input Errors           = 80398

   Output Drops          = 0            Giants/Runts          = 0/0

   Dma Mem Error         = 0            Input Overrun         = 0

TREASU-7606-IGW-1#sh mls cef exception status

Current IPv4 FIB exception state = FALSE

Current IPv6 FIB exception state = FALSE

Current MPLS FIB exception state = FALSE

TREASU-7606-IGW-1#sh mls cef 0.0.0.0

Codes: decap - Decapsulation, + - Push Label

Index  Prefix              Adjacency

64     0.0.0.0/32          receive

108768 0.0.0.0/8           drop

134368 0.0.0.0/0           Gi1/2, c471.fe03.6480

134400 0.0.0.0/0           drop

0 Helpful

Go to solution
Konstantin Dunaev
Level 4
Level 4
In response to lonamatiyahoocom

‎01-23-2012 07:10 AM

"sh ibc" shows that you you have ~250Mbit egress and ~50 Mbit ingress traffic processed by CPU.

Now is the question - why.

CEF looks normall.

Have you checked  the featured set, which is supported by your HW? may be you some strange policy-map or QoS statements which are not supported by HW and should sent to the CPU.

Just have a look carefully ally your active "funny features"

0 Helpful

Go to solution
lonamatiyahoocom
Level 1
Level 1
In response to Konstantin Dunaev

‎01-24-2012 12:14 AM

we do not have  any strange qos commands.

just

mls qos trust dscp

and an access-list to filter traffic

0 Helpful
Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Customers Also Viewed These Support Documents
Review Cisco Networking products for a $25 gift card