7606 Router cpu utilization is hitting 99%!

[lonamatiyahoocom]

7606-IGW-1#more bootdisk:crashinfo_20100824-055939-UTC

Writing crashinfo to bootdisk:crashinfo_20100824-055939-UTCYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x9469914, alignment 8

Pool: Processor  Free: 4921508  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP Input", ipl= 0, pid= 202

-Traceback= 8553BAC 85542B0 9430154 9439D64 9469918 942F8D4 946B154 942F720 B065610 B065798 8B91528 8B91BC4 8B92338 8B94CFC 8B8F8A8 8B73080

004120: *Aug 23 13:38:22.446 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004121: *Aug 23 13:38:32.450 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004122: *Aug 23 13:38:42.466 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004123: *Aug 23 13:38:44.426 UTC: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x9469914, alignment 8

Pool: Processor  Free: 4683836  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 9430154 9439D64 9469918 942F7E8 942F530 A7D5BE4 A7CB838

004124: *Aug 23 13:38:52.470 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004125: *Aug 23 13:39:02.482 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004126: *Aug 23 13:39:12.490 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004127: *Aug 23 13:39:14.426 UTC: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x9469914, alignment 8

Pool: Processor  Free: 5064576  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP Input", ipl= 0, pid= 202

-Traceback= 8553BAC 85542B0 9430154 9439D64 9469918 942F8D4 946B154 942F720 B0655AC B065798 8B91528 8B91BC4 8B92338 8B94CFC 8B8F8A8 8B73080

004128: *Aug 23 13:39:22.506 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004129: *Aug 23 13:39:32.506 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004130: *Aug 23 13:39:42.506 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004131: *Aug 23 13:39:44.426 UTC: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x9469914, alignment 8

Pool: Processor  Free: 5399612  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP Input", ipl= 0, pid= 202

-Traceback= 8553BAC 85542B0 9430154 9439D64 9469918 942F8D4 946B154 942F720 B0655AC B065798 8B91528 8B91BC4 8B92338 8B94CFC 8B8F8A8 8B73080

004132: *Aug 23 13:39:52.506 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004133: *Aug 23 13:40:02.522 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004134: *Aug 23 13:40:12.522 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004135: *Aug 23 13:40:14.426 UTC: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x9469914, alignment 8

Pool: Processor  Free: 5001240  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP Input", ipl= 0, pid= 202

-Traceback= 8553BAC 85542B0 9430154 9439D64 9469918 942F8D4 946B154 942F720 B0655AC B065798 8B91528 8B91BC4 8B92338 8B94040 8B8F7B4 8B73080

004136: *Aug 23 13:40:22.530 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004137: *Aug 23 13:40:32.554 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004138: *Aug 23 13:40:42.578 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004139: *Aug 23 13:40:44.430 UTC: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x9469914, alignment 8

Pool: Processor  Free: 4847268  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 9430154 9439D64 9469918 942F7E8 942F530 A7D5BE4 A7CB838

004140: *Aug 23 13:40:52.606 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004141: *Aug 23 13:41:02.614 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004142: *Aug 23 13:41:12.626 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004143: *Aug 23 13:41:14.446 UTC: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x9469914, alignment 8

Pool: Processor  Free: 4596772  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP Input", ipl= 0, pid= 202

-Traceback= 8553BAC 85542B0 9430154 9439D64 9469918 942F8D4 946B154 942F720 B0655AC B065798 8B91528 8B91BC4 8B92338 8B94040 8B8F7B4 8B73080

004144: *Aug 23 13:41:22.630 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004145: *Aug 23 13:41:32.638 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004146: *Aug 23 13:41:42.638 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004147: *Aug 23 13:41:44.462 UTC: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x9469914, alignment 8

Pool: Processor  Free: 5020716  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 9430154 9439D64 9469918 942F7E8 942F530 A7D5BE4 A7CB838

004148: *Aug 23 13:41:52.646 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004149: *Aug 23 13:42:02.658 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004150: *Aug 23 13:42:12.674 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004151: *Aug 23 13:42:14.478 UTC: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x9469914, alignment 8

Pool: Processor  Free: 5479568  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 9430154 9439D64 9469918 942F7E8 942F530 A7D5BE4 A7CB838

004152: *Aug 23 13:42:22.690 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004153: *Aug 23 13:42:32.694 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004154: *Aug 23 13:42:42.706 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004155: *Aug 23 13:42:44.478 UTC: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x9469914, alignment 8

Pool: Processor  Free: 5044544  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 9430154 9439D64 9469918 942F7E8 942F530 A7D5BE4 A7CB838

004156: *Aug 23 13:42:52.710 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004157: *Aug 23 13:43:02.730 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004158: *Aug 23 13:43:12.730 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004159: *Aug 23 13:43:14.502 UTC: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x9469914, alignment 8

Pool: Processor  Free: 4698184  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 9430154 9439D64 9469918 942F7E8 942F530 A7D5BE4 A7CB838

004160: *Aug 23 13:43:22.746 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004161: *Aug 23 13:43:32.754 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

004162: *Aug 23 13:43:42.778 UTC: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat localtre. No memory available -Process= "Chunk Manager", ipl= 2, pid= 1

-Traceback= 8553BAC 85542B0 942F578 A7D5BE4 A7CB838

[rsimoni]

so the reason for such high CPU utilization is that some traffic is sent to the CPU to be forwarded instead of remaining on the hw plane and be handled there (by the PFC).

From the logs it seems that all the traffic 'punted' to the CPU comes from 3 interfaces:

Gi1/3.42

Gi5/1

Gi5/2.11

When you connect that iBGP session the new BGP traffic arrives to one of those interfaces?

let's also do the following checks:

- sh tcam int g1/3.42 acl in ip

- sh tcam int g5/2.11 acl in ip

- sh tcam int g5/1 acl in ip

- sh mls cef exception status  (when CPU is high)

let's focus on the first interface and check some random flows:

but first i need to see the int config

show run int gi1/3.42 (remove IP address if you wish).

then let's examine a couple of flows

show ip cef 185.14.248.65

show mls cef lookup 185.14.248.65

show ip cef 159.255.161.70

show mls cef lookup 159.255.161.70

NOTE: if Gi1/3.42 is under some vrf you need to do the checks above for that vrf.

i.e: show ip cef vrf 185.14.248.65

show mls cef vrf 185.14.248.65

Message was edited by: Riccardo Simoni

[dlerkamal]

Dear riccardo ,

i send you the output to your account.

thanks

[andy akpa]

hello Lorna i need your support , i' m having the same issue with my cisco 7606 Router  cpu utilization is hitting 70% , can you please help me ?