CISCO 6509 high CPU

[andresdavid]

Hello everybody, I have this problem for 2 days and I don't know what to do, 

bb01.network.ro>sh processes cpu | exclude 0.00%__0.00%__0.00%

CPU utilization for five seconds: 76%/67%; one minute: 84%; five minutes: 91%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
2 56640 661700 85 0.00% 0.01% 0.00% 0 Load Meter
8 34163576 1740542 19628 0.00% 0.88% 0.67% 0 Check heaps
11 4609760 7352294 626 0.15% 0.12% 0.13% 0 ARP Input
20 469000 3278408 143 0.00% 0.01% 0.00% 0 IPC Periodic Tim
23 26914948 162975847 165 0.00% 0.45% 0.51% 0 IPC Seat Manager
50 1405336 3451726 407 0.07% 0.07% 0.07% 0 Per-Second Jobs
51 2748004 116538 23580 0.87% 0.10% 0.06% 0 Per-minute Jobs
63 3541328 146553080 24 0.07% 0.07% 0.07% 0 Net Input
65 4048 463 8742 0.31% 0.19% 0.26% 2 SSH Process
84 2107256 5299555 397 0.07% 0.05% 0.05% 0 DHCP Snooping
218 659728 743227 887 0.00% 0.03% 0.02% 0 Compute load avg
241 526228 93192878 5 0.00% 0.03% 0.02% 0 ACE Tunnel Task
242 297764 6528667 45 0.07% 0.01% 0.00% 0 ACE Config Prop
253 1524904 3120700 488 0.00% 0.02% 0.01% 0 esw_vlan_stat_pr
266 1424060 2545041 559 0.07% 0.08% 0.08% 0 CDP Protocol
272 425015252 275710734 1541 2.63% 5.69% 8.45% 0 IP Input
298 98168 20468998 4 0.07% 0.00% 0.00% 0 Ethernet Timer C
299 1543800 374615844 4 0.23% 0.17% 0.16% 0 Ethernet Msec Ti
315 3613924 2274463 1588 0.15% 0.10% 0.11% 0 QOS Stats Gather
322 690244 120644 5721 0.00% 0.01% 0.00% 0 CEF background p
327 443864 77336 5739 0.07% 0.00% 0.00% 0 IP Background
333 526848 11680137 45 0.00% 0.03% 0.04% 0 TCP Timer
363 2090900 4989795 419 0.00% 0.06% 0.07% 0 CEF: IPv4 proces
376 95343748 35597873 2678 0.71% 0.78% 1.05% 0 FM core
384 1511928 1655151 913 0.00% 0.04% 0.05% 0 HIDDEN VLAN Proc
403 517800 93032054 5 0.00% 0.03% 0.02% 0 RADIUS
495 27500908 125788459 218 1.19% 0.70% 0.72% 0 Port manager per
555 153215372 12309297 12447 1.35% 2.20% 1.76% 0 IP NAT Ager
557 525376 32595667 16 0.00% 0.02% 0.01% 0 IGMP Input
558 302768 3315067 91 0.00% 0.01% 0.00% 0 PIM Process
559 344156 32519050 10 0.00% 0.01% 0.00% 0 Mwheel Process
562 4821188 317657 15177 0.00% 0.06% 0.09% 0 BGP Scanner
570 283060 13253757 21 0.00% 0.01% 0.00% 0 MLD

[Joseph W. Doherty]

I've crossed my fingers.  ; )

When you can, do a new show nat stats, and post.

[andresdavid]

bb01.netw.ro#show ip nat statistics
Total active translations: 22745 (0 static, 22745 dynamic; 22741 extended)
Outside interfaces:
Vlan1881, Vlan1882
Inside interfaces:
Vlan10
Hits: 1459980288 Misses: 0
CEF Translated packets: 1444177306, CEF Punted packets: 1182276784
Expired translations: 118814670
Dynamic mappings:
-- Inside Source
[Id: 9] access-list 1309 pool NAT_9 refcount 276
pool NAT_9: netmask 255.255.255.252
start 86.xxx.xx.0 end 86.xxx.xx.3
type generic, total addresses 4, allocated 1 (25%), misses 0
[Id: 10] access-list 1310 pool NAT_10 refcount 106
pool NAT_10: netmask 255.255.255.252
start 86.xxx.xx.4 end 86.xxx.xx.7
type generic, total addresses 4, allocated 1 (25%), misses 0
[Id: 11] access-list 1311 pool NAT_11 refcount 592
pool NAT_11: netmask 255.255.255.252
start 86.xxx.xx.8 end 86.xxx.xx.11
type generic, total addresses 4, allocated 1 (25%), misses 0
[Id: 12] access-list 1312 pool NAT_12 refcount 317
pool NAT_12: netmask 255.255.255.252
start 86.xxx.xx7.12 end 86.xxx.xx.15
type generic, total addresses 4, allocated 1 (25%), misses 0
[Id: 13] access-list 1313 pool NAT_13 refcount 1167
pool NAT_13: netmask 255.255.255.252
start 86.xxx.xx.16 end 86.xxx.xx.19
type generic, total addresses 4, allocated 1 (25%), misses 0
[Id: 14] access-list 1314 pool NAT_14 refcount 1885
pool NAT_14: netmask 255.255.255.252
start 86.xxx.xx.20 end 86.xxx.xx.23
type generic, total addresses 4, allocated 2 (50%), misses 136
[Id: 15] access-list 1315 pool NAT_15 refcount 1751
pool NAT_15: netmask 255.255.255.252
start 86.xxx.xx.24 end 86.xxx.xx.27
type generic, total addresses 4, allocated 1 (25%), misses 0
[Id: 16] access-list 1316 pool NAT_16 refcount 768
pool NAT_16: netmask 255.255.255.252
start 86.xxx.xx.28 end 86.xxx.xx.31
type generic, total addresses 4, allocated 1 (25%), misses 0
[Id: 17] access-list 1317 pool NAT_17 refcount 782
pool NAT_17: netmask 255.255.255.252
start 94.xxx.xx.0 end 94.xxx.xx.3
type generic, total addresses 4, allocated 1 (25%), misses 0
[Id: 18] access-list 1318 pool NAT_18 refcount 1026
pool NAT_18: netmask 255.255.255.252
start 94.xxx.xx.4 end 94.xxx.xx.7
type generic, total addresses 4, allocated 1 (25%), misses 0
[Id: 19] access-list 1319 pool NAT_19 refcount 1078
pool NAT_19: netmask 255.255.255.252
start 94.xxx.xx.8 end 94.xxx.xx.11
type generic, total addresses 4, allocated 2 (50%), misses 34
[Id: 20] access-list 1320 pool NAT_20 refcount 772
pool NAT_20: netmask 255.255.255.252
start 94.xxx.xx.12 end 94.xxx.xx.15
type generic, total addresses 4, allocated 1 (25%), misses 0
[Id: 21] access-list 1321 pool NAT_21 refcount 841
pool NAT_21: netmask 255.255.255.252
start 94.xxx.xx.16 end 94.xxx.xx.19
type generic, total addresses 4, allocated 1 (25%), misses 0
[Id: 22] access-list 1322 pool NAT_22 refcount 893
pool NAT_22: netmask 255.255.255.252
start 94.xxx.xx.20 end 94.xxx.xx.23
type generic, total addresses 4, allocated 1 (25%), misses 0
[Id: 23] access-list 1323 pool NAT_23 refcount 1792
pool NAT_23: netmask 255.255.255.252
start 94.xxx.xx.24 end 94.xxx.xx.27
type generic, total addresses 4, allocated 1 (25%), misses 0
[Id: 24] access-list 1324 pool NAT_24 refcount 1474
pool NAT_24: netmask 255.255.255.252
start 94.xxx.xx.28 end 94.xxx.xx.31
type generic, total addresses 4, allocated 2 (50%), misses 0
[Id: 25] access-list 1325 pool NAT_25 refcount 1080
pool NAT_25: netmask 255.255.255.252
start 94.xxx.xx.32 end 94.xxx.xx.35
type generic, total addresses 4, allocated 1 (25%), misses 0
[Id: 26] access-list 1326 pool NAT_26 refcount 768
pool NAT_26: netmask 255.255.255.252
start 94.xxx.xx.36 end 94.xxx.xx.39
type generic, total addresses 4, allocated 1 (25%), misses 0
[Id: 27] access-list 1327 pool NAT_27 refcount 865
pool NAT_27: netmask 255.255.255.252
start 94.xxx.xx.40 end 94.xxx.xx.43
type generic, total addresses 4, allocated 1 (25%), misses 0
[Id: 28] access-list 1328 pool NAT_28 refcount 824
pool NAT_28: netmask 255.255.255.252
start 94.xxx.xx.44 end 94.xxx.xx.47
type generic, total addresses 4, allocated 1 (25%), misses 0
[Id: 29] access-list 1329 pool NAT_29 refcount 601
pool NAT_29: netmask 255.255.255.252
start 94.xxx.xx.48 end 94.xxx.xx.51
type generic, total addresses 4, allocated 1 (25%), misses 0
[Id: 30] access-list 1330 pool NAT_30 refcount 1200
pool NAT_30: netmask 255.255.255.252
start 94.xxx.xx.52 end 94.xxx.xx.55
type generic, total addresses 4, allocated 2 (50%), misses 0
[Id: 31] access-list 1331 pool NAT_31 refcount 647
pool NAT_31: netmask 255.255.255.252
start 94.xxx.xx.56 end 94.xxx.xx.59
type generic, total addresses 4, allocated 1 (25%), misses 0
[Id: 32] access-list 1332 pool NAT_32 refcount 1284
pool NAT_32: netmask 255.255.255.252
start 94.xxx.xx.60 end 94.xxx.xx.63
type generic, total addresses 4, allocated 2 (50%), misses 0
[Id: 33] access-list 1666 pool NAT_666 refcount 0
pool NAT_666: netmask 255.255.255.252
start 46.xxx.xxx.0 end 46.xxx.xxx.3
type generic, total addresses 4, allocated 0 (0%), misses 0

I am also attaching a picture of the cpu:

[Joseph W. Doherty]

Well, those stats look much better, and they're encouraging.

Active NAT flows about 1/10 of earlier stats, still without "misses".

Dramatic decrease in "IP Input" process CPU usage.

No one is complaining of any "new" network issues?

If not, possibly you have a long term improvement.  We'll see how it goes for the rest of the week.

[andresdavid]

I see that the CPU does not increase more than 25-30%, I will see how it behaves these days, the internet works without problems for all clients.

thank you for all the support @Joseph W. Doherty  @MHM Cisco World  @Leo Laohoo 

[MHM Cisco World]

Finally happy ending of this issue.

Great job team 

Have a nice day 

MHM

[Rich R]

Coming to this thread late so I'll just add some general comments.
- Like somebody said earlier some features will always get punted to RP instead in being switched in line-card - depends on hardware and software versions - so always check docs and release notes.  Enabling some types of spanning can also sometimes cause all traffic to get punted without any warning (I got caught by this many years ago).
- With NAT the first packet in a flow will always be process switched via RP to establish the flow.  After that (and providing there's enough TCAM) the following packets are generally switched by the line card.  However certain types of traffic which require ALG  processing will generally always get punted to RP - like DNS for example.  If you don't need an ALG you can disable it to reduce the processing overhead - then only the packet header gets NAT like every other packet rather than having to inspect and (where required) change the contents/payload of the packet.  That might be the case if it's a simple internet access where only your client IP gets NAT but server IPs of DNS server and in DNS replies never need to be changed.
eg: no ip nat service dns
- 6500 is pretty old architecture/technology now.  The newer platforms recognised the need to handle more of the CPU intensive operations in "hardware switching" - hence the use of QFP and the other more recent switching technologies (UADP, Silicon One) to achieve that in newer platforms.  IOS-XE also enabled the use of multi-core processors to spread the load when necessary.
- As already mentioned - ACL logging should be exceptional for packet filtering ACLs - that will always cause punts and even when they're rate limited impact can be high.  So as a rule do not log on packet filter ACLs.  For control plane stuff like VTY ACL - that's slightly different because that's getting punted anyway but still be careful.