NAT ISSUE CPU

[choylee]

Hello,

I have an router Cisco 7206 VXR with IOS Version 12.4(24)T .

I have configured a static NAT on the port 80.

By default there is a maximum number of allowed NAT entries = Range is from 1 to 2147483647.

Also I have seen in this forum NAT entry requires about 160-312 bytes of memory.

Is it normal to have 30% of CPU for 16366 NAT translation ?  how can I lower it ?

Why is the memory isn't used ?

r01#show processes memory | include NAT
Processor Pool Total:  376142292 Used:   59983884 Free:  316158408
      I/O Pool Total:   33554432 Used:    4374304 Free:   29180128
Transient Pool Total:   16777216 Used:      62156 Free:   16715060

PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs Process

256   0       4272        200864       7224          0          0 IP NAT Ager
257   0          0             0             7204          0          0 IP NAT WLAN
268   0          0             0             7204          0          0 NAT MIB Helper

Configuration :

ip nat translation timeout 30
ip nat translation tcp-timeout 15
ip nat translation dns-timeout 300
ip nat translation icmp-timeout 120
ip nat translation arp-ping-timeout 120
ip nat inside source static tcp 10.100.0.40 80 X.X.X.X 80 redundancy redundancy-hsrp extendable

CPU utilization for five seconds: 27%/24%; one minute: 28%; five minutes: 30%
PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
256       32228      278210        115  0.71%  0.79%  0.79%   0 IP NAT Ager
140        1600    35313136          0  0.47%  0.48%  0.47%   0 HQF Shaper Backg
  41      570332      143190       3983  0.47%  0.45%  0.47%   0 Per-Second Jobs
268      115008     3380968         34  0.47%  0.50%  0.49%   0 NAT MIB Helper
  82      248360      919053        270  0.23%  0.22%  0.29%   0 IP Input
271         504         593        849  0.15%  0.01%  0.00%   3 SSH Process
254        1212     8879337          0  0.15%  0.16%  0.15%   0 HSRP Common

r01#show ip nat  statistics
Total active translations: 15882 (6 static, 15876 dynamic; 15880 extended)
Peak translations: 16366, occurred 00:10:15 ago
Outside interfaces:
  FastEthernet2/0.2874, FastEthernet4/0.10
Inside interfaces:
  FastEthernet1/0
Hits: 29937454  Misses: 0
CEF Translated packets: 29868520, CEF Punted packets: 68936
Expired translations: 2582310
Dynamic mappings:
-- Inside Source
[Id: 1] route-map NoNat interface FastEthernet2/0.2874 refcount 12
Appl doors: 0
Normal doors: 0
Queued Packets: 0

Thanks

[Giuseppe Larosa]

Hello ChoyLee,

you can use

sh proc cpu sorted

sh proc cpu sorted 1min

to see what processes are using most cpu resources

we cannot say if NAT is causing this cpu usage with info you have provided up to now, notice that C7200 is still a SW based router so it uses cpu also for forwarding (even for CEF)

Hope to help

Giuseppe

[choylee]

Hello,

In fact , I have begun my http migration and since yesterday the CPU was 3 % and today 30%

For you information, we have about 500 hits per seconds.

Also , I perform "ip cef " on my router

r01#sh proc cpu sorted
CPU utilization for five seconds: 29%/25%; one minute: 29%; five minutes: 29%
PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
256       86388      290298        297  0.79%  0.76%  0.78%   0 IP NAT Ager
140        2584    36817103          0  0.47%  0.48%  0.47%   0 HQF Shaper Backg
  41      597752      149512       3998  0.47%  0.45%  0.47%   0 Per-Second Jobs
  82      276364      977835        282  0.47%  0.33%  0.25%   0 IP Input
268      181220     4805901         37  0.47%  0.49%  0.47%   0 NAT MIB Helper
274         404         530        762  0.39%  0.27%  0.08%   4 SSH Process
254        1556     9262529          0  0.15%  0.17%  0.16%   0 HSRP Common
  79       56280      389032        144  0.07%  0.04%  0.05%   0 ADJ resolve proc
  19       62384      481656        129  0.07%  0.04%  0.05%   0 ARP Input
  32          20      148682          0  0.07%  0.00%  0.00%   0 GraphIt
  11           0           1          0  0.00%  0.00%  0.00%   0 IPC Zone Manager
  10           0        2479          0  0.00%  0.00%  0.00%   0 IPC Dynamic Cach
  13          16      145190          0  0.00%  0.00%  0.00%   0 IPC Deferred Por
  12           4      145190          0  0.00%  0.00%  0.00%   0 IPC Periodic Tim
   9           0           2          0  0.00%  0.00%  0.00%   0 ATM VC Auto Crea
  16       66560      148692        447  0.00%  0.00%  0.00%   0 EnvMon
PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
  17           0           1          0  0.00%  0.00%  0.00%   0 OIR Handler
  18           0           1          0  0.00%  0.00%  0.00%   0 Crash writer
   8           0           2          0  0.00%  0.00%  0.00%   0 ATM AutoVC Perio
  14           0           1          0  0.00%  0.00%  0.00%   0 IPC Seat Manager
  15           0           1          0  0.00%  0.00%  0.00%   0 IPC BackPressure
   7           0           2          0  0.00%  0.00%  0.00%   0 Timers
   5      167732       17744       9452  0.00%  0.08%  0.10%   0 Check heaps
  24           0           1          0  0.00%  0.00%  0.00%   0 Policy Manager
  25           0           2          0  0.00%  0.00%  0.00%   0 DDR Timers
  26           0           4          0  0.00%  0.00%  0.00%   0 Entity MIB API
  27          20         204         98  0.00%  0.00%  0.00%   0 EEM ED Syslog
  28           0           2          0  0.00%  0.00%  0.00%   0 Serial Backgroun
  29           0           1          0  0.00%  0.00%  0.00%   0 RO Notify Timers
  30           0           1          0  0.00%  0.00%  0.00%   0 RMI RM Notify Wa
  31           0           2          0  0.00%  0.00%  0.00%   0 SMART
  21           0           2          0  0.00%  0.00%  0.00%   0 ATM Idle Timer
  33           0           2          0  0.00%  0.00%  0.00%   0 Dialer event

r01#sh proc cpu sorted 1min
CPU utilization for five seconds: 28%/26%; one minute: 28%; five minutes: 28%
PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
256       87488      290556        301  0.71%  0.75%  0.77%   0 IP NAT Ager
140        2600    36849634          0  0.47%  0.49%  0.47%   0 HQF Shaper Backg
268      182588     4835667         37  0.47%  0.48%  0.47%   0 NAT MIB Helper
  41      598308      149644       3998  0.47%  0.46%  0.47%   0 Per-Second Jobs
254        1564     9270775          0  0.15%  0.16%  0.15%   0 HSRP Common
  82      276600      978597        282  0.07%  0.15%  0.20%   0 IP Input
  37       19276       99342        194  0.00%  0.09%  0.10%   0 Net Background
   5      168052       17763       9460  0.00%  0.08%  0.10%   0 Check heaps
  16       66724      148824        448  0.07%  0.05%  0.02%   0 EnvMon
251         528     1486675          0  0.00%  0.04%  0.05%   0 CCPROXY_CT
  19       62488      482066        129  0.00%  0.04%  0.05%   0 ARP Input
  79       56364      389337        144  0.00%  0.04%  0.05%   0 ADJ resolve proc
274         452         589        767  0.07%  0.03%  0.05%   4 SSH Process
   2         240       29764          8  0.00%  0.03%  0.02%   0 Load Meter
141         300     1486577          0  0.07%  0.02%  0.00%   0 RBSCP Background
  50       47524        2667      17819  0.00%  0.01%  0.00%   0 Per-minute Jobs
PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
163       31504        9969       3160  0.00%  0.01%  0.00%   0 BGP Scanner
107         264      210439          1  0.07%  0.01%  0.00%   0 CEF: IPv4 proces
188       39332       27200       1446  0.00%  0.01%  0.00%   0 BGP Router
  40          32      148852          0  0.00%  0.00%  0.00%   0 TTY Background
150       14592       21829        668  0.00%  0.00%  0.00%   0 BGP I/O
  13          16      145319          0  0.00%  0.00%  0.00%   0 IPC Deferred Por
104          36        3641          9  0.00%  0.00%  0.00%   0 IP Background
  20         808      155220          5  0.00%  0.00%  0.00%   0 ARP Background
149          56      290551          0  0.00%  0.00%  0.00%   0 Inspect process
  99           0         497          0  0.00%  0.00%  0.00%   0 Authentication P
  26           0           4          0  0.00%  0.00%  0.00%   0 Entity MIB API
  27          20         204         98  0.00%  0.00%  0.00%   0 EEM ED Syslog
  28           0           2          0  0.00%  0.00%  0.00%   0 Serial Backgroun
  29           0           1          0  0.00%  0.00%  0.00%   0 RO Notify Timers
  31           0           2          0  0.00%  0.00%  0.00%   0 SMART
  32          20      148815          0  0.00%  0.00%  0.00%   0 GraphIt
  30           0           1          0  0.00%  0.00%  0.00%   0 RMI RM Notify Wa
  34           0           1          0  0.00%  0.00%  0.00%   0 SERIAL A'detect 

Thanks you

[Giuseppe Larosa]

Hello ChoyLee,

sorry for having asked additional information.

most of cpu usage is caused by interrupts and this could be seen also on first post.

CPU utilization for five seconds: 29%/25%; one minute: 29%; five  minutes: 29%

so you should investigate on why a consistent part of traffic is process switched instead of being processed by CEF.

CEF is more efficient and causes less load. NAT can be the root cause of this as you noted.

the following document is specific of  a case like this

http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af0.shtml

following this document you can go further in this analysis

Hope to help

Giuseppe

[choylee]

Thank you

I think I am on the right track (Causes of high CPU utilization due to interrupts)

I have CEF (ip cef) and Netflow configuration on my routeur (route-cache flow )

I note in your web link that CEF cannot dropping packet while there is a lot of drops.

I suppose that is the issue and continue to search the resolution.

r01#show ip cef switching statistics

Path   Reason                                  Drop       Punt  Punt2Host
RP LES Packet destined for us             0      35655          0
RP LES Total                                       0      35655          0

RP PAS No route                                  7          0          7
RP PAS Packet destined for us             0      35655          0
RP PAS No adjacency                      15749          0          0
RP PAS TTL expired                              0          0       4234
RP PAS Features                               3550          0      17579
RP PAS Neighbor resolution req          8492          2          0
RP PAS Total                                    27798      35657      21820

All    Total                                           27798      71312      21820

r01#show ip cache flow
IP packet size distribution (4517478 total packets):
   1-32   64   96  128  160  192  224  256  288  320  352  384  416  448  480
   .000 .687 .003 .046 .001 .000 .078 .000 .000 .000 .000 .000 .001 .000 .000

    512  544  576 1024 1536 2048 2560 3072 3584 4096 4608
   .000 .005 .010 .114 .045 .000 .000 .000 .000 .000 .000

IP Flow Switching Cache, 4456704 bytes
  770 active, 64766 inactive, 690322 added
  1688010 ager polls, 0 flow alloc failures
  Active flows timeout in 30 minutes
  Inactive flows timeout in 15 seconds
IP Sub Flow Cache, 533256 bytes
  0 active, 16384 inactive, 0 added, 0 added to flow
  0 alloc failures, 0 force free
  1 chunk, 1 chunk added
  last clearing of statistics never
Protocol         Total    Flows   Packets Bytes  Packets Active(Sec) Idle(Sec)
--------         Flows     /Sec     /Flow  /Pkt     /Sec     /Flow     /Flow
TCP-Telnet           4      0.0         4    56      0.0       6.9      15.6
TCP-WWW         684479    116.6         6   213    703.3       0.1       1.5
TCP-BGP             51      0.0         3   490      0.0       5.8      15.5
TCP-other          981      0.1         5    60      0.9       8.1      14.7
UDP-DNS           1543      0.2         1    63      0.3       0.7      15.4
UDP-other         2282      0.3       136   202     52.8      12.4      15.4
ICMP               212      0.0        12    55      0.4       1.3      15.4
Total:          689552    117.4         6   212    758.0       0.2       1.6

[choylee]

Hello,

I have so much "drop and punt" packets.

I think I disable the cef on my router.

TYPE OF ENTRY IN ADJACENCY TABLE

Cache           : MAC header rewrite string and outgoing interface.

Receive         : Directly connected interface i.e. fa0/0 192.168.1.1/24

                            192.168.1.1/32 receive

                             192.168.1.255/32 receive

                             192.168.1.0/32 receive

Drop             : Destination for which packet will be dropped.

                   A – Packet denied by access-list

                     B – Route pointing to Null0 Interface

Punt             : CEF can not forward this and sends it to other best switching method.

Thanks

[Giuseppe Larosa]

Hello Choy,

disabling CEF is not recommended a lot of features are CEF dependent.

I would try using a different  IOS image just to see if the behavior change as you noted in the other thread in WAN forum

I would suggest to move back to 12.4(20)T

Hope to help

Giuseppe

[choylee]

Thank you

I am going to verify with two IOS the next week :

First update : 12.4(20)T
Routers > Cisco 7206VXR Router > IOS Software > 12.4.20T > fichier c7200-adventerprisek9-mz.124-20.T.bin

Second  update : 12.2 SRE (c7200-adventerprisek9-mz.122-33.SRE.bin) because :  https://supportforums.cisco.com/thread/345008

Routers > Cisco 7206VXR Router > IOS Software > 12.2 SRE >  fichier c7200-adventerprisek9-mz.122-33.SRE.bin

Note :  My router have rebooted one more time

Crashinfo on my cisco  : Cause 00000010 (Code 0x4): Address Error (load or instruction fetch) exception

[choylee]

Hello,

I am despaired

I upgraded the IOS 12.2 SRE and I have the same behaviour

r01#show processes cpu sorted
CPU utilization for five seconds: 11%/9%; one minute: 10%; five minutes: 9%
PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
   4       77788        9676       8039  1.11%  0.20%  0.13%   0 Check heaps
237       88420       63843       1384  0.39%  0.41%  0.39%   0 Per-Second Jobs
168       15152    15778238          0  0.39%  0.37%  0.35%   0 HQF Input Shaper
   3         212         249        851  0.31%  0.20%  0.05%   2 SSH Process
167       16092    15778235          1  0.31%  0.29%  0.29%   0 HQF Shaper Backg
  86       78364      416287        188  0.23%  0.22%  0.24%   0 IP Input
243         736     3955736          0  0.15%  0.13%  0.14%   0 HSRP Common
   2         444       12708         34  0.07%  0.02%  0.00%   0 Load Meter
   9       40444      207111        195  0.07%  0.07%  0.07%   0 ARP Input
142         124       88092          1  0.07%  0.01%  0.00%   0 CEF: IPv4 proces
  23           0       12708          0  0.07%  0.00%  0.00%   0 IPC Event Notifi
113        1896     1979357          0  0.07%  0.06%  0.07%   0 IPAM Manager
242         184      124037          1  0.07%  0.10%  0.08%   0 IP NAT Ager
  80       26704      177761        150  0.07%  0.06%  0.07%   0 ADJ resolve proc
  14           0           1          0  0.00%  0.00%  0.00%   0 Policy Manager
  15           0           2          0  0.00%  0.00%  0.00%   0 DDR Timers
  17           8          39        205  0.00%  0.00%  0.00%   0 EEM ED Syslog
  16           4           3       1333  0.00%  0.00%  0.00%   0 Entity MIB API
  19           0           1          0  0.00%  0.00%  0.00%   0 RO Notify Timers
  13           0           1          0  0.00%  0.00%  0.00%   0 AAA_SERVER_DEADT
  21           0           2          0  0.00%  0.00%  0.00%   0 ATM AutoVC Perio
  22           4           2       2000  0.00%  0.00%  0.00%   0 ATM VC Auto Crea
  18          76           3      25333  0.00%  0.00%  0.00%   0 PrstVbl
  12           0           1          0  0.00%  0.00%  0.00%   0 ATM ASYNC PROC
  25           0        1060          0  0.00%  0.00%  0.00%   0 IPC Dynamic Cach
  26           0       12708          0  0.00%  0.00%  0.00%   0 IPC Service NonC
  27           0           1          0  0.00%  0.00%  0.00%   0 IPC Zone Manager
  20           0           1          0  0.00%  0.00%  0.00%   0 RMI RM Notify Wa
  11           0           2          0  0.00%  0.00%  0.00%   0 ATM Idle Timer
  30           0           1          0  0.00%  0.00%  0.00%   0 IPC Process leve
  31           0           1          0  0.00%  0.00%  0.00%   0 IPC Seat Manager
  32           0           1          0  0.00%  0.00%  0.00%   0 IPC Seat RX Cont
  33           0           1          0  0.00%  0.00%  0.00%   0 IPC Seat TX Cont
  34           0        6354          0  0.00%  0.00%  0.00%   0 IPC Keep Alive M
  24           4       62038          0  0.00%  0.00%  0.00%   0 IPC Mcast Pendin
  36           0        6354          0  0.00%  0.00%  0.00%   0 Compute SRP rate
  28          12       62038          0  0.00%  0.00%  0.00%   0 IPC Periodic Tim

r01#show ip nat statistics
Total active translations: 3257 (5 static, 3252 dynamic; 3256 extended)
Outside interfaces:
  FastEthernet2/0.2874, FastEthernet4/0.10
Inside interfaces:
  FastEthernet1/0
Hits: 5169583  Misses: 0
CEF Translated packets: 5168966, CEF Punted packets: 44688
Expired translations: 413597
Dynamic mappings:

Thanks

[choylee]

Hello,

In fact, I deleted the Nat configuration on my 7206 router.

Now, the Nat is done by iptable on Linux host behind my BGP router .

To conclude, I really think the issue was my NPE400 on my router .

It's not powerfull enought

Thanks