NAT causing CPU to increase

[smuneio]

I have a Cisco 7513 and then I have four Cisco 3640's out in the field hanging off the 7513 via Serial ports. Each Cisco 3640 has 4 t1's feeding to the 7513. Just last week I took NAT off the Cisco 3640's (due to high CPU usage) and added one big NAT pool on the 7513, all traffic from the 3640's feed into this one nat pool on the 7513 and then out my 100 Meg pipe to the internet. I have about 2000 High speed internet customers running off these 3640's. Before I enabled NAT on the 7513 my CPU ran about 30%, now that I have NAT running the CPU will sit at about 85%. This is how I have NAT configured:

ip nat translation timeout 900

ip nat translation tcp-timeout 3600

ip nat pool RAT-OVLD 66.193.237.16 66.193.237.20 prefix-length 24

ip nat inside source list 7 pool RAT-OVLD overload

access-list 7 permit 10.0.0.0 0.0.3.255

access-list 7 permit 192.168.2.0 0.0.0.255

access-list 7 permit 10.1.0.0 0.0.3.255

access-list 7 permit 10.0.4.0 0.0.3.255

is there a better way to configure NAT?

Here is a show process CPU:

CPU utilization for five seconds: 78%/65%; one minute: 75%; five minutes: 78%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

1 488 73323 6 0.00% 0.00% 0.00% 0 Load Meter

2 0 3 0 0.00% 0.00% 0.00% 0 PPP auth

3 24 3056 7 0.00% 0.00% 0.00% 0 DHCPD Timer

4 195300 43350 4505 0.00% 0.05% 0.01% 0 Check heaps

5 0 1 0 0.00% 0.00% 0.00% 0 Chunk Manager

6 156 201 776 0.00% 0.00% 0.00% 0 Pool Manager

7 0 2 0 0.00% 0.00% 0.00% 0 Timers

8 0 2 0 0.00% 0.00% 0.00% 0 Serial Backgroun

9 0 1 0 0.00% 0.00% 0.00% 0 OIR Handler

10 0 1 0 0.00% 0.00% 0.00% 0 IPC Zone Manager

11 1520 366434 4 0.00% 0.00% 0.00% 0 IPC Periodic Tim

12 1072 366434 2 0.00% 0.00% 0.00% 0 IPC Deferred Por

13 17320 72661 238 0.00% 0.00% 0.00% 0 IPC Seat Manager

14 556172 2218627 250 0.00% 0.14% 0.07% 0 ARP Input

15 1448 86381 16 0.00% 0.00% 0.00% 0 HC Counter Timer

16 0 6 0 0.00% 0.00% 0.00% 0 DDR Timers

17 0 2 0 0.00% 0.00% 0.00% 0 Dialer event

18 0 1 0 0.00% 0.00% 0.00% 0 Entity MIB API

19 0 1 0 0.00% 0.00% 0.00% 0 SERIAL A'detect

20 0 1 0 0.00% 0.00% 0.00% 0 Microcode Loader

21 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd

22 19956 424095 47 0.00% 0.00% 0.00% 0 Net Background

23 12 939 12 0.00% 0.00% 0.00% 0 Logger

24 2096 366431 5 0.00% 0.00% 0.00% 0 TTY Background

25 3680 366434 10 0.00% 0.00% 0.00% 0 Per-Second Jobs

26 0 1 0 0.00% 0.00% 0.00% 0 Inode Table Dest

27 0 1 0 0.00% 0.00% 0.00% 0 Inode Table Refr

28 0 1 0 0.00% 0.00% 0.00% 0 stuckinfo_proces

29 0 1 0 0.00% 0.00% 0.00% 0 IP Crashinfo Inp

30 0 1 0 0.00% 0.00% 0.00% 0 DSX3MIB ll handl

31 0 2 0 0.00% 0.00% 0.00% 0 VSI Master

32 8700 366433 23 0.00% 0.00% 0.00% 0 RSP Background

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

33 0 1 0 0.00% 0.00% 0.00% 0 Memory Scanner

34 88 6110 14 0.00% 0.00% 0.00% 0 Slave Time

35 0 1 0 0.00% 0.00% 0.00% 0 Slave IPC OIR

36 48 178 269 0.00% 0.00% 0.00% 0 Exec

37 6396 103758 61 0.00% 0.00% 0.00% 0 Chassis Daemon

38 8 6111 1 0.00% 0.00% 0.00% 0 RSP Chassis Back

39 3328 311732 10 0.00% 0.00% 0.00% 0 MIP Mailbox

40 0 17 0 0.00% 0.00% 0.00% 0 vcq_proc

41 0 1 0 0.00% 0.00% 0.00% 0 CT3 Mailbox

42 0 1 0 0.00% 0.00% 0.00% 0 CE3 Mailbox

43 19392 146478 132 0.00% 0.00% 0.00% 0 IPC CBus process

44 0 2 0 0.00% 0.00% 0.00% 0 ATM OAM Input

45 0 2 0 0.00% 0.00% 0.00% 0 ATM OAM TIMER

46 98169796 175244920 560 9.74% 8.41% 8.00% 0 IP Input

47 290336 147157 1972 0.00% 0.06% 0.05% 0 CDP Protocol

48 28 607 46 0.00% 0.00% 0.00% 0 MOP Protocols

49 0 1 0 0.00% 0.00% 0.00% 0 X.25 Encaps Mana

50 52 6456 8 0.00% 0.00% 0.00% 0 LDP Background

51 0 1 0 0.00% 0.00% 0.00% 0 frr_tunnel

52 11932 13159 906 0.00% 0.00% 0.00% 0 IP Background

53 808 6142 131 0.00% 0.00% 0.00% 0 IP RIB Update

54 0 1 0 0.00% 0.00% 0.00% 0 SNMP Timers

55 0 10 0 0.00% 0.00% 0.00% 0 PPP IP Add Route

56 72 1490 48 0.00% 0.00% 0.00% 0 TCP Timer

57 40 56 714 0.00% 0.00% 0.00% 0 TCP Protocols

58 0 1 0 0.00% 0.00% 0.00% 0 Probe Input

59 0 1 0 0.00% 0.00% 0.00% 0 RARP Input

60 0 1 0 0.00% 0.00% 0.00% 0 HTTP Timer

61 0 1 0 0.00% 0.00% 0.00% 0 Socket Timers

62 2540 6971 364 0.00% 0.00% 0.00% 0 DHCPD Receive

63 62644 6110 10252 0.16% 0.01% 0.00% 0 IP Cache Ager

64 0 1 0 0.00% 0.00% 0.00% 0 PAD InCall

65 0 2 0 0.00% 0.00% 0.00% 0 X.25 Background

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

66 292 6110 47 0.00% 0.00% 0.00% 0 TCP Intercept Ti

67 0 2 0 0.00% 0.00% 0.00% 0 SPX Input

68 8252 21619 381 0.00% 0.00% 0.00% 0 Adj Manager

69 127136 596384 213 0.00% 0.03% 0.06% 0 CEF process

70 10736 1829076 5 0.00% 0.00% 0.00% 0 MDFS RP process

71 0 2 0 0.00% 0.00% 0.00% 0 TC-ATM Proc

72 0 2 0 0.00% 0.00% 0.00% 0 Tag Input

73 0 1 0 0.00% 0.00% 0.00% 0 Inspect Timer

74 0 1 0 0.00% 0.00% 0.00% 0 Authentication P

75 0 1 0 0.00% 0.00% 0.00% 0 IDS Timer

76 5040 5373 938 0.00% 0.14% 0.36% 2 Virtual Exec

77 0 6 0 0.00% 0.00% 0.00% 0 Crypto Support

78 0 1 0 0.00% 0.00% 0.00% 0 Encrypt Proc

79 0 4 0 0.00% 0.00% 0.00% 0 Key Proc

80 10216 366664 27 0.00% 0.00% 0.00% 0 Crypto SM

81 0 3 0 0.00% 0.00% 0.00% 0 Crypto CA

82 464 24442 18 0.00% 0.00% 0.00% 0 Crypto IKMP

83 184 40740 4 0.00% 0.00% 0.00% 0 IPSEC key engine

84 0 1 0 0.00% 0.00% 0.00% 0 IPSEC manual key

85 0 1 0 0.00% 0.00% 0.00% 0 SYSMGT Events

86 7400 3718 1990 0.00% 0.00% 0.00% 0 Syslog Traps

87 1776 732896 2 0.00% 0.00% 0.00% 0 cbus utilization

88 4328 87029 49 0.00% 0.00% 0.00% 0 Net Input

89 2016 73323 27 0.00% 0.00% 0.00% 0 Compute load avg

90 58828 6110 9628 0.00% 0.01% 0.00% 0 Per-minute Jobs

91 29992 33191 903 0.00% 0.01% 0.00% 0 IP SNMP

92 0 1 0 0.00% 0.00% 0.00% 0 SNMP ConfCopyPro

93 4 2 2000 0.00% 0.00% 0.00% 0 SNMP Traps

94 4 2 2000 0.00% 0.00% 0.00% 0 CCP manager

95 32664 259482 125 0.00% 0.00% 0.00% 0 PPP manager

96 26985860 286184608 94 1.88% 1.69% 1.75% 0 Multilink PPP

97 27341456 157516064 173 0.81% 0.90% 0.88% 0 Multilink PPP ou

98 4 3 1333 0.00% 0.00% 0.00% 0 Multilink event

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

99 1348012 20368134 66 0.24% 0.29% 0.26% 0 IP NAT Ager

100 0 1 0 0.00% 0.00% 0.00% 0 IP SNMP PROXY MG

101 0 1 0 0.00% 0.00% 0.00% 0 SNMP Manager

103 496 19325 25 0.00% 0.00% 0.00% 0 CEF Scanner

104 628 103855 6 0.00% 0.00% 0.00% 0 DHCPD Database

Can anyone help?

[Ladislaus]

A nat sessions takes up a certain amount of RAM and consumes a certain amount of CPU. And putting 2000 customers, each one doing a lot of connections, naturally puts quite a strain on your router. And there's not much you can do about that.

Now, what you can do, is try and minimize the numbers of NAT sessions and thereby the load on your CPU.

Lower the timeouts, this way old sessions won't be in your way for as long.

I administer, among other things, a school with a 1605 (quite far from a 7513 but anyway), we had about 2500 NAT sessions going through it, the CPU was at a constant 85%. I took a look at what was actually going through (show ip nat translations) and found out that 70% of the traffic was virus related. By using a simple access list the CPU usage was lowered to about 35%.

Check yours to just to get a better view of what is causing the load.

Lastly, upgrade your routers..

Your previous set up seemed like a better choice if you ask me. Split up your clients and put them on smaller routers each one taking care of NAT then aggregate it to your 7513.

Debugging will be easier and above all, upgrades. Upgrading will simply be purchasing a new router and moving a few clients to it.

[Georg Pauwen]

Hello,

I ran your output through the output interpreter, here is what it says:

WARNING: Interrupt CPU Utilization is 65%, which is very high (>60%)

CPU interrupts are primarily caused by fast switching of traffic. Interrupts are

also generated any time a character is output from the console or auxiliary

ports of a router.

TRY THIS: One of the following may be causing this to happen:

- Configured voice ports: Even if there is no traffic, software continues to

monitor channel associated signaling (CAS).

- Active ATM interfaces: Even with no traffic, ATM interfaces continue to send

null cells (per ATM standard).

- An inappropriate switching path is configured on the router. If you have a

Cisco 7000 or Cisco 7500 series router, try improving its performance by

using the 'ip route-cache {path}' command (path can be cef, distributed, or

cbus, depending on the platform). If there are access lists linked to

interfaces or if ip accounting is turned on, configure NetFlow switching

using the 'ip route-cache flow' command.

- The CPU is performing memory alignment corrections. If there are

%ALIGN-3-CORRECT messages logged, then the high CPU utilization is caused by

memory alignment corrections. Capture the output of the 'show align' command,

decode the tracebacks and search for a bug in your version of IOS.

- The router is overloaded with traffic. Use the 'show interfaces' command and

paste into Output Interpreter to determine which interface is overloaded

- There maybe a bug in the version of IOS running on the router. Check the Bug

Navigator

for a bug that reports similar symptoms in a similar environment.

REFERENCE: For additional troubleshooting information, please visit Troubleshooting

High CPU Utilization on Cisco Routers

Since the voice and ATM stuff probably doesn´t apply to you, I would try and configure the fast switching first. If that doesn´t help, can you post the IOS version you are running ?

Regards,

Georg