ā09-23-2011 12:19 AM - edited ā03-04-2019 01:42 PM
Hi ,
We have a 3845 router deployed at a tier 2 ISP for PATing user based sessions to the Internet ,the issue happening is when the NAT table size reaches approximately 2,000,000 the router gets hung and has to be restarted
the router has a memory of 256 MB RAM and approximately 100 MB free ,below is the show version and show proc mem
Can some one help me identify the issue we are facing
Cisco IOS Software, 3800 Software (C3845-ENTBASE-M), Version 12.4(9)T3, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2007 by Cisco Systems, Inc.
Compiled Sat 24-Mar-07 00:05 by prod_rel_team
ROM: System Bootstrap, Version 12.4(13r)T, RELEASE SOFTWARE (fc1)
uptime is 1 hour, 9 minutes
System returned to ROM by power-on
System restarted at 12:05:53 UTC Fri Sep 16 2011
System image file is "flash:c3845-entbase-mz.124-9.T3.bin"
Cisco 3845 (revision 1.0) with 222208K/39936K bytes of memory.
Processor board ID FHK1118F0HT
2 FastEthernet interfaces
2 Gigabit Ethernet interfaces
4 Serial interfaces
4 Channelized (E1 or T1)/PRI ports
DRAM configuration is 64 bits wide with parity enabled.
479K bytes of NVRAM.
62720K bytes of ATA System CompactFlash (Read/Write)
Configuration register is 0x2102
Processor Pool Total: 169056224 Used: 69902956 Free: 99153268
I/O Pool Total: 40893952 Used: 11728272 Free: 29165680
PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 57736864 25274508 30778204 637 315627 *Init*
0 0 12128 263332 12128 0 0 *Sched*
0 0 1727472 1297972 642904 8 8 *Dead*
1 0 46224632 421764 45820620 0 0 Chunk Manager
2 0 252 252 4044 0 0 Load Meter
4 0 3364 252 10216 0 0 Check heaps
5 0 692420 38360 329932 45 0 Pool Manager
6 0 252 252 7044 0 0 Timers
7 0 0 0 7044 0 0 IPC Dynamic Cach
8 0 0 0 7044 0 0 IPC Zone Manager
9 0 0 0 7044 0 0 IPC Periodic Tim
10 0 0 0 7044 0 0 IPC Deferred Por
11 0 744 0 7788 0 0 IPC Seat Manager
12 0 0 0 7044 0 0 IPC BackPressure
13 0 0 0 13044 0 0 OIR Handler
14 0 0 0 25044 0 0 Crash writer
15 0 252 252 7044 0 0 Environmental mo
16 0 4060 452436 8584 783320 783320 ARP Input
17 0 252 252 7044 0 0 ATM Idle Timer
18 0 252 252 7044 0 0 AAA high-capacit
19 0 0 0 7044 0 0 AAA_SERVER_DEADT
20 0 0 0 13044 0 0 Policy Manager
21 0 252 252 7044 0 0 DDR Timers
22 0 7896 0 14940 50 50 Entity MIB API
23 0 262884 0 272928 1496565 1496533 EEM ED Syslog
24 0 0 0 7044 0 0 HC Counter Timer
25 0 252 252 7044 0 0 Serial Backgroun
26 0 0 0 7044 0 0 RO Notify Timers
27 0 10052 0 14096 0 0 RMI RM Notify Wa
28 0 252 252 7044 0 0 SMART
29 0 252 252 7044 0 0 GraphIt
30 0 252 252 13044 0 0 Dialer event
31 0 0 0 7044 0 0 SERIAL A'detect
32 0 252 252 13044 0 0 XML Proxy Client
33 706 3170672 3078200 99848 311 310 Virtual Exec
34 0 0 0 4044 0 0 Inode Table Dest
35 0 0 0 7044 0 0 Critical Bkgnd
36 0 50260 800 13800 1726 1726 Net Background
37 0 2404 2004 13444 0 0 IDB Work
38 0 235168 96008 247960 1829403 1829404 Logger
39 0 252 560 7044 0 0 TTY Background
40 0 0 144620 10044 0 0 Per-Second Jobs
41 0 0 0 7044 0 0 AggMgr Process
42 0 0 0 4044 0 0 dev_device_inser
43 0 0 0 4044 0 0 dev_device_remov
44 0 0 0 13044 0 0 sal_dpc_process
45 0 0 0 7044 0 0 ARL Table Manage
46 0 252 252 7044 0 0 ESWPPM
47 0 252 252 4044 0 0 Eswilp Storm Con
48 0 252 252 7044 0 0 ESWILPPM
49 0 252 252 4044 0 0 Eswilp Storm Con
50 0 252 252 7044 0 0 Netclock Backgro
51 0 252 252 7044 0 0 SM Monitor
52 0 252 252 7044 0 0 Bryce I2C CMD Qu
54 0 252 252 7044 0 0 Ether-Switch RBC
55 0 0 0 4044 0 0 IGMP Snooping Pr
56 0 0 0 4044 0 0 IGMP Snooping Re
57 0 252 252 7044 0 0 Call Management
59 0 252 252 7044 0 0 Dot1x Mgr Proces
60 0 0 0 7044 0 0 MAB Framework
61 0 0 0 7044 0 0 EAP Framework
62 0 2500 252 9292 0 0 DTP Protocol
63 0 252 252 7044 0 0 PI MATM Aging Pr
64 0 252 252 7044 0 0 EtherChnl
65 0 252 252 7044 0 0 AAA Dictionary R
66 0 252 252 7044 0 0 AAA Server
67 0 0 0 7044 0 0 AAA ACCT Proc
68 0 0 0 7044 0 0 ACCT Periodic Pr
69 0 12944 252 19736 567 567 CDP Protocol
70 0 252 252 13044 0 0 Ethernet LMI
71 0 7466872 905724 1592508 1629871 1629868 IP Input
72 0 0 0 7044 0 0 ICMP event handl
73 0 252 252 7044 0 0 TurboACL
74 0 252 252 7044 0 0 TurboACL chunk
75 0 5704 252 10656 28 28 MOP Protocols
76 0 504 504 13044 0 0 PPP Hooks
78 0 0 0 13044 0 0 SSS Manager
79 0 0 0 13044 0 0 SSS Test Client
80 0 0 0 7044 0 0 SSS Feature Mana
81 0 0 0 7044 0 0 SSS Feature Time
82 0 0 0 13044 0 0 VPDN call manage
83 0 0 0 13100 0 0 L2X Socket proce
84 0 0 0 13044 0 0 L2X SSS manager
85 0 252 252 13044 0 0 L2TP mgmt daemon
86 0 0 0 7044 0 0 X.25 Encaps Mana
87 0 0 0 7044 0 0 IP Traceroute
88 0 580 0 10624 0 0 IP Background
89 0 164 0 10208 0 0 IP RIB Update
90 0 252 252 13044 0 0 PPP IP Route
91 0 252 252 13044 0 0 PPP IPCP
92 0 0 0 7044 0 0 SNMP Timers
93 0 73444 0 73444 0 0 CEF process
94 0 0 4092 13044 31 31 TCP Timer
95 0 186068 0 13044 0 0 TCP Protocols
96 0 0 0 25044 0 0 COPS
97 0 0 0 7044 0 0 Socket Timers
98 0 416 252 7208 0 0 Dot1x Supplicant
99 0 416 252 7208 0 0 Dot1x Supplicant
100 0 416 252 7208 0 0 Dot1x Supplicant
101 0 500 156 7544 0 0 L2MM
102 0 0 0 7044 0 0 MRD
103 0 5052 0 12096 0 0 IGMPSN
104 0 252 252 7044 0 0 RLM groups Proce
105 0 122064 252 128856 0 0 SCTP Main Proces
106 0 0 0 7044 0 0 IUA Main Process
107 0 252 252 7044 0 0 RUDPV1 Main Proc
108 0 0 0 7044 0 0 bsm_timers
109 0 0 0 7044 0 0 bsm_xmt_proc
110 0 0 0 10044 0 0 CES Client SVC R
111 0 120988 252 119692 0 0 DHCPD Receive
112 0 252 252 7044 0 0 Dialer Forwarder
113 0 0 4697092 7044 0 0 IP Cache Ager
114 0 252 252 10044 0 0 Adj Manager
115 0 252 252 13044 0 0 ATM OAM Input
116 0 252 252 13044 0 0 ATM OAM TIMER
117 0 940 0 10984 0 0 HTTP CORE
118 0 0 0 7044 0 0 RARP Input
119 0 0 0 7044 0 0 PAD InCall
120 0 252 252 13044 0 0 X.25 Background
121 0 252 252 7044 0 0 PPP Bind
122 0 252 252 7044 0 0 PPP SSS
123 0 0 0 7044 0 0 MQC Flow Event B
124 0 252 252 7044 0 0 RBSCP Background
125 0 252 252 7044 0 0 CRM_CALL_UPDATE_
127 0 252 252 7044 0 0 AAA Cached Serve
128 0 252 252 7044 0 0 ENABLE AAA
129 0 0 0 7044 0 0 EM Background Pr
130 0 0 0 7044 0 0 Key chain liveke
131 0 252 252 7044 0 0 LINE AAA
132 0 252 252 7044 0 0 LOCAL AAA
133 0 1032 252 7824 0 0 TPLUS
134 0 252 252 7044 0 0 Control-plane ho
135 0 0 0 7044 0 0 PM Callback
136 0 252 252 7044 0 0 AAA SEND STOP EV
137 0 0 0 10044 0 0 EEM ED CLI
138 0 0 0 10044 0 0 EEM ED Counter
139 0 0 0 10044 0 0 EEM ED Interface
140 0 0 0 10044 0 0 EEM ED IOSWD
141 0 0 0 10044 0 0 EEM ED Memory-th
142 0 0 0 10044 0 0 EEM ED None
143 0 0 0 10044 0 0 EEM ED OIR
144 0 76 0 10120 0 0 EEM ED Resource
145 0 0 0 10044 0 0 EEM ED SNMP
146 0 0 0 10044 0 0 EEM ED Timer
147 0 0 0 10044 0 0 EEM ED Track
148 0 13408 3684 17428 0 0 EEM Server
149 0 252 252 7044 0 0 RMON Recycle Pro
150 0 252 252 7044 0 0 RMON Deferred Se
151 0 0 0 7044 0 0 Syslog Traps
152 0 9240 1564 14564 0 0 VLAN Manager
153 0 0 0 7044 0 0 DHCPD Timer
154 0 10012 252 19144 0 0 EEM Policy Direc
155 0 444298256 444299172 14828 4569177 4569177 Syslog
156 0 0 0 7044 0 0 VPDN Scal
157 0 0 0 7044 11160 11160 Net Input
158 0 252 252 7044 0 0 Compute load avg
159 0 0 0 7044 0 0 Per-minute Jobs
160 0 164 0 7208 0 0 CEF Scanner
161 0 252 0 25296 0 0 tHUB
162 0 504 252 13296 0 0 tENM
163 0 74676 3124 70888 1440147 1440147 IP NAT Ager
164 0 0 0 7044 0 0 IP NAT WLAN
165 0 792 252 7584 108 108 NTP
166 0 172 0 7216 0 0 DHCPD Database
81560476 Total
ā09-23-2011 12:56 AM
Hi,
You could upgrade the memory of the router to support a bigger nat table if it can or you could try to change the nat timeout for tcp sessions from the default of 24 hrs to a lesser value with the ip nat timeout command.
Regards.
Alain.
ā09-23-2011 01:24 AM
ā09-23-2011 03:36 AM
hi,
did it solve anything ? You'll have to clear your dynamic nat entries first.
Regards.
Alain.
ā09-23-2011 08:03 PM
The router hang still continues and have to reboot the router to bring it up
And yes after applying the NAT timeouts I did reset the NAT table
ā09-24-2011 01:10 AM
Hi,
Maybe you could try this:
http://www.cisco.com/en/US/docs/ios/12_3t/12_3t4/feature/guide/gt_natrl.pdf
Regards.
Alain.
ā09-24-2011 09:42 AM
Hi Sudhir,
Are you sure it goes upto 2E6 translations? Following is an excerpt from NAT FAQ doc which says each 10000 translation consumes 3Meg of DRAM.
Q. How many concurrent NAT sessions are supported in Cisco IOS NAT?
A. The NAT session limit is bounded by the amount of available DRAM in the router. Each NAT translation consumes about 312 bytes in DRAM. As a result, 10,000 translations (more than would generally be handled on a single router) consume about 3 MB. Therefore, typical routing hardware has more than enough memory to support thousands of NAT translations.
This means 2Million translations would mean 600 Meg of DRAM at least.
Are you sure all the 2Million translations are for valid traffic. Please make sure of that.
This is the NAT doc that I mentioned -
http://www.cisco.com/en/US/tech/tk648/tk361/technologies_q_and_a_item09186a00800e523b.shtml
Can you collect the "show ip nat stat" and "show proc mem sorted" the next time before you reload the box.
Thanks.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide