Buy or Renew
Buy or Renew
Cisco + Splunk: It’s a new day for your data. Learn more.
 
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
2004
Views
0
Helpful
6
Replies

Cisco 3945 E PAT issue

Sudhir Menon
Level 1
Level 1

‎09-23-2011 12:19 AM - edited ‎03-04-2019 01:42 PM

Hi ,

We have a 3845  router deployed at a tier  2 ISP  for  PATing user based sessions to the Internet ,the issue happening is when the NAT table size reaches approximately 2,000,000 the router gets hung and has to be restarted 

the router has a memory of 256 MB RAM and approximately 100 MB free ,below is the show version and show proc mem

Can some one help me identify the issue we are facing

Cisco IOS Software, 3800 Software (C3845-ENTBASE-M), Version 12.4(9)T3, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2007 by Cisco Systems, Inc.
Compiled Sat 24-Mar-07 00:05 by prod_rel_team

ROM: System Bootstrap, Version 12.4(13r)T, RELEASE SOFTWARE (fc1)

uptime is 1 hour, 9 minutes
System returned to ROM by power-on
System restarted at 12:05:53 UTC Fri Sep 16 2011
System image file is "flash:c3845-entbase-mz.124-9.T3.bin"

Cisco 3845 (revision 1.0) with 222208K/39936K bytes of memory.
Processor board ID FHK1118F0HT
2 FastEthernet interfaces
2 Gigabit Ethernet interfaces
4 Serial interfaces
4 Channelized (E1 or T1)/PRI ports
DRAM configuration is 64 bits wide with parity enabled.
479K bytes of NVRAM.
62720K bytes of ATA System CompactFlash (Read/Write)

Configuration register is 0x2102

Processor Pool Total:  169056224 Used:   69902956 Free:   99153268
      I/O Pool Total:   40893952 Used:   11728272 Free:   29165680

PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs Process
   0   0   57736864   25274508   30778204        637     315627 *Init*         
   0   0      12128     263332      12128          0          0 *Sched*        
   0   0    1727472    1297972     642904          8          8 *Dead*         
   1   0   46224632     421764   45820620          0          0 Chunk Manager  
   2   0        252        252       4044          0          0 Load Meter     
   4   0       3364        252      10216          0          0 Check heaps    
   5   0     692420      38360     329932         45          0 Pool Manager   
   6   0        252        252       7044          0          0 Timers         
   7   0          0          0       7044          0          0 IPC Dynamic Cach
   8   0          0          0       7044          0          0 IPC Zone Manager
   9   0          0          0       7044          0          0 IPC Periodic Tim
  10   0          0          0       7044          0          0 IPC Deferred Por
  11   0        744          0       7788          0          0 IPC Seat Manager
  12   0          0          0       7044          0          0 IPC BackPressure
  13   0          0          0      13044          0          0 OIR Handler    
  14   0          0          0      25044          0          0 Crash writer   
  15   0        252        252       7044          0          0 Environmental mo
  16   0       4060     452436       8584     783320     783320 ARP Input      
  17   0        252        252       7044          0          0 ATM Idle Timer 
  18   0        252        252       7044          0          0 AAA high-capacit
  19   0          0          0       7044          0          0 AAA_SERVER_DEADT
  20   0          0          0      13044          0          0 Policy Manager 
  21   0        252        252       7044          0          0 DDR Timers     
  22   0       7896          0      14940         50         50 Entity MIB API 
  23   0     262884          0     272928    1496565    1496533 EEM ED Syslog  
  24   0          0          0       7044          0          0 HC Counter Timer
  25   0        252        252       7044          0          0 Serial Backgroun
  26   0          0          0       7044          0          0 RO Notify Timers
  27   0      10052          0      14096          0          0 RMI RM Notify Wa
  28   0        252        252       7044          0          0 SMART          
  29   0        252        252       7044          0          0 GraphIt        
  30   0        252        252      13044          0          0 Dialer event   
  31   0          0          0       7044          0          0 SERIAL A'detect
  32   0        252        252      13044          0          0 XML Proxy Client
  33 706    3170672    3078200      99848        311        310 Virtual Exec   
  34   0          0          0       4044          0          0 Inode Table Dest
  35   0          0          0       7044          0          0 Critical Bkgnd 
  36   0      50260        800      13800       1726       1726 Net Background 
  37   0       2404       2004      13444          0          0 IDB Work       
  38   0     235168      96008     247960    1829403    1829404 Logger         
  39   0        252        560       7044          0          0 TTY Background 
  40   0          0     144620      10044          0          0 Per-Second Jobs
  41   0          0          0       7044          0          0 AggMgr Process 
  42   0          0          0       4044          0          0 dev_device_inser
  43   0          0          0       4044          0          0 dev_device_remov
  44   0          0          0      13044          0          0 sal_dpc_process
  45   0          0          0       7044          0          0 ARL Table Manage
  46   0        252        252       7044          0          0 ESWPPM         
  47   0        252        252       4044          0          0 Eswilp Storm Con
  48   0        252        252       7044          0          0 ESWILPPM       
  49   0        252        252       4044          0          0 Eswilp Storm Con
  50   0        252        252       7044          0          0 Netclock Backgro
  51   0        252        252       7044          0          0 SM Monitor     
  52   0        252        252       7044          0          0 Bryce I2C CMD Qu
  54   0        252        252       7044          0          0 Ether-Switch RBC
  55   0          0          0       4044          0          0 IGMP Snooping Pr
  56   0          0          0       4044          0          0 IGMP Snooping Re
  57   0        252        252       7044          0          0 Call Management
  59   0        252        252       7044          0          0 Dot1x Mgr Proces
  60   0          0          0       7044          0          0 MAB Framework  
  61   0          0          0       7044          0          0 EAP Framework  
  62   0       2500        252       9292          0          0 DTP Protocol   
  63   0        252        252       7044          0          0 PI MATM Aging Pr
  64   0        252        252       7044          0          0 EtherChnl      
  65   0        252        252       7044          0          0 AAA Dictionary R
  66   0        252        252       7044          0          0 AAA Server     
  67   0          0          0       7044          0          0 AAA ACCT Proc  
  68   0          0          0       7044          0          0 ACCT Periodic Pr
  69   0      12944        252      19736        567        567 CDP Protocol   
  70   0        252        252      13044          0          0 Ethernet LMI   
  71   0    7466872     905724    1592508    1629871    1629868 IP Input       
  72   0          0          0       7044          0          0 ICMP event handl
  73   0        252        252       7044          0          0 TurboACL       
  74   0        252        252       7044          0          0 TurboACL chunk 
  75   0       5704        252      10656         28         28 MOP Protocols  
  76   0        504        504      13044          0          0 PPP Hooks      
  78   0          0          0      13044          0          0 SSS Manager    
  79   0          0          0      13044          0          0 SSS Test Client
  80   0          0          0       7044          0          0 SSS Feature Mana
  81   0          0          0       7044          0          0 SSS Feature Time
  82   0          0          0      13044          0          0 VPDN call manage
  83   0          0          0      13100          0          0 L2X Socket proce
  84   0          0          0      13044          0          0 L2X SSS manager
  85   0        252        252      13044          0          0 L2TP mgmt daemon
  86   0          0          0       7044          0          0 X.25 Encaps Mana
  87   0          0          0       7044          0          0 IP Traceroute  
  88   0        580          0      10624          0          0 IP Background  
  89   0        164          0      10208          0          0 IP RIB Update  
  90   0        252        252      13044          0          0 PPP IP Route   
  91   0        252        252      13044          0          0 PPP IPCP       
  92   0          0          0       7044          0          0 SNMP Timers    
  93   0      73444          0      73444          0          0 CEF process    
  94   0          0       4092      13044         31         31 TCP Timer      
  95   0     186068          0      13044          0          0 TCP Protocols  
  96   0          0          0      25044          0          0 COPS           
  97   0          0          0       7044          0          0 Socket Timers  
  98   0        416        252       7208          0          0 Dot1x Supplicant
  99   0        416        252       7208          0          0 Dot1x Supplicant
100   0        416        252       7208          0          0 Dot1x Supplicant
101   0        500        156       7544          0          0 L2MM           
102   0          0          0       7044          0          0 MRD            
103   0       5052          0      12096          0          0 IGMPSN         
104   0        252        252       7044          0          0 RLM groups Proce
105   0     122064        252     128856          0          0 SCTP Main Proces
106   0          0          0       7044          0          0 IUA Main Process
107   0        252        252       7044          0          0 RUDPV1 Main Proc
108   0          0          0       7044          0          0 bsm_timers     
109   0          0          0       7044          0          0 bsm_xmt_proc   
110   0          0          0      10044          0          0 CES Client SVC R
111   0     120988        252     119692          0          0 DHCPD Receive  
112   0        252        252       7044          0          0 Dialer Forwarder
113   0          0    4697092       7044          0          0 IP Cache Ager  
114   0        252        252      10044          0          0 Adj Manager    
115   0        252        252      13044          0          0 ATM OAM Input  
116   0        252        252      13044          0          0 ATM OAM TIMER  
117   0        940          0      10984          0          0 HTTP CORE      
118   0          0          0       7044          0          0 RARP Input     
119   0          0          0       7044          0          0 PAD InCall     
120   0        252        252      13044          0          0 X.25 Background
121   0        252        252       7044          0          0 PPP Bind       
122   0        252        252       7044          0          0 PPP SSS        
123   0          0          0       7044          0          0 MQC Flow Event B
124   0        252        252       7044          0          0 RBSCP Background
125   0        252        252       7044          0          0 CRM_CALL_UPDATE_
127   0        252        252       7044          0          0 AAA Cached Serve
128   0        252        252       7044          0          0 ENABLE AAA     
129   0          0          0       7044          0          0 EM Background Pr
130   0          0          0       7044          0          0 Key chain liveke
131   0        252        252       7044          0          0 LINE AAA       
132   0        252        252       7044          0          0 LOCAL AAA      
133   0       1032        252       7824          0          0 TPLUS          
134   0        252        252       7044          0          0 Control-plane ho
135   0          0          0       7044          0          0 PM Callback    
136   0        252        252       7044          0          0 AAA SEND STOP EV
137   0          0          0      10044          0          0 EEM ED CLI     
138   0          0          0      10044          0          0 EEM ED Counter 
139   0          0          0      10044          0          0 EEM ED Interface
140   0          0          0      10044          0          0 EEM ED IOSWD   
141   0          0          0      10044          0          0 EEM ED Memory-th
142   0          0          0      10044          0          0 EEM ED None    
143   0          0          0      10044          0          0 EEM ED OIR     
144   0         76          0      10120          0          0 EEM ED Resource
145   0          0          0      10044          0          0 EEM ED SNMP    
146   0          0          0      10044          0          0 EEM ED Timer   
147   0          0          0      10044          0          0 EEM ED Track   
148   0      13408       3684      17428          0          0 EEM Server     
149   0        252        252       7044          0          0 RMON Recycle Pro
150   0        252        252       7044          0          0 RMON Deferred Se
151   0          0          0       7044          0          0 Syslog Traps   
152   0       9240       1564      14564          0          0 VLAN Manager   
153   0          0          0       7044          0          0 DHCPD Timer    
154   0      10012        252      19144          0          0 EEM Policy Direc
155   0  444298256  444299172      14828    4569177    4569177 Syslog         
156   0          0          0       7044          0          0 VPDN Scal      
157   0          0          0       7044      11160      11160 Net Input      
158   0        252        252       7044          0          0 Compute load avg
159   0          0          0       7044          0          0 Per-minute Jobs
160   0        164          0       7208          0          0 CEF Scanner    
161   0        252          0      25296          0          0 tHUB           
162   0        504        252      13296          0          0 tENM           
163   0      74676       3124      70888    1440147    1440147 IP NAT Ager    
164   0          0          0       7044          0          0 IP NAT WLAN    
165   0        792        252       7584        108        108 NTP            
166   0        172          0       7216          0          0 DHCPD Database 
                                 81560476 Total

Labels:
0 Helpful
6 Replies 6

cadet alain
VIP Alumni
VIP Alumni

‎09-23-2011 12:56 AM

Hi,

You could upgrade the memory of the router to support a bigger nat table if it can or you could try to change the nat timeout for tcp sessions from the default of 24 hrs to a lesser value with the ip nat timeout command.

Regards.

Alain.

Don't forget to rate helpful posts.
0 Helpful

Sudhir Menon
Level 1
Level 1
In response to cadet alain

‎09-23-2011 01:24 AM

Hi Alain ,

I have done the same attached is the show run of the router

114237-Config.txt.zip
0 Helpful

cadet alain
VIP Alumni
VIP Alumni
In response to Sudhir Menon

‎09-23-2011 03:36 AM

hi,

did it solve anything ? You'll have to clear your dynamic nat entries first.

Regards.

Alain.

Don't forget to rate helpful posts.
0 Helpful

Sudhir Menon
Level 1
Level 1
In response to cadet alain

‎09-23-2011 08:03 PM

The router hang still continues and have to reboot the router to bring it up

And yes after applying the NAT timeouts I did reset the NAT table 

0 Helpful

cadet alain
VIP Alumni
VIP Alumni
In response to Sudhir Menon

‎09-24-2011 01:10 AM

Hi,

Maybe you could try this:

http://www.cisco.com/en/US/docs/ios/12_3t/12_3t4/feature/guide/gt_natrl.pdf

Regards.

Alain.

Don't forget to rate helpful posts.
0 Helpful

skarthic
Cisco Employee
Cisco Employee

‎09-24-2011 09:42 AM

Hi Sudhir,

Are you sure it goes upto 2E6 translations? Following is an excerpt from NAT FAQ doc which says each 10000 translation consumes 3Meg of DRAM.

Q. How many concurrent NAT sessions are supported in Cisco IOS NAT?

A. The NAT session limit is bounded by the amount of available DRAM in the router. Each NAT translation consumes about 312 bytes in DRAM. As a result, 10,000 translations (more than would generally be handled on a single router) consume about 3 MB. Therefore, typical routing hardware has more than enough memory to support thousands of NAT translations.


This means 2Million translations would mean 600 Meg of DRAM at least.

Are you sure all the 2Million translations are for valid traffic. Please make sure of that.

This is the NAT doc that I mentioned -

http://www.cisco.com/en/US/tech/tk648/tk361/technologies_q_and_a_item09186a00800e523b.shtml

Can you collect the "show ip nat stat" and "show proc mem sorted" the next time before you reload the box.

Thanks.

0 Helpful
Customers Also Viewed These Support Documents
Review Cisco Networking products for a $25 gift card