Router getting hanged after 8 to 12 hours, after restarting it's working, at that time when we are accessing router it's not giving us console , it's only showing low memory
we found these logs on the router
and when we are running any command it's showing only one commad output
i have changed the ip nat translation timout 60 second
i have been also change ip nat translation tcp-timeout 5
i have change ip virtual-reasassambly in and out both
but still the same after 8 to 12 hours it's getting hanged and no internet
process memory showing low
DTDU-INTERNET-ROUTER#show logging history
Syslog History Table:1 maximum table entries,
saving level warnings or higher
20 messages ignored, 0 dropped, 0 recursion drops
659 table entries flushed
SNMP notifications enabled, 660 notifications sent
entry number 660 : SYS-2-CHUNKEXPANDFAIL
Could not expand chunk pool for ipnat node. No memory available
timestamp: 6215216
DTDU-INTERNET-ROUTER#
DTDU-INTERNET-ROUTER#show startup-config
Using 6305 out of 196600 bytes
!
! Last configuration change at 15:33:56 UTC Wed Dec 2 2015
version 15.1
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname DTDU-INTERNET-ROUTER
!
boot-start-marker
boot-end-marker
!
!
logging buffered 4096
no logging console
enable secret 5 $1$L2Gx.x.xBPDmof3RBxYim3.A0w40
enable password 7 x.x.x
!
no aaa new-model
ip source-route
!
!
!
no ip dhcp use vrf connected
!
!
ip cef
ip name-server x/x/x/x/
ip name-server x/x/x/x
multilink bundle-name authenticated
!
!
license udi pid CISCO1841 sn FCZ1108754J
!
!
ip tcp synwait-time 10
!
class-map match-any DROP
match protocol fasttrack
match protocol kazaa2
match protocol netshow
match protocol gnutella
match protocol edonkey
!
!
policy-map DROP
class DROP
drop
!
--More-- !
!
!
!
interface FastEthernet0/0
description $FW_OUTSIDE$
bandwidth 1024
ip address 83.x.x.x 255.255.255.252
ip nbar protocol-discovery
duplex auto
speed auto
!
interface FastEthernet0/1
description $FW_INSIDE$
ip address 213.x.x.x 255.255.255.240
ip nat inside
ip virtual-reassembly in max-fragments 64 max-reassemblies 64 timeout 5
duplex auto
speed auto
!
interface ATM0/0/0
no ip address
no atm ilmi-keepalive
hold-queue 224 in
pvc 0/50
encapsulation aal5mux ppp dialer
dialer pool-member 1
!
!
interface FastEthernet0/1/0
description $ETH-WAN$
no ip address
ip virtual-reassembly in max-fragments 64 max-reassemblies 64 timeout 5
load-interval 30
duplex auto
speed auto
pppoe enable group global
pppoe-client dial-pool-number 2
pppoe-client dial-pool-number 1
!
interface FastEthernet0/1/1
no ip address
shutdown
duplex auto
speed auto
!
interface Dialer0
bandwidth 2000
ip address negotiated
ip access-group 105 in
--More-- ip nat outside
ip virtual-reassembly in max-fragments 64 max-reassemblies 64 timeout 5
encapsulation ppp
dialer pool 1
dialer-group 1
ppp pap sent-username itadmin password 7 045F1E021A731C1E59
!
interface Dialer1
no ip address
ip access-group 105 in
ip mtu 1452
ip nat outside
ip virtual-reassembly in max-fragments 64 max-reassemblies 64 timeout 5
encapsulation ppp
shutdown
dialer pool 2
dialer-group 2
ppp authentication chap pap callin
ppp chap hostname itadmin
ppp chap password 7 045F1E021A731C1E59
ppp pap sent-username itadmin password 7 045F1E021A731C1E59
ppp ipcp dns request
!
ip forward-protocol nd
!
!
ip http server
ip nat translation timeout 60
ip nat inside source list 110 interface Dialer0 overload
ip route 0.0.0.0 0.0.0.0 Dialer0
ip route 91.x.x.x 255.255.255.240 83.x.x.x.
ip route 121.x.x.x 255.255.255.0 83.111.x.x.x.
ip route 202.x.x.x 255.255.255.248 83.x.x.x
ip route 210.x.x.x 255.255.255.0 83.x.x.x.
ip route 210.x.x.x 255.255.255.255 Dialer0 name webmail.xxx.com
ip route 210.x.x.x 255.255.255.255 Dialer0 name mail.xxx.com
!
dialer-list 1 protocol ip permit
route-map ADSL permit 5
match ip address 5
set default interface FastEthernet0/0
!
!
!
!
!
line con 0
line aux 0
line vty 0 4
password 7 x.x.x
login
transport input all
!
scheduler allocate 20000 1000
end
DTDU-INTERNET-ROUTER#
DTDU-INTERNET-ROUTER#show version
Cisco IOS Software, 1841 Software (C1841-BROADBAND-M), Version 15.1(4)M5, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2012 by Cisco Systems, Inc.
Compiled Tue 04-Sep-12 15:15 by prod_rel_team
ROM: System Bootstrap, Version 12.4(13r)T, RELEASE SOFTWARE (fc1)
DTDU-INTERNET-ROUTER uptime is 17 hours, 24 minutes
System returned to ROM by power-on
System image file is "flash:c1841-broadband-mz.151-4.M5.bin"
Last reload type: Normal Reload
Cisco 1841 (revision 6.0) with 118784K/12288K bytes of memory.
Processor board ID FCZ1108754J
4 FastEthernet interfaces
1 ATM interface
DRAM configuration is 64 bits wide with parity disabled.
191K bytes of NVRAM.
63808K bytes of ATA CompactFlash (Read/Write)
License Info:
License UDI:
-------------------------------------------------
Device# PID SN
-------------------------------------------------
*0 CISCO1841 FCZ1108754J
Configuration register is 0x2102
DTDU-INTERNET-ROUTER#sho
DTDU-INTERNET-ROUTER#show processes cpu history table
DTDU-INTERNET-ROUTER 07:54:20 AM Thursday Dec 3 2015 UTC
222222222222222222222222222222222222222222222222333332222222
222222229999944444000000000066666888880000011111888884444433
100
90
80
70
60
50
40 *****
30 ***** ********** *****
20 ************************************************************
10 ************************************************************
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per second (last 60 seconds)
345549354634923254344776266523677778266676744333212255457733
831128347972292901900446175736544868196774106429645743426530
100 *
90 * * *
80 # * *** * **
70 # * # *** ** ****** **** * **
60 # * # *** *** *#*##* *#*#** **
50 ** # *** # * *#* *** *##### ##*#** * ** *#*
40 *****# *****# ******#* *#* **##### #####*** * ****##
30 *#####*###*##***#*##**#* ### *###### ######*#**** **######**
20 ################################################* *#########
10 ############################################################
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
89594454449456988
82694226755815815
100 * * *
90 ** * * * *
80 ** * * ***
--More-- 70 ** * * ****
60 **** * ****
--More-- 50 **** ***********
40 *****************
--More-- 30 #****************
20 ##************#**
--More-- 10 #################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
--More-- 0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
--More-- * = maximum CPU% # = average CPU%
--More--
DTDU-INTERNET-ROUTER#
DTDU-INTERNET-ROUTER#show processes memory
Processor Pool Total: 48682816 Used: 48594740 Free: 88076
I/O Pool Total: 12582912 Used: 5239296 Free: 7343616
PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 71851180 38177868 30912900 640 19948 *Init*
0 0 12352 79259800 12352 1 1 *Sched*
0 0 49922504 17620676 548708 2411 2410 *Dead*
0 0 0 0 394708 0 0 *MallocLite*
1 0 11996724 0 12003956 2692 2692 Chunk Manager
2 0 240 240 4232 0 0 Load Meter
3 0 31060 664 56196 0 0 LICENSE AGENT
4 0 0 0 7232 0 0 RO Notify Timers
5 0 3380 240 10372 0 0 Check heaps
6 0 70911452 70568940 518064 49 0 Pool Manager
7 0 0 0 7232 0 0 DiscardQ Backgro
8 0 240 240 7232 0 0 Timers
9 0 0 0 4232 0 0 WATCH_AFS
10 0 0 0 7232 0 0 License Client N
11 0 0 0 13232 0 0 OIR Handler
12 0 0 0 7232 0 0 IPC Event Notifi
13 0 0 0 7232 0 0 IPC Dynamic Cach
14 0 0 0 7232 0 0 IPC Session Serv
15 0 0 0 7232 0 0 IPC Zone Manager
16 0 0 0 7232 0 0 IPC Periodic Tim
17 0 0 0 7232 0 0 IPC Deferred Por
18 0 0 0 7232 0 0 IPC Process leve
19 0 1704 0 8936 0 0 IPC Seat Manager
20 0 0 0 7232 0 0 IPC Check Queue
21 0 0 0 7232 0 0 IPC Seat RX Cont
22 0 0 0 7232 0 0 IPC Seat TX Cont
23 0 0 0 7232 0 0 IPC Keep Alive M
24 0 0 0 7232 0 0 IPC Loadometer
25 0 240 240 7232 0 0 Environmental mo
26 0 0 0 25232 0 0 Crash writer
27 0 352 0 7584 0 0 Exception contro
28 0 22376 1600 46280 0 0 VTEMPLATE Backgr
29 0 17420 660 23992 15 15 ARP Input
30 0 885916 885916 7232 12 12 ARP Background
31 0 240 240 7232 0 0 ATM Idle Timer
32 0 0 0 7232 0 0 ATM ASYNC PROC
33 0 0 0 7232 0 0 CEF MIB API
34 0 0 0 7232 0 0 AAA_SERVER_DEADT
35 0 0 0 13232 0 0 Policy Manager
36 0 66004 240 72996 0 0 DDR Timers
37 0 20148 0 21248 32 32 Entity MIB API
38 0 240 240 7232 0 0 PrstVbl
39 0 240 240 7232 0 0 Serial Backgroun
40 0 0 0 4232 0 0 RMI RM Notify Wa
41 0 240 240 7232 0 0 SMART
42 0 240 240 7232 0 0 GraphIt
--More-- PID TTY Allocated Freed Holding Getbufs Retbufs Process
43 0 240 240 13232 0 0 Dialer event
44 0 0 0 7232 0 0 SERIAL A'detect
45 0 240 240 13232 0 0 XML Proxy Client
46 0 0 0 7232 0 0 Critical Bkgnd
47 0 87552 51764 15832 18873 18873 Net Background
48 0 80776 80776 13232 0 0 IDB Work
49 0 240 240 13232 2984 2984 Logger
50 0 240 612 7232 0 0 TTY Background
51 0 132376 0 139608 0 0 IF-MGR control p
52 0 808 0 7232 4 4 IF-MGR event pro
53 0 0 0 4232 0 0 Inode Table Dest
54 0 0 0 7232 0 0 AggMgr Process
55 0 0 0 7232 0 0 Transport Port A
56 0 0 0 7232 0 0 HC Counter Timer
57 0 240 240 7232 0 0 ESWPPM
58 0 240 240 7232 0 0 Eswilp Storm Con
59 0 0 0 4232 0 0 dev_device_inser
60 0 0 0 4232 0 0 dev_device_remov
61 0 240 240 7232 0 0 SM Monitor
62 0 300776 3796 310384 0 0 EEM ED ND
63 0 6616 0 16848 0 0 EEM ED Identity
64 0 0 0 7232 0 0 IGMP Snooping Pr
65 0 0 0 7232 0 0 IGMP Snooping Re
66 0 649172 1484 629688 0 0 USB Startup
68 0 0 0 7232 0 0 Platform Compact
69 0 240 240 7232 0 0 Call Management
70 0 0 0 10232 0 0 ATM Periodic
71 0 0 0 10232 0 0 ATM ARP INPUT
72 0 240 240 13232 0 0 ATM OAM Input
73 0 240 240 13232 0 0 ATM OAM TIMER
74 0 0 0 10232 0 0 ATM DSLSAR Perio
75 0 240 0 4472 0 0 Async write proc
76 0 240 0 4472 0 0 Async write proc
77 0 240 0 4472 0 0 Async write proc
78 0 240 0 4472 0 0 Async write proc
79 0 0 0 4232 0 0 DFS flush period
80 0 0 0 7304 0 0 LED Timers
81 0 239085692 239084448 8476 160 160 DSL State Machin
82 0 240 240 7232 0 0 Call Reconfig
83 0 168 168 7232 0 0 HWIC FE Link Mon
84 0 240 240 7232 0 0 PI MATM Aging Pr
85 0 1104 240 8096 0 0 DTP Protocol
86 0 240 240 7232 0 0 Dot1x Mgr Proces
87 0 0 0 7232 0 0 MAB Framework
88 0 0 0 7232 0 0 EAP Framework
89 0 3168 0 28400 0 0 SMI Director DB
90 0 0 0 7232 0 0 SMI CDP Update H
91 0 704 0 25936 0 0 SMI Backup Proce
DTDU-INTERNET-ROUTER#show memory fragment detail
Processor memory
Free memory size : 44684 Number of free blocks: 740
Address Bytes Prev Next Ref PrevF NextF Alloc PC what
64FB9C98 0000000032 64FA9C64 64FB9CE8 001 -------- -------- 61F80734 Init
64FB9CE8 0000000032 64FB9C98 64FB9D38 000 0 64F4D1FC 627A114C Iterator
64FB9D38 0000000036 64FB9CE8 64FB9D8C 001 -------- -------- 623E5CB0 SDB String
64F4D1B0 0000000028 64F4D164 64F4D1FC 001 -------- -------- 603C5EB4 Init
64F4D1FC 0000000028 64F4D1B0 64F4D248 000 64FB9CE8 64F4EF38 627A1184 Iterator Hash Entry
64F4D248 0000000028 64F4D1FC 64F4D294 001 -------- -------- 6262F568 Init
64F4EEC0 0000000072 64F4EE74 64F4EF38 001 -------- -------- 627C6E34 Init
64F4EF38 0000000080 64F4EEC0 64F4EFB8 000 64F4D1FC 654BC6BC 627A1184 Iterator Hash Entry
64F4EFB8 0000000040 64F4EF38 64F4F010 001 -------- -------- 6262F568 Init
654BC670 0000000028 654BC61C 654BC6BC 001 -------- -------- 61D32F28 MDB Lookup List
654BC6BC 0000000036 654BC670 654BC710 000 64F4EF38 65551E2C 627A114C Iterator
654BC710 0000000028 654BC6BC 654BC75C 001 -------- -------- 61D32F28 MDB Lookup List
65551DD4 0000000040 65551D50 65551E2C 001 -------- -------- 6100EADC Parser Mode Q1
65551E2C 0000000060 65551DD4 65551E98 000 654BC6BC 66520694 627A1184 Iterator Hash Entry
65551E98 0000000040 65551E2C 65551EF0 001 -------- -------- 6212BC9C Init
Any Solution to solve this issue ???????????
