08-22-2011 02:31 AM - edited 03-07-2019 01:48 AM
Hi,
Anybody could identify what is the problem with the router 2851???????
x400B01ECz 0x41BC96F8z 0x41BC96DCz
*Aug 10 11:17:53.533: %SYS-2-MALLOCFAIL: Memory allocation of 20000 bytes failed from 0x400B57E8, alignment 0
Pool: Processor Free: 711968 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP Input", ipl= 0, pid= 84, -Traceback= 0x4007D850z 0x40094494z 0x400B3530z 0x40B9E1D8z 0x40B9E6C8z 0x40BA5DA4z 0x40BA6A48z 0x40BA2FA8z 0x412C5C3Cz 0x412C5CB8z 0x412C7FB8z 0x412C807Cz 0x412C852Cz 0x412C85A4z 0x412D630Cz 0x40F95464z
*Aug 10 11:17:56.773: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat node. No memory available -Process= "Chunk Manager", ipl= 4, pid= 1, -Traceback= 0x400B01ECz 0x41BC96F8z 0x41BC96DCz
*Aug 10 11:18:07.321: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat node. No memory available -Process= "Chunk Manager", ipl= 4, pid= 1, -Traceback= 0x400B01ECz 0x41BC96F8z 0x41BC96DCz
*Aug 10 11:18:17.449: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat node. No memory available -Process= "Chunk Manager", ipl= 4, pid= 1, -Traceback= 0x400B01ECz 0x41BC96F8z 0x41BC96DCz
*Aug 10 11:18:23.533: %SYS-2-MALLOCFAIL: Memory allocation of 20000 bytes failed from 0x400B57E8, alignment 0
Pool: Processor Free: 711968 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP Input", ipl= 0, pid= 84, -Traceback= 0x4007D850z 0x40094494z 0x400B3530z 0x40B9E1D8z 0x40B9E6C8z 0x40BA5DA4z 0x40BA6A48z 0x40BA2FA8z 0x412C5C3Cz 0x412C5CB8z 0x412C7FB8z 0x412C807Cz 0x412C852Cz 0x412C85A4z 0x412D630Cz 0x40F95464z
*Aug 10 11:18:28.461: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat node. No memory available -Process= "Chunk Manager", ipl= 4, pid= 1, -Traceback= 0x400B01ECz 0x41BC96F8z 0x41BC96DCz
*Aug 10 11:18:38.477: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat node. No memory available -Process= "Chunk Manager", ipl= 4, pid= 1, -Traceback= 0x400B01ECz 0x41BC96F8z 0x41BC96DCz
*Aug 10 11:18:48.489: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat node. No memory available -Process= "Chunk Manager", ipl= 4, pid= 1, -Traceback= 0x400B01ECz 0x41BC96F8z 0x41BC96DCz
*Aug 10 11:18:53.545: %SYS-2-MALLOCFAIL: Memory allocation of 20000 bytes failed from 0x400B57E8, alignment 0
Pool: Processor Free: 711968 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP Input", ipl= 0, pid= 84, -Traceback= 0x4007D850z 0x40094494z 0x400B3530z 0x40B9E1D8z 0x40B9E6C8z 0x40BA5DA4z 0x40BA6A48z 0x40BA2FA8z 0x412C5C3Cz 0x412C5CB8z 0x412C7FB8z 0x412C807Cz 0x412C852Cz 0x412C85A4z 0x412D630Cz 0x40F95464z
*Aug 10 11:18:58.529: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat node. No memory available -Process= "Chunk Manager", ipl= 4, pid= 1, -Traceback= 0x400B01ECz 0x41BC96F8z 0x41BC96DCz
*Aug 10 11:19:08.581: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat node. No memory available -Process= "Chunk Manager", ipl= 4, pid= 1, -Traceback= 0x400B01ECz 0x41BC96F8z 0x41BC96DCz
*Aug 10 11:19:18.601: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat node. No memory available -Process= "Chunk Manager", ipl= 4, pid= 1, -Traceback= 0x400B01ECz 0x41BC96F8z 0x41BC96DCz
*Aug 10 11:19:23.549: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x400B37F8, alignment 8
Pool: Processor Free: 711968 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP Input", ipl= 0, pid= 84, -Traceback= 0x4007D850z 0x40097A34z 0x400B1540z 0x400B030Cz 0x400B3D9Cz 0x400AFF88z 0x4143C338z 0x4143CF44z 0x4143E6ACz 0x41443C90z 0x4143B740z 0x4140A778z 0x412B89E4z 0x412B8C1Cz 0x412A2818z 0x412A3294z
*Aug 10 11:19:29.489: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat node. No memory available -Process= "Chunk Manager", ipl= 4, pid= 1, -Traceback= 0x400B01ECz 0x41BC96F8z 0x41BC96DCz
08-22-2011 04:58 AM
Hello Manesh,
Processes in your router are trying to allocate memory but your available free memory is so much fragmented (small free blocks scattered across the entire RAM space) that the processes are unable to find a single free contiguous block of the necessary size. This may happen after a prolonged router uptime, as Cisco IOS does not have any memory compaction or defrag routines. It may also signify a bug in your IOS, or too less RAM for your current needs.
What I also see here is that it is the IP Input process that tries to allocated memory. That would mean that some traffic needs to be process-based instead of going through faster routing/switching paths like CEF, which is not common for a plainly routed IP traffic. Is it possible to post your configuration and the version of IOS you are running currently?
Also please be sure to read the following Cisco document - it describes various best practices regarding troubleshooting memory allocation issues:
http://www.cisco.com/en/US/products/sw/iosswrel/ps1831/products_tech_note09186a00800a6f3a.shtml
Best regards,
Peter
08-22-2011 05:38 AM
Please post the output of show version.
Regards,
jerry
08-22-2011 03:30 PM
Tracebacks ... Yum, yum. It normally means that you've hit a software bug. As what Jerry has stated, please post the output to the command "sh version" or upgrade your IOS.
08-23-2011 07:33 AM
Hi Leolahoo..
Cisco IOS Software, 2800 Software (C2800NM-IPBASE-M), Version 12.4(24)T4, RELEAS E SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Fri 03-Sep-10 05:39 by prod_rel_team
ROM: System Bootstrap, Version 12.4(13r)T11, RELEASE SOFTWARE (fc1)
Router uptime is 5 days, 4 hours, 26 minutes
System returned to ROM by power-on
System image file is "flash:c2800nm-ipbase-mz.124-24.T4.bin"
Cisco 2851 (revision 53.50) with 514048K/10240K bytes of memory.
Processor board ID FGL1512119T
2 Gigabit Ethernet interfaces
DRAM configuration is 64 bits wide with parity enabled.
191K bytes of non-volatile configuration memory.
126976K bytes of ATA CompactFlash (Read/Write)
Configuration register is 0x2102
08-23-2011 08:35 AM
Your router ran out of memory. Now this is getting interesting, I want to see the output of the following command:
show mem sum (just the first page)
show proc mem sort
Regards,
jerry
08-23-2011 03:22 PM
Jerry and Leo,
If you look closely, the problem seems to be related to the NAT:
*Aug 10 11:18:58.529: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat node.
Perhaps there is an excessive count of NAT entries, consuming the available memory. It would also be interesting to see the output of the show ip nat statistics command.
Leo: a traceback does not necessarily mean that the IOS is buggy right away. It merely states that a process encountered an unhandled exception but it may be caused by external factors.
Best regards,
Peter
08-23-2011 04:02 PM
Hi Peter,
You might be right but I am asking my question based on the traceback decode.
Regards,
jerry
08-23-2011 04:06 PM
Hi Jerry,
Certainly. I have just pointed out a particular observation but please do not let me detract you.
Best regards,
Peter
08-23-2011 04:11 PM
Hi Peter,
Not a problem. Let me share a little info here. I am seeing arp, ip arp and mac address in the decode. I am not so sure what is causing the memory to run out. If the poster show us the process memory utilization, we might have more clue.
Regards,
jerry
08-23-2011 04:06 PM
Nuts. Missed that one. Thanks for that Peter.
08-27-2012 12:25 AM
Dear Jerry, Leo and Peter.
Kindly i do have the same problem and i just restarted the router its the internet router and everything was down so i had to restart it for everyone to work this problem happened 2 times now every 4 or 5 days sorta.
and i do have some nated ips on that router too.
so please what you need me to post for you and i will do it to solve this prob
08-27-2012 12:45 AM
Hello Mohamed,
Can you post your current configuration? We may find some configuration parts that result in excessive memory consumption.
Have you tried upgrading your IOS? Assuming that this problem may be the result of some software bug, judiciously upgrading might help. What exact IOS are you running now and what router type do you use?
Best regards,
Peter
08-27-2012 01:04 AM
Hi peter,
here is the show version of the router, and this router was working for almost 2 years or so with no probs why now ? you know.
Cisco IOS Software, 2800 Software (C2800NM-ADVSECURITYK9-M), Version 12.4(20)T2, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Sat 31-Jan-09 13:46 by prod_rel_team
ROM: System Bootstrap, Version 12.4(13r)T11, RELEASE SOFTWARE (fc1)
Living-in-internet-R uptime is 51 minutes
System returned to ROM by reload at 06:58:57 UTC Mon Aug 27 2012
System image file is "flash:c2800nm-advsecurityk9-mz.124-20.T2.bin"
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
If you require further assistance please contact us by sending email to
Cisco 2811 (revision 53.51) with 247808K/14336K bytes of memory.
Processor board ID FCZ132070EX
2 FastEthernet interfaces
2 Serial(sync/async) interfaces
1 ATM interface
1 Virtual Private Network (VPN) Module
DRAM configuration is 64 bits wide with parity enabled.
191K bytes of non-volatile configuration memory.
62720K bytes of ATA CompactFlash (Read/Write)
Configuration register is 0x2102
i will post the configuration but i will remove the real ips with X.X.X.X sorry.
Living-in-internet-R#
Living-in-internet-R#
Living-in-internet-R#
Living-in-internet-R#
Living-in-internet-R#sh run
Building configuration...
Current configuration : 9682 bytes
!
version 12.4
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname Living-in-internet-R
!
boot-start-marker
boot-end-marker
!
logging message-counter syslog
logging buffered 51201 warnings
!
aaa new-model
!
!
aaa authentication login ssl local
!
!
aaa session-id common
!
dot11 syslog
ip source-route
!
!
ip cef
!
!
ip name-server 4.2.2.2
ip name-server 8.8.8.8
ip multicast-routing
!
multilink bundle-name authenticated
!
!
!
crypto pki trustpoint TP-self-signed-1655078679
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-1655078679
revocation-check none
rsakeypair TP-self-signed-1655078679
!
!
crypto pki certificate chain TP-self-signed-1655078679
certificate self-signed 01
3082024C 308201B5 A0030201 02020101 300D0609 2A864886 F70D0101 04050030
31312F30 2D060355 04031326 494F532D 53656C66 2D536967 6E65642D 43657274
69666963 6174652D 31363535 30373836 3739301E 170D3132 30383237 30373031
33315A17 0D323030 31303130 30303030 305A3031 312F302D 06035504 03132649
4F532D53 656C662D 5369676E 65642D43 65727469 66696361 74652D31 36353530
37383637 3930819F 300D0609 2A864886 F70D0101 01050003 818D0030 81890281
8100A633 3E543710 5B56616F 0CD10226 72AFFF57 264DE0BF 45129BED E490AC26
39C6B08E 23B7F409 F39DE34C F8F3872B 90BD1F3A 9D6DD291 BC4F9ED6 55854BF8
B2B301E2 F8FF3B3D 411F207B 20241AAB 2D13814C D7E03746 8D96BAE8 205E7325
BC394BED 122C3893 E4D92181 8C3FDE7C 1C30C96D 7D32481C 4B8D3CCA 6F5FF241
BC0D0203 010001A3 74307230 0F060355 1D130101 FF040530 030101FF 301F0603
551D1104 18301682 144C6976 696E672D 696E2D69 6E746572 6E65742D 52301F06
03551D23 04183016 8014BB10 E778310D D1B0E049 A2F0E094 23902307 5267301D
0603551D 0E041604 14BB10E7 78310DD1 B0E049A2 F0E09423 90230752 67300D06
092A8648 86F70D01 01040500 03818100 A2573B93 3AF0A175 4CB7ED39 BAD78ED4
2AA446D4 978D7DC6 C7D04E08 CA0C60B3 2AE77C2D 1CF92AC0 04917E6F C4C70D65
F27C6E3D 503A201F 7709F687 8352DA0E 69E0135D 8359A1FA 2F1DC31A F6BE4870
A25CEA26 BF8EC4D4 CAC8D164 0BE9C074 21F9BE6D A21382A4 937D7F28 53513055
28A7F9CD 7459BF81 9E3904FA 7F36680B
quit
!
!
archive
log config
hidekeys
!
!
!
!
!
!
!
!
interface Loopback100
no ip address
!
interface FastEthernet0/0
description <<""Inside"">>
ip address 192.168.201.254 255.255.255.0
ip pim sparse-mode
ip nat inside
ip virtual-reassembly
duplex auto
speed auto
!
interface FastEthernet0/1
description <<""Internet"">>
ip address X.X.X.X 255.255.255.224
ip nat outside
ip virtual-reassembly
duplex auto
speed auto
!
interface ATM0/0/0
no ip address
shutdown
no atm ilmi-keepalive
dsl operating-mode auto
!
interface Serial0/1/0
no ip address
shutdown
clock rate 2000000
!
interface Serial0/1/1
no ip address
shutdown
clock rate 2000000
!
ip local pool pool 20.0.0.1 20.0.0.254
ip forward-protocol nd
ip route 0.0.0.0 0.0.0.0 X.X.X.X
ip route 10.0.0.0 255.0.0.0 192.168.201.1
ip route 192.168.60.0 255.255.255.0 192.168.201.1
ip http server
ip http access-class 23
ip http authentication local
ip http secure-server
ip http timeout-policy idle 60 life 86400 requests 10000
!
!
ip pim rp-address 192.168.201.254
ip nat inside source list internet interface FastEthernet0/1 overload
ip nat inside source static 192.168.201.1 X.X.X.X
ip nat inside source static 10.10.90.100 X.X.X.X
ip nat inside source static 10.10.90.60 X.X.X.X
ip nat inside source static 10.10.110.6 X.X.X.X
ip nat inside source static 10.10.106.105 X.X.X.X
ip nat inside source static 10.10.106.106 X.X.X.X
ip nat inside source static 10.10.106.107 X.X.X.X
ip nat inside source static 10.10.90.80 X.X.X.X
ip nat inside source static 10.10.110.15 X.X.X.X
ip nat inside source static 10.10.110.10 X.X.X.X
!
ip access-list extended IDS_fa0/0_in_0
permit ip host 10.10.105.31 any
permit ip any any
permit tcp any any
permit udp any any
ip access-list extended internet
permit ip 10.10.91.0 0.0.0.255 any
permit tcp 10.10.91.0 0.0.0.255 any
permit udp 10.10.91.0 0.0.0.255 any
permit ip 10.10.120.0 0.0.0.255 any
permit ip 10.10.90.0 0.0.0.255 any
permit tcp 10.10.90.0 0.0.0.255 any
permit udp 10.10.90.0 0.0.0.255 any
permit ip 10.10.110.0 0.0.0.255 any
permit tcp 10.10.110.0 0.0.0.255 any
permit udp 10.10.110.0 0.0.0.255 any
permit ip 10.10.6.0 0.0.0.255 any
deny tcp 10.10.6.0 0.0.0.255 eq smtp any
permit icmp 10.10.20.0 0.0.0.255 any echo
permit icmp 10.10.20.0 0.0.0.255 any echo-reply
permit ip 10.10.20.0 0.0.0.255 host X.X.X.X
permit ip 10.10.20.0 0.0.0.255 host X.X.X.X
permit ip 10.10.20.0 0.0.0.255 host X.X.X.X
deny tcp 10.10.20.0 0.0.0.255 eq smtp any
permit icmp 10.10.40.0 0.0.0.255 any echo
permit icmp 10.10.40.0 0.0.0.255 any echo-reply
deny tcp 10.10.40.0 0.0.0.255 eq smtp any
permit ip 192.168.201.0 0.0.0.255 any
permit tcp 192.168.201.0 0.0.0.255 any
permit udp 192.168.201.0 0.0.0.255 any
permit ip 10.10.109.0 0.0.0.255 any
permit tcp 10.10.109.0 0.0.0.255 any
permit udp 10.10.109.0 0.0.0.255 any
permit ip host 10.10.105.100 any
permit ip 10.10.105.0 0.0.0.255 any
permit icmp 10.10.30.0 0.0.0.255 any echo
permit icmp 10.10.30.0 0.0.0.255 any echo-reply
deny tcp 10.10.30.0 0.0.0.255 eq smtp any
!
logging trap debugging
logging 10.10.105.100
logging 10.10.120.150
access-list 23 permit 10.10.10.0 0.0.0.7
access-list 100 permit ip 10.0.0.0 0.0.0.255 any
access-list 100 permit tcp any any
access-list 100 permit udp any any
access-list 190 permit ip any any
access-list 190 permit tcp any any
access-list 190 permit udp any any
access-list 191 permit ip any any
access-list 191 permit tcp any any
access-list 191 permit udp any any
!
!
!
!
!
control-plane
!
banner exec ^CC
% Password expiration warning.
-----------------------------------------------------------------------
Cisco Router and Security Device Manager (SDM) is installed on this device and
it provides the default username "cisco" for one-time use. If you have already
used the username "cisco" to login to the router and your IOS image supports the
"one-time" user option, then this username has already expired. You will not be
able to login to the router with this username after you exit this session.
It is strongly suggested that you create a new username with a privilege level
of 15 using the following command.
username
Replace
use.
-----------------------------------------------------------------------
^C
banner login ^CC
-----------------------------------------------------------------------
Cisco Router and Security Device Manager (SDM) is installed on this device.
This feature requires the one-time use of the username "cisco"
with the password "cisco". The default username and password have a privilege level of 15.
Please change these publicly known initial credentials using SDM or the IOS CLI.
Here are the Cisco IOS commands.
username
no username cisco
Replace
For more information about SDM please follow the instructions in the QUICK START
GUIDE for your router or go to http://www.cisco.com/go/sdm
-----------------------------------------------------------------------
^C
!
line con 0
line aux 0
line vty 0 4
exec-timeout 0 0
privilege level 15
logging synchronous
transport input all
line vty 5 15
exec-timeout 0 0
privilege level 15
logging synchronous
transport input all
!
scheduler allocate 20000 1000
!
web gateway ssl-gateway
ip interface FastEthernet0/1 port 443
http-redirect port 80
ssl encryption rc4-md5
ssl trustpoint TP-self-signed-1655078679
logging enable
inservice
!
web gateway web-gateway
ip interface FastEthernet0/1 port 1025
ssl encryption rc4-md5
ssl trustpoint TP-self-signed-1655078679
logging enable
inservice
!
web install svc flash:/web/svc_1.pkg sequence 1
!
web context ssl
title "ssl"
ssl encryption rc4-md5
ssl authenticate verify all
!
!
policy group DEFAULT_POLICY
functions svc-required
svc address-pool "pool"
svc default-domain "cisco.com"
svc keep-client-installed
svc split include 10.0.0.0 255.0.0.0
svc dns-server primary 4.2.2.2
default-group-policy DEFAULT_POLICY
aaa authentication list ssl
aaa authentication domain ssl
gateway ssl-gateway domain ssl
inservice
!
!
web context web
title "web"
ssl encryption rc4-md5
ssl authenticate verify all
!
url-list "url"
heading "Vimportant"
url-text "ipv" url-value "10.10.120.253"
!
acl "web"
permit url "http://10.10.120.253"
!
nbns-list "ADMIN-NBNS"
nbns-server 1.1.1.1
!
port-forward "core"
local-port 23 remote-server "10.10.120.254" remote-port 23 description "telnettocore"
!
policy group policy
url-list "url"
acl "web"
port-forward "core"
nbns-list "ADMIN-NBNS"
functions file-access
functions file-browse
functions file-entry
hide-url-bar
default-group-policy policy
aaa authentication list ssl
aaa authentication domain web
gateway ssl-gateway domain web
user-profile location flash:
inservice
!
end
And no i didnt Upgrade the IOS yet.
08-27-2012 02:00 AM
Hi Mohamed,
I've reviewed your configuration and I haven't found anything obvious. Can you also post the error message you were getting? Also, if the error appears again, can you capture and post the output of the show memory command?
Thank you!
Best regards,
Peter
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide