cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
17723
Views
10
Helpful
49
Replies

Router showing some kind of error???

Manesh Rajan
Level 1
Level 1

Hi,

Anybody could identify what is the problem with the router 2851???????

x400B01ECz 0x41BC96F8z 0x41BC96DCz

*Aug 10 11:17:53.533: %SYS-2-MALLOCFAIL: Memory allocation of 20000 bytes failed from 0x400B57E8, alignment 0

Pool: Processor  Free: 711968  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP Input", ipl= 0, pid= 84,  -Traceback= 0x4007D850z 0x40094494z 0x400B3530z 0x40B9E1D8z 0x40B9E6C8z 0x40BA5DA4z 0x40BA6A48z 0x40BA2FA8z 0x412C5C3Cz 0x412C5CB8z 0x412C7FB8z 0x412C807Cz 0x412C852Cz 0x412C85A4z 0x412D630Cz 0x40F95464z

*Aug 10 11:17:56.773: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat node. No memory available -Process= "Chunk Manager", ipl= 4, pid= 1,  -Traceback= 0x400B01ECz 0x41BC96F8z 0x41BC96DCz

*Aug 10 11:18:07.321: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat node. No memory available -Process= "Chunk Manager", ipl= 4, pid= 1,  -Traceback= 0x400B01ECz 0x41BC96F8z 0x41BC96DCz

*Aug 10 11:18:17.449: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat node. No memory available -Process= "Chunk Manager", ipl= 4, pid= 1,  -Traceback= 0x400B01ECz 0x41BC96F8z 0x41BC96DCz

*Aug 10 11:18:23.533: %SYS-2-MALLOCFAIL: Memory allocation of 20000 bytes failed from 0x400B57E8, alignment 0

Pool: Processor  Free: 711968  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP Input", ipl= 0, pid= 84,  -Traceback= 0x4007D850z 0x40094494z 0x400B3530z 0x40B9E1D8z 0x40B9E6C8z 0x40BA5DA4z 0x40BA6A48z 0x40BA2FA8z 0x412C5C3Cz 0x412C5CB8z 0x412C7FB8z 0x412C807Cz 0x412C852Cz 0x412C85A4z 0x412D630Cz 0x40F95464z

*Aug 10 11:18:28.461: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat node. No memory available -Process= "Chunk Manager", ipl= 4, pid= 1,  -Traceback= 0x400B01ECz 0x41BC96F8z 0x41BC96DCz

*Aug 10 11:18:38.477: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat node. No memory available -Process= "Chunk Manager", ipl= 4, pid= 1,  -Traceback= 0x400B01ECz 0x41BC96F8z 0x41BC96DCz

*Aug 10 11:18:48.489: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat node. No memory available -Process= "Chunk Manager", ipl= 4, pid= 1,  -Traceback= 0x400B01ECz 0x41BC96F8z 0x41BC96DCz

*Aug 10 11:18:53.545: %SYS-2-MALLOCFAIL: Memory allocation of 20000 bytes failed from 0x400B57E8, alignment 0

Pool: Processor  Free: 711968  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP Input", ipl= 0, pid= 84,  -Traceback= 0x4007D850z 0x40094494z 0x400B3530z 0x40B9E1D8z 0x40B9E6C8z 0x40BA5DA4z 0x40BA6A48z 0x40BA2FA8z 0x412C5C3Cz 0x412C5CB8z 0x412C7FB8z 0x412C807Cz 0x412C852Cz 0x412C85A4z 0x412D630Cz 0x40F95464z

*Aug 10 11:18:58.529: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat node. No memory available -Process= "Chunk Manager", ipl= 4, pid= 1,  -Traceback= 0x400B01ECz 0x41BC96F8z 0x41BC96DCz

*Aug 10 11:19:08.581: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat node. No memory available -Process= "Chunk Manager", ipl= 4, pid= 1,  -Traceback= 0x400B01ECz 0x41BC96F8z 0x41BC96DCz

*Aug 10 11:19:18.601: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat node. No memory available -Process= "Chunk Manager", ipl= 4, pid= 1,  -Traceback= 0x400B01ECz 0x41BC96F8z 0x41BC96DCz

*Aug 10 11:19:23.549: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x400B37F8, alignment 8

Pool: Processor  Free: 711968  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "IP Input", ipl= 0, pid= 84,  -Traceback= 0x4007D850z 0x40097A34z 0x400B1540z 0x400B030Cz 0x400B3D9Cz 0x400AFF88z 0x4143C338z 0x4143CF44z 0x4143E6ACz 0x41443C90z 0x4143B740z 0x4140A778z 0x412B89E4z 0x412B8C1Cz 0x412A2818z 0x412A3294z

*Aug 10 11:19:29.489: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat node. No memory available -Process= "Chunk Manager", ipl= 4, pid= 1,  -Traceback= 0x400B01ECz 0x41BC96F8z 0x41BC96DCz

49 Replies 49

Peter Paluch
Cisco Employee
Cisco Employee

Hello Manesh,

Processes in your router are trying to allocate memory but your available free memory is so much fragmented (small free blocks scattered across the entire RAM space) that the processes are unable to find a single free contiguous block of the necessary size. This may happen after a prolonged router uptime, as Cisco IOS does not have any memory compaction or defrag routines. It may also signify a bug in your IOS, or too less RAM for your current needs.

What I also see here is that it is the IP Input process that tries to allocated memory. That would mean that some traffic needs to be process-based instead of going through faster routing/switching paths like CEF, which is not common for a plainly routed IP traffic. Is it possible to post your configuration and the version of IOS you are running currently?

Also please be sure to read the following Cisco document - it describes various best practices regarding troubleshooting memory allocation issues:

http://www.cisco.com/en/US/products/sw/iosswrel/ps1831/products_tech_note09186a00800a6f3a.shtml

Best regards,

Peter

Jerry Ye
Cisco Employee
Cisco Employee

Please post the output of show version.

Regards,

jerry

Leo Laohoo
Hall of Fame
Hall of Fame

Tracebacks ... Yum, yum.  It normally means that you've hit a software bug.  As what Jerry has stated, please post the output to the command "sh version" or upgrade your IOS.

Hi Leolahoo..

Cisco IOS Software, 2800 Software (C2800NM-IPBASE-M), Version 12.4(24)T4, RELEAS                                                                                        E SOFTWARE (fc2)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2010 by Cisco Systems, Inc.

Compiled Fri 03-Sep-10 05:39 by prod_rel_team

ROM: System Bootstrap, Version 12.4(13r)T11, RELEASE SOFTWARE (fc1)

Router uptime is 5 days, 4 hours, 26 minutes

System returned to ROM by power-on

System image file is "flash:c2800nm-ipbase-mz.124-24.T4.bin"

Cisco 2851 (revision 53.50) with 514048K/10240K bytes of memory.

Processor board ID FGL1512119T

2 Gigabit Ethernet interfaces

DRAM configuration is 64 bits wide with parity enabled.

191K bytes of non-volatile configuration memory.

126976K bytes of ATA CompactFlash (Read/Write)

Configuration register is 0x2102

Your router ran out of memory. Now this is getting interesting, I want to see the output of the following command:

show mem sum (just the first page)

show proc mem sort

Regards,

jerry

Jerry and Leo,

If you look closely, the problem seems to be related to the NAT:

*Aug 10 11:18:58.529: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat node.

Perhaps there is an excessive count of NAT entries, consuming the available memory. It would also be interesting to see the output of the show ip nat statistics command.

Leo: a traceback does not necessarily mean that the IOS is buggy right away. It merely states that a process encountered an unhandled exception but it may be caused by external factors.

Best regards,

Peter

Hi Peter,

You might be right but I am asking my question based on the traceback decode.

Regards,

jerry

Hi Jerry,

Certainly. I have just pointed out a particular observation but please do not let me detract you.

Best regards,

Peter

Hi Peter,

Not a problem. Let me share a little info here. I am seeing arp, ip arp and mac address in the decode. I am not so sure what is causing the memory to run out. If the poster show us the process memory utilization, we might have more clue.

Regards,

jerry

Nuts.  Missed that one.  Thanks for that Peter. 

mohamednselim
Level 1
Level 1

Dear Jerry, Leo and Peter.

Kindly i do have the same problem and i just restarted the router its the internet router and everything was down so i had to restart it for everyone to work this problem happened 2 times now every 4 or 5 days sorta.

and i do have some nated ips on that router too.

so please what you need me to post for you and i will do it to solve this prob

Hello Mohamed,

Can you post your current configuration? We may find some configuration parts that result in excessive memory consumption.

Have you tried upgrading your IOS? Assuming that this problem may be the result of some software bug, judiciously upgrading might help. What exact IOS are you running now and what router type do you use?

Best regards,

Peter

Hi peter,

here is the show version of the router, and this router was working for almost 2 years or so with no probs why now ? you know.

Cisco IOS Software, 2800 Software (C2800NM-ADVSECURITYK9-M), Version 12.4(20)T2, RELEASE SOFTWARE (fc4)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2009 by Cisco Systems, Inc.

Compiled Sat 31-Jan-09 13:46 by prod_rel_team

ROM: System Bootstrap, Version 12.4(13r)T11, RELEASE SOFTWARE (fc1)

Living-in-internet-R uptime is 51 minutes

System returned to ROM by reload at 06:58:57 UTC Mon Aug 27 2012

System image file is "flash:c2800nm-advsecurityk9-mz.124-20.T2.bin"

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to

export@cisco.com.

Cisco 2811 (revision 53.51) with 247808K/14336K bytes of memory.

Processor board ID FCZ132070EX

2 FastEthernet interfaces

2 Serial(sync/async) interfaces

1 ATM interface

1 Virtual Private Network (VPN) Module

DRAM configuration is 64 bits wide with parity enabled.

191K bytes of non-volatile configuration memory.

62720K bytes of ATA CompactFlash (Read/Write)

Configuration register is 0x2102

i will post the configuration but i will remove the real ips with X.X.X.X sorry.

Living-in-internet-R#

Living-in-internet-R#

Living-in-internet-R#

Living-in-internet-R#

Living-in-internet-R#sh run

Building configuration...

Current configuration : 9682 bytes

!

version 12.4

service timestamps debug datetime msec

service timestamps log datetime msec

no service password-encryption

!

hostname Living-in-internet-R

!

boot-start-marker

boot-end-marker

!

logging message-counter syslog

logging buffered 51201 warnings

!

aaa new-model

!

!

aaa authentication login ssl local

!

!

aaa session-id common

!

dot11 syslog

ip source-route

!

!

ip cef

!

!

ip name-server 4.2.2.2

ip name-server 8.8.8.8

ip multicast-routing

!

multilink bundle-name authenticated

!

!

!

crypto pki trustpoint TP-self-signed-1655078679

enrollment selfsigned

subject-name cn=IOS-Self-Signed-Certificate-1655078679

revocation-check none

rsakeypair TP-self-signed-1655078679

!

!

crypto pki certificate chain TP-self-signed-1655078679

certificate self-signed 01

  3082024C 308201B5 A0030201 02020101 300D0609 2A864886 F70D0101 04050030

  31312F30 2D060355 04031326 494F532D 53656C66 2D536967 6E65642D 43657274

  69666963 6174652D 31363535 30373836 3739301E 170D3132 30383237 30373031

  33315A17 0D323030 31303130 30303030 305A3031 312F302D 06035504 03132649

  4F532D53 656C662D 5369676E 65642D43 65727469 66696361 74652D31 36353530

  37383637 3930819F 300D0609 2A864886 F70D0101 01050003 818D0030 81890281

  8100A633 3E543710 5B56616F 0CD10226 72AFFF57 264DE0BF 45129BED E490AC26

  39C6B08E 23B7F409 F39DE34C F8F3872B 90BD1F3A 9D6DD291 BC4F9ED6 55854BF8

  B2B301E2 F8FF3B3D 411F207B 20241AAB 2D13814C D7E03746 8D96BAE8 205E7325

  BC394BED 122C3893 E4D92181 8C3FDE7C 1C30C96D 7D32481C 4B8D3CCA 6F5FF241

  BC0D0203 010001A3 74307230 0F060355 1D130101 FF040530 030101FF 301F0603

  551D1104 18301682 144C6976 696E672D 696E2D69 6E746572 6E65742D 52301F06

  03551D23 04183016 8014BB10 E778310D D1B0E049 A2F0E094 23902307 5267301D

  0603551D 0E041604 14BB10E7 78310DD1 B0E049A2 F0E09423 90230752 67300D06

  092A8648 86F70D01 01040500 03818100 A2573B93 3AF0A175 4CB7ED39 BAD78ED4

  2AA446D4 978D7DC6 C7D04E08 CA0C60B3 2AE77C2D 1CF92AC0 04917E6F C4C70D65

  F27C6E3D 503A201F 7709F687 8352DA0E 69E0135D 8359A1FA 2F1DC31A F6BE4870

  A25CEA26 BF8EC4D4 CAC8D164 0BE9C074 21F9BE6D A21382A4 937D7F28 53513055

  28A7F9CD 7459BF81 9E3904FA 7F36680B

            quit

!

!

archive

log config

  hidekeys

!

!

!

!

!

!

!

!

interface Loopback100

no ip address

!

interface FastEthernet0/0

description <<""Inside"">>

ip address 192.168.201.254 255.255.255.0

ip pim sparse-mode

ip nat inside

ip virtual-reassembly

duplex auto

speed auto

!

interface FastEthernet0/1

description <<""Internet"">>

ip address X.X.X.X 255.255.255.224

ip nat outside

ip virtual-reassembly

duplex auto

speed auto

!

interface ATM0/0/0

no ip address

shutdown

no atm ilmi-keepalive

dsl operating-mode auto

!

interface Serial0/1/0

no ip address

shutdown

clock rate 2000000

!

interface Serial0/1/1

no ip address

shutdown

clock rate 2000000

!

ip local pool pool 20.0.0.1 20.0.0.254

ip forward-protocol nd

ip route 0.0.0.0 0.0.0.0 X.X.X.X

ip route 10.0.0.0 255.0.0.0 192.168.201.1

ip route 192.168.60.0 255.255.255.0 192.168.201.1

ip http server

ip http access-class 23

ip http authentication local

ip http secure-server

ip http timeout-policy idle 60 life 86400 requests 10000

!

!

ip pim rp-address 192.168.201.254

ip nat inside source list internet interface FastEthernet0/1 overload

ip nat inside source static 192.168.201.1 X.X.X.X

ip nat inside source static 10.10.90.100 X.X.X.X

ip nat inside source static 10.10.90.60 X.X.X.X

ip nat inside source static 10.10.110.6 X.X.X.X

ip nat inside source static 10.10.106.105 X.X.X.X

ip nat inside source static 10.10.106.106 X.X.X.X

ip nat inside source static 10.10.106.107 X.X.X.X

ip nat inside source static 10.10.90.80 X.X.X.X

ip nat inside source static 10.10.110.15 X.X.X.X

ip nat inside source static 10.10.110.10 X.X.X.X

!

ip access-list extended IDS_fa0/0_in_0

permit ip host 10.10.105.31 any

permit ip any any

permit tcp any any

permit udp any any

ip access-list extended internet

permit ip 10.10.91.0 0.0.0.255 any

permit tcp 10.10.91.0 0.0.0.255 any

permit udp 10.10.91.0 0.0.0.255 any

permit ip 10.10.120.0 0.0.0.255 any

permit ip 10.10.90.0 0.0.0.255 any

permit tcp 10.10.90.0 0.0.0.255 any

permit udp 10.10.90.0 0.0.0.255 any

permit ip 10.10.110.0 0.0.0.255 any

permit tcp 10.10.110.0 0.0.0.255 any

permit udp 10.10.110.0 0.0.0.255 any

permit ip 10.10.6.0 0.0.0.255 any

deny   tcp 10.10.6.0 0.0.0.255 eq smtp any

permit icmp 10.10.20.0 0.0.0.255 any echo

permit icmp 10.10.20.0 0.0.0.255 any echo-reply

permit ip 10.10.20.0 0.0.0.255 host X.X.X.X

permit ip 10.10.20.0 0.0.0.255 host X.X.X.X

permit ip 10.10.20.0 0.0.0.255 host X.X.X.X

deny   tcp 10.10.20.0 0.0.0.255 eq smtp any

permit icmp 10.10.40.0 0.0.0.255 any echo

permit icmp 10.10.40.0 0.0.0.255 any echo-reply

deny   tcp 10.10.40.0 0.0.0.255 eq smtp any

permit ip 192.168.201.0 0.0.0.255 any

permit tcp 192.168.201.0 0.0.0.255 any

permit udp 192.168.201.0 0.0.0.255 any

permit ip 10.10.109.0 0.0.0.255 any

permit tcp 10.10.109.0 0.0.0.255 any

permit udp 10.10.109.0 0.0.0.255 any

permit ip host 10.10.105.100 any

permit ip 10.10.105.0 0.0.0.255 any

permit icmp 10.10.30.0 0.0.0.255 any echo

permit icmp 10.10.30.0 0.0.0.255 any echo-reply

deny   tcp 10.10.30.0 0.0.0.255 eq smtp any

!

logging trap debugging

logging 10.10.105.100

logging 10.10.120.150

access-list 23 permit 10.10.10.0 0.0.0.7

access-list 100 permit ip 10.0.0.0 0.0.0.255 any

access-list 100 permit tcp any any

access-list 100 permit udp any any

access-list 190 permit ip any any

access-list 190 permit tcp any any

access-list 190 permit udp any any

access-list 191 permit ip any any

access-list 191 permit tcp any any

access-list 191 permit udp any any

!

!

!

!

!

control-plane

!

banner exec ^CC

% Password expiration warning.

-----------------------------------------------------------------------

Cisco Router and Security Device Manager (SDM) is installed on this device and

it provides the default username "cisco" for  one-time use. If you have already

used the username "cisco" to login to the router and your IOS image supports the

"one-time" user option, then this username has already expired. You will not be

able to login to the router with this username after you exit this session.

It is strongly suggested that you create a new username with a privilege level

of 15 using the following command.

username privilege 15 secret 0

Replace and with the username and password you want to

use.

-----------------------------------------------------------------------

^C

banner login ^CC

-----------------------------------------------------------------------

Cisco Router and Security Device Manager (SDM) is installed on this device.

This feature requires the one-time use of the username "cisco"

with the password "cisco". The default username and password have a privilege level of 15.

Please change these publicly known initial credentials using SDM or the IOS CLI.

Here are the Cisco IOS commands.

username   privilege 15 secret 0

no username cisco

Replace and with the username and password you want to use.

For more information about SDM please follow the instructions in the QUICK START

GUIDE for your router or go to http://www.cisco.com/go/sdm

-----------------------------------------------------------------------

^C

!

line con 0

line aux 0

line vty 0 4

exec-timeout 0 0

privilege level 15

logging synchronous

transport input all

line vty 5 15

exec-timeout 0 0

privilege level 15

logging synchronous

transport input all

!

scheduler allocate 20000 1000

!

web gateway ssl-gateway

ip interface FastEthernet0/1 port 443

http-redirect port 80

ssl encryption rc4-md5

ssl trustpoint TP-self-signed-1655078679

logging enable

inservice

!

web gateway web-gateway

ip interface FastEthernet0/1 port 1025

ssl encryption rc4-md5

ssl trustpoint TP-self-signed-1655078679

logging enable

inservice

!

web install svc flash:/web/svc_1.pkg sequence 1

!

web context ssl

title "ssl"

ssl encryption rc4-md5

ssl authenticate verify all

!

!

policy group DEFAULT_POLICY

   functions svc-required

   svc address-pool "pool"

   svc default-domain "cisco.com"

   svc keep-client-installed

   svc split include 10.0.0.0 255.0.0.0

   svc dns-server primary 4.2.2.2

default-group-policy DEFAULT_POLICY

aaa authentication list ssl

aaa authentication domain ssl

gateway ssl-gateway domain ssl

inservice

!

!

web context web

title "web"

ssl encryption rc4-md5

ssl authenticate verify all

!

url-list "url"

   heading "Vimportant"

   url-text "ipv" url-value "10.10.120.253"

!

acl "web"

   permit url "http://10.10.120.253"

!

nbns-list "ADMIN-NBNS"

   nbns-server 1.1.1.1

!

port-forward "core"

   local-port 23 remote-server "10.10.120.254" remote-port 23 description "telnettocore"

!

policy group policy

   url-list "url"

   acl "web"

   port-forward "core"

   nbns-list "ADMIN-NBNS"

   functions file-access

   functions file-browse

   functions file-entry

   hide-url-bar

default-group-policy policy

aaa authentication list ssl

aaa authentication domain web

gateway ssl-gateway domain web

user-profile location flash:

inservice

!

end

And no i didnt Upgrade the IOS yet.

Hi Mohamed,

I've reviewed your configuration and I haven't found anything obvious. Can you also post the error message you were getting? Also, if the error appears again, can you capture and post the output of the show memory command?

Thank you!

Best regards,

Peter

Review Cisco Networking products for a $25 gift card