Data structure error -- non-tail element contains NULL next pointer

asentchernaev · ‎02-17-2011

From 3 days I see the following errors ( huge count, maybe 1 per se
cond, never seen before):

-------------------------------------------------------------------------------------------------

Feb 17 16:46:50 192.168.1.254 803218: Feb 17 16:46:49.270: %UTIL-3-DLL: Data structure error -- non-tail element contains NULL next pointer, -Traceback= 0x440EB244z 0x4133B7A0z 0x4133BF9Cz 0x41338FE8z 0x420ABFFCz 0x420AC0BCz 0x420B9E24z 0x41BA85B8z 0x4209D0DCz 0x4209DDF0z 0x4209E3D4z 0x4209E440z 0x420B683Cz 0x4234922Cz 0x420857D4z 0x4208714Cz

---------------------------------------------------------------------------------------------------

It is 2821 voice router servicing 25 ip phones. It is production voice router.

-------------------------------------------------------------------------------------------------------

VRTR1#show ver
Cisco IOS Software, 2800 Software (C2800NM-ADVIPSERVICESK9-M), Version 12.4(24)T 1, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Fri 19-Jun-09 15:13 by prod_rel_team

ROM: System Bootstrap, Version 12.4(13r)T11, RELEASE SOFTWARE (fc1)

VRTR1 uptime is 9 weeks, 5 days, 21 hours, 22 minutes
System returned to ROM by power-on
System restarted at 19:29:52 UTC/GMT Fri Dec 10 2010
System image file is "flash:c2800nm-advipservicesk9-mz.124-24.T1.bin"

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Cisco 2821 (revision 53.51) with 243712K/18432K bytes of memory.
Processor board ID FTX1338AHMB
2 Gigabit Ethernet interfaces
6 ISDN Basic Rate interfaces
1 terminal line
1 Virtual Private Network (VPN) Module
2 Voice FXO interfaces
1 cisco service engine(s)
DRAM configuration is 64 bits wide with parity enabled.
191K bytes of non-volatile configuration memory.
126000K bytes of ATA CompactFlash (Read/Write)

Configuration register is 0x2102

----------------------------------------------------------------------------------------------------------------

Copying the error in But toolKit and pointing to 12.4(24)T1 can't find anything - but the output is only responseless blank output so I'm not sure that I'm using it correctly.

asentchernaev · ‎02-17-2011

I restarted the router and the errors completely stoped - till to this moment. I've made script on our syslog just to receive sms if the error appears even once. Meanwhile I checked the log files from the beginning of router's installation (around a year ago) so I found this error still twice - but with no such frequency and count - once in November and twice in one day in January. There are few main points.

1. The count:this way it is different - frequency - 1 per second and the total count from 14-th to17-th (till the restart in yestarday) - 8053. This count has been never seen before.

2.The first error appears on 14 February on 20:31 in non-working time.

3.I see that there is on additional message "DATACORRUPTION-1-DATAINCONSISTENCY" just before the error appears for the first time (1 second before the error). I cheked and it doesn't exist before. Actually this error is no ever seen in all the log files and it is completely new.

====================================================================

Feb 14 20:14:31 192.168.1.254 787010: .Feb 14 20:14:30.929: %ISDN-6-CONNECT: Interface BRI0/1/1:1 is now connected to N/A N/A
Feb 14 20:31:45 192.168.1.254 787011: Feb 14 20:31:45.290: %DATACORRUPTION-1-DATAINCONSISTENCY: Doubly-linked list corruption, -PC= 0x4133B498z, -Traceback= 0x4197656Cz 0x4133B498z 0x4133B7E4z 0x4133BF9Cz 0x41338FE8z 0x420ABFFCz 0x420AC0BCz 0x420B9E24z 0x41BA85B8z 0x43D8BD78z 0x43D8BF08z 0x43D8C3E0z 0x43D8D50Cz 0x4209C428z 0x4209DDD8z 0x4209E3D4z

#error appears for the first time now. It was followed by other 8052 errors.
Feb 14 20:31:46 192.168.1.254 787012: Feb 14 20:31:45.294: %UTIL-3-DLL: Data structure error -- non-tail element contains NULL next pointer, -Traceback= 0x440EB244z 0x4133B7A0z 0x4133BF9Cz 0x41338FE8z 0x420ABFFCz 0x420AC0BCz 0x420B9E24z 0x41BA85B8z 0x4209D0DCz 0x4209D4CCz 0x43D8CFA4z 0x43D8D0A4z 0x4320374Cz 0x43203730z

....................................................................................................

Feb 14 21:09:15 192.168.1.254 787087: Feb 14 21:09:14.919: %ISDN-6-DISCONNECT: Interface BRI0/1/1:1 disconnected from unknown , call lasted 3283 seconds

======================================================================

4. The both the new error and the following regular error appear during active phone call coming from outside - 17 minues after it's beginning. It is phone call from our manager via the ipsec vpn.

asentchernaev · ‎02-18-2011

5. Additionally I saw another message for 5 times just before the other event appearing for the first time.

========================================================================

Feb 14 18:13:34 192.168.1.254 786928: Feb 14 18:13:33.701: %SEC-6-IPACCESSLOGNP: list 50 permitted 0 192.168.20.40 -> 0.0.0.0, 2 packets
Feb 14 19:48:01 192.168.1.254 786929: .Feb 14 19:47:57.682: %SYS-6-RESERVE_MEMORY_CREATED: Reserve memory is being created
Feb 14 19:48:01 192.168.1.254 786930: .Feb 14 19:48:01.166: %FW-6-SESS_AUDIT_TRAIL_START: Start sip session: initiator (192.168.1.19:58263) -- responder (124.172.158.31:5060)
Feb 14 19:48:02 192.168.1.254 786931: .Feb 14 19:48:01.430: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x400B3CD0, alignment 8
Feb 14 19:48:02 192.168.1.254 786932: Pool: Processor Free: 1790232 Cause: Memory fragmentation
Feb 14 19:48:02 192.168.1.254 786933: Alternate Pool: None Free: 0 Cause: No Alternate pool
Feb 14 19:48:02 192.168.1.254 786934: -Process= "Chunk Manager", ipl= 3, pid= 1, -Traceback= 0x4007D6FCz 0x40097838z 0x400B10D8z 0x400AFBB4z 0x400AFD40z 0x4320374Cz 0x43203730z
Feb 14 19:48:02 192.168.1.254 786935: .Feb 14 19:48:01.434: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat entry. No memory available -Process= "Chunk Manager", ipl= 3, pid= 1, -Traceback= 0x400AFD84z 0x4320374Cz 0x43203730z
Feb 14 19:48:20 192.168.1.254 786936: .Feb 14 19:48:19.990: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat node. No memory available -Process= "Chunk Manager", ipl= 3, pid= 1, -Traceback= 0x400AFD84z 0x4320374Cz 0x43203730z
Feb 14 19:48:31 192.168.1.254 786937: .Feb 14 19:48:30.022: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat node. No memory available -Process= "Chunk Manager", ipl= 3, pid= 1, -Traceback= 0x400AFD84z 0x4320374Cz 0x43203730z
Feb 14 19:48:32 192.168.1.254 786938: .Feb 14 19:48:31.267: %FW-6-SESS_AUDIT_TRAIL: Stop sip session: initiator (192.168.1.19:58263) sent 1 bytes -- responder (124.172.158.31:5060) sent 0 bytes
Feb 14 19:48:33 192.168.1.254 786939: .Feb 14 19:48:32.035: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x400B3CD0, alignment 8
======================================================================

it is happens 5-6 times just before the error for Data corruption.