- Cisco Community
- Technology and Support
- Networking
- Routing
- Re: Cisco 7206VXR running 12.3(8)T suspected memory leak
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-25-2011 01:28 AM - edited 03-04-2019 11:52 AM
Hi Guys,
We have a "core" router in our WAN that has started misbehaving. I first noticed the problem when I couldn't remotely log into the unit, we have a connection through the aux port to another 7206 on the same site and this also wouldn't allow access - although I was able to trap some logging messages that were indicating that router had no available memory:
Mar 9 17:57:39.432: %SYS-2-MALLOCFAIL: Memory allocation of 10000 bytes failed from 0x608B2E20, alignment 0
Pool: Processor Free: 3827300 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "SNMP ENGINE", ipl= 0, pid= 147
-Traceback= 60786144 608A0598 608A3818 608B2E28 6101D310 6101D4F0 60FFAF98 60FFB27C 610223E0 60FF2140 6101A0D8 6086A888 6086A86C
%% Low on memory; try again later
To resolve we had to send a field engineer out - and in the end we had to reboot the router. We closely monitored the unit and memory utilisation appears to be steadily incrementing to the point that we have to do a manual reload every few days. The unit only has 128mb of memory, although we have other similar units running the same protocols and services, and same traffic profile and their memory utilisation sits steadily around 50%. So we actually replaced the NPE engine - which effectively changed the memory. No difference. I disabled netflow on the router and it immediately freed up 10% of the memory, but the issue still continues.
Last night I had to perform another reload, but before doing this I disabled all serial and lan interfaces and monitored memory - it was still incrementing. The dead process seems to be taking a huge chunk....also PDU dispatcher which I know is related to the SNMP engine, appears to be taking more than its fair share - especially when compared to other similar routers in our network. Here's the output of SHOW PROCESS MEMORY SORTED:
Total: 54564032, Used: 42297572, Free: 12266460
PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 43774368 13289940 22984988 0 0 *Init*
0 0 127051652 118895552 17750564 258176 0 *Dead*
6 0 112026940 109590076 4214860 280684 767700 Pool Manager
52 0 174701108 4322292 701796 392508 0 IP Input
137 0 597808 0 658692 0 0 CCPROXY_CT
23 0 279536 179652 251984 0 0 ARP Input
155 0 2432936 2053688 217748 0 0 OSPF Router 140
91 0 217744 0 210860 0 0 PPTP Mgmt
70 0 199664 0 209548 0 0 IP RIB Update
108 0 159872 0 166756 0 0 QOS_MODULE_MAIN
116 0 275528 0 160644 0 0 Proxy Session Ap
4 0 65580 0 90464 0 0 EDDRI_MAIN
110 0 79844 0 89728 0 0 CCVPM_HTSP
103 0 34655144 2938240 79872 0 0 TPLUS
75 0 74904 0 74904 0 0 CEF process
136 0 188 188 60884 0 0 CC-API_VCM
140 0 1696500 48676 38780 13520 0 PIM Process
41 0 27432 188 34164 0 0 VNM DSPRM MAIN
128 0 24892 376 30096 0 0 EM Applet Direct
3 0 57648 161460 28272 17388 0 OSPF Hello 140
98 0 1696 0 26580 0 0 Border Element p
97 0 1884 188 26580 0 0 gk process
54 0 24088 5748 25224 0 0 CDP Protocol
109 0 180 0 25064 0 0 RPMS_PROC_MAIN
146 0 449015212 0 24928 0 0 PDU DISPATCHER
143 0 0 812 24884 0 0 traffic_shape
83 0 0 0 24884 0 0 COPS
107 0 0 0 24884 0 0 VoIP AAA
81 0 30332 188 23260 0 0 DHCPD Receive
31 0 16176 0 23060 0 0 Critical Bkgnd
104 2 124932 115052 20300 0 0 Virtual Exec
147 0 262494064 711485256 18040 0 0 SNMP ENGINE
26 0 10860 15848 17744 0 3640 Entity MIB API
111 0 10472 0 17356 0 0 CCVPM_R2
77 0 731316 0 14328 0 0 TCP Protocols
145 0 31453124 31452040 14028 0 0 IP SNMP
65 0 0 0 12900 0 0 L2X Data Daemon
88 0 188 188 12896 0 0 X.25 Background
29 0 188 188 12884 0 0 Dialer event
92 0 188 188 12884 0 0 PPTP Data
33 0 188 188 12884 0 0 Logger
30 0 188 188 12884 0 0 XML Proxy Client
134 0 0 0 12884 0 0 TCP Driver
14 0 0 0 12884 0 0 Policy Manager
148 0 0 0 12884 0 0 SNMP ConfCopyPro
44 0 188 188 12884 0 0 ATM OAM Input
45 0 188 188 12884 0 0 ATM OAM TIMER
149 0 34448 34448 12884 0 0 SNMP Traps
90 0 0 0 12884 0 0 L2F management d
76 0 0 17715588 12884 0 0 TCP Timer
73 0 188 188 12884 0 0 PPP IPCP
72 0 188 188 12884 0 0 PPP IP Route
32 0 84728 0 12884 17016 0 Net Background
56 0 0 0 12884 0 0 SSS Test Client
64 0 188 188 12884 0 0 L2TP mgmt daemon
63 0 0 0 12884 0 0 L2X SSS manager
55 0 0 0 12884 0 0 SSS Manager
62 0 0 0 12884 0 0 L2X Socket proce
59 0 376 376 12884 0 0 PPP Hooks
61 0 0 0 12884 0 0 VPDN call manage
0 0 12044 157917540 12044 0 0 *Sched*
100 0 6168 188 11928 0 0 LOCAL AAA
79 0 1696 0 11580 0 0 HTTP CORE
122 0 3904 1868 10224 0 0 EM Server
69 0 244 2224 10128 0 0 IP Background
1 0 7528 4440 9972 0 0 Chunk Manager
35 0 2097504 1835316 9884 0 0 Per-Second Jobs
66 0 0 5668 9884 0 1300 EM FD Syslog
67 0 0 0 9884 0 0 EM FD SNMP
68 0 0 0 9884 0 0 EM FD Memory Thr
16 0 0 0 9884 0 0 OIR Handler
74 0 188 188 9884 0 0 Adj Manager
11 0 212 212 9884 0 0 ATM AutoVC Perio
139 0 2936 188 9640 0 0 IGMP Input
113 0 1696 0 8580 0 0 http client proc
124 0 1696 0 8580 0 0 SAA Event Proces
127 0 1196 0 8080 0 0 trunk conditioni
119 0 812 0 7696 0 0 SYSMGT Events
150 0 564 188 7260 0 0 NTP
112 0 328 0 7212 0 0 CCSWVOICE
121 0 188 188 6904 0 0 cerf_daemon_proc
87 0 0 0 6896 0 0 PAD InCall
114 0 0 0 6892 0 0 TACACS+
46 0 188 188 6884 0 0 TurboACL
80 0 0 0 6884 0 0 RARP Input
82 0 0 0 6884 0 0 IP Cache Ager
43 0 940 940 6884 0 0 Flash MIB Update
86 0 188 188 6884 0 0 RBSCP Background
84 0 188 188 6884 0 0 PPP SSS
85 0 188 188 6884 0 0 PPP Bind
89 0 188 188 6884 0 0 Dialer Forwarder
42 0 0 0 6884 0 0 CES Line Conditi
39 0 0 0 6884 0 0 SONET alarm time
47 0 188 188 6884 0 0 AAA Server
78 0 0 0 6884 0 0 Socket Timers
48 0 0 0 6884 0 0 AAA ACCT Proc
95 0 0 0 6884 0 0 bsm_xmt_proc
96 0 188 188 6884 0 0 RLM groups Proce
40 0 0 0 6884 0 0 POS APS Event Pr
50 0 188 188 6884 0 0 AAA Dictionary R
51 0 0 0 6884 0 0 AC Mgr
34 0 157191176 188 6884 0 0 TTY Background
49 0 0 0 6884 0 0 ACCT Periodic Pr
102 0 188 188 6884 0 0 LINE AAA
28 0 188 188 6884 0 0 GraphIt
27 0 0 0 6884 0 0 SERIAL A'detect
105 0 188 188 6884 0 0 VSP_MGR
106 0 0 0 6884 0 0 EM Background Pr
71 0 0 0 6884 0 0 SNMP Timers
24 0 0 0 6884 0 0 HC Counter Timer
22 0 0 0 6884 0 0 IPC BackPressure
21 0 0 0 6884 0 0 IPC Seat Manager
20 0 0 0 6884 0 0 IPC Deferred Por
53 0 0 0 6884 0 0 ICMP event handl
57 0 0 0 6884 0 0 SSS Feature Mana
101 0 188 188 6884 0 0 ENABLE AAA
115 0 0 0 6884 0 0 FB/KS Log HouseK
19 0 0 0 6884 0 0 IPC Periodic Tim
117 0 188 188 6884 0 0 RMON Recycle Pro
118 0 188 188 6884 0 0 RMON Deferred Se
58 0 0 0 6884 0 0 SSS Feature Time
120 0 188 5744844 6884 0 0 AAA SEND STOP EV
38 0 0 0 6884 0 0 CSP Timer
93 0 188 188 6884 0 0 RUDPV1 Main Proc
123 0 0 0 6884 0 0 RMON Packets
94 0 0 0 6884 0 0 bsm_timers
125 0 0 0 6884 0 0 VPDN Scal
126 0 0 0 6884 0 0 trunk conditioni
60 0 0 0 6884 0 0 X.25 Encaps Mana
18 0 0 0 6884 0 0 IPC Zone Manager
131 0 0 0 6884 0 0 Net Input
132 0 188 188 6884 0 0 Compute load avg
133 0 82524 512 6884 0 0 Per-minute Jobs
17 0 0 0 6884 0 0 IPC Dynamic Cach
135 0 0 0 6884 0 0 TCP Listener
99 0 188 188 6884 0 0 CRM_CALL_UPDATE_
15 0 0 0 6884 0 0 EnvMon
138 0 0 0 6884 0 0 CEF Scanner
25 0 188 188 6884 0 0 DDR Timers
13 0 188 188 6884 0 0 AAA high-capacit
141 0 0 174614784 6884 0 0 Mwheel Process
142 0 188 188 6884 0 0 IP Flow Backgrou
12 0 188 188 6884 0 0 ATM VC Auto Crea
8 0 188 188 6884 0 0 Timers
10 0 188 188 6884 0 0 ATM Idle Timer
9 0 188 188 6884 0 0 Serial Backgroun
151 0 0 0 6884 0 0 Time Range Proce
7 0 0 0 6884 0 0 AAA_SERVER_DEADT
152 0 0 0 6884 0 0 xcpa-driver
153 0 0 0 6884 0 0 DHCPD Timer
154 0 0 0 6884 0 0 DHCPD Database
5 0 0 0 6884 0 0 Check heaps
37 0 0 0 3884 0 0 Inode Table Dest
2 0 188 188 3884 0 0 Load Meter
49411560 Total
I used the Cisco Output Interpeter and it highlighted that the *Dead* process might be an issue. Although I can't see any related bugs for the current IOS version 12.3(8)T
I think this is a bug, in fact I would put money on it, but would like some help in trying to identify the cause. TAC cases for this unit can only be raised on our behalf by a third-party, so we'll be probably be waiting a few days before this is actioned.
Any help figuring this one out would be appreciated. I have output for SHOW BUFFERS sand SHOW TECH-SUPPORT if it helps
Cheers
Steve
Solved! Go to Solution.
- Labels:
-
Other Routing
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-25-2011 03:54 AM
Hello,
The top function allocating memory in the dead process being " AAA Interface Struct", we have a matching bug
(assuming that the dead process keeps growing and growing).
This is CSCee57076, duplicate of CSCee35379, and the fix is in 12.3(8)T1 and above.
Regards,
Herve
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-25-2011 03:17 AM
Hi Steven,
Can you show us the first few lines of "sh memory allocating-process totals" and "sh memory dead totals" ?
Regards,
Herve
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-25-2011 03:22 AM
Hi Herve,
Here's the output you requested:
...#sh memory allocating-process
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 63BF6B40 54564032 33027488 21536544 20830496 20263312
I/O 7000000 16777216 11016544 5760672 4970072 4895424
Processor memory
Address Bytes Prev Next Ref Alloc Proc Alloc PC What
63BF6B40 0000000276 00000000 63BF6C7C 001 *Dead* 607D98C4 AAA Interfa
ce Struct
63BF6C7C 0000000428 63BF6B40 63BF6E50 001 *Dead* 607D98C4 AAA Attr Li
st
63BF6E50 0000000340 63BF6C7C 63BF6FCC 001 *Dead* 607D98C4 AAA Interfa
ce Struct
63BF6FCC 0000000356 63BF6E50 63BF7158 001 *Dead* 607D98C4 GENERAL_DB
63BF7158 0000000104 63BF6FCC 63BF71E8 000 6075CCB8 (fragment)
63BF71E8 0000000276 63BF7158 63BF7324 001 *Dead* 607D98C4 AAA Interfa
ce Struct
63BF7324 0000000356 63BF71E8 63BF74B0 001 *Dead* 607D98C4 GENERAL_DB
63BF74B0 0000000356 63BF7324 63BF763C 001 *Dead* 607D98C4 GENERAL_DB
63BF763C 0000000356 63BF74B0 63BF77C8 001 *Dead* 607D98C4 GENERAL_DB
63BF77C8 0000000096 63BF763C 63BF7850 000 6075CCB8 (fragment)
63BF7850 0000000356 63BF77C8 63BF79DC 001 *Dead* 607D98C4 GENERAL_DB
63BF79DC 0000000428 63BF7850 63BF7BB0 001 *Dead* 607D98C4 AAA Attr Li
st
63BF7BB0 0000000144 63BF79DC 63BF7C68 000 607437A8 (fragment)
63BF7C68 0000000356 63BF7BB0 63BF7DF4 001 *Dead* 607D98C4 GENERAL_DB
63BF7DF4 0000000276 63BF7C68 63BF7F30 001 *Dead* 607D98C4 AAA Interfa
ce Struct
63BF7F30 0000000356 63BF7DF4 63BF80BC 001 *Dead* 607D98C4 GENERAL_DB
63BF80BC 0000000276 63BF7F30 63BF81F8 001 *Dead* 607D98C4 AAA Interfa
ce Struct
63BF81F8 0000000032 63BF80BC 63BF8240 000 6075B148 (fragment)
63BF8240 0000000276 63BF81F8 63BF837C 001 *Dead* 607D98C4 AAA Interfa
ce Struct
63BF837C 0000000296 63BF8240 63BF84CC 000 60EDA11C (coalesced)
63BF84CC 0000000428 63BF837C 63BF86A0 001 *Dead* 607D98C4 AAA Attr Li
st
63BF86A0 0000000276 63BF84CC 63BF87DC 001 *Dead* 607D98C4 AAA Interfa
ce Struct
63BF87DC 0000000036 63BF86A0 63BF8828 000 6075B148 (fragment)
63BF8828 0000000476 63BF87DC 63BF8A2C 001 *Dead* 607D98C4 AAA Attr Li
st
63BF8A2C 0000000276 63BF8828 63BF8B68 001 *Dead* 607D98C4 AAA Interfa
ce Struct
63BF8B68 0000000276 63BF8A2C 63BF8CA4 001 *Dead* 607D98C4 AAA Interfa
ce Struct
63BF8CA4 0000000036 63BF8B68 63BF8CF0 000 6075B148 (fragment)
63BF8CF0 0000000276 63BF8CA4 63BF8E2C 001 *Dead* 607D98C4 AAA Interfa
ce Struct
63BF8E2C 0000000356 63BF8CF0 63BF8FB8 001 *Dead* 607D98C4 GENERAL_DB
63BF8FB8 0000000108 63BF8E2C 63BF904C 000 6075CCB8 (fragment)
63BF904C 0000000356 63BF8FB8 63BF91D8 001 *Dead* 607D98C4 GENERAL_DB
...#sh memory dead totals
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 63BF6B40 54564032 33144160 21419872 20830496 20263312
I/O 7000000 16777216 11016544 5760672 4970072 4895424
Dead Proc Summary for: Processor
PC Total Name
0x607D98C4 2491 AAA Interface Struct
0x607D98C4 2491 AAA Attr List
0x607D98C4 2491 GENERAL_DB
0x607558D8 16 AAA Acct DB ch
0x60751308 3 AAA DB Chunk
0x607C4304 1 Chain Cache No
0x6074378C 1 AAA attr list handle IDs
0x608574E8 1 AAA Acct AVLnode chunk
0x60742D24 1 AAA attr list handle IDs
0x607558AC 1 AAA Acct DB chunk
0x607577A0 1 AAA Acct Rec chunk
0x607AAAA4 1 TTY timers array
0x607D98C4 1 AAA Unique Id Hash Table
Dead Proc Summary for: I/O
PC Total Name
cheers
Steve
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-25-2011 03:54 AM
Hello,
The top function allocating memory in the dead process being " AAA Interface Struct", we have a matching bug
(assuming that the dead process keeps growing and growing).
This is CSCee57076, duplicate of CSCee35379, and the fix is in 12.3(8)T1 and above.
Regards,
Herve
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-25-2011 06:53 AM
Thanks a million Herve!!!
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide
