Cisco 6500 %SYS-SP-2-MALLOCFAIL: Memory allocation and Module Pwrdown

realshiro · ‎04-01-2015

I found log as below , what's the root cause ?

But now everything is comeback with reload the box.

------------------ show module ------------------

Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
1 0017.59f4.f440 to 0017.59f4.f46f 1.5 12.2(14r)S5 12.2(18)SXE6 Ok
2 0017.9453.49c4 to 0017.9453.49f3 2.3 12.2(14r)S5 12.2(18)SXE6 Ok
3 0017.e0b9.800c to 0017.e0b9.800f 2.2 12.2(14r)S5 12.2(18)SXE6 PwrDown
4 0017.e0b9.6c3c to 0017.e0b9.6c3f 2.2 12.2(14r)S5 12.2(18)SXE6 PwrDown
5 0013.c43a.f734 to 0013.c43a.f737 5.2 8.4(2) 12.2(18)SXE6 Ok
7 0017.95a3.b326 to 0017.95a3.b32d 4.0 7.2(1) 3.2(1) Ok
8 0018.197e.221a to 0018.197e.2221 6.1 7.2(1) 5.0(2) Ok

Mod Sub-Module Model Serial Hw Status
--- --------------------------- ------------------ ------------ ------- -------
1 Centralized Forwarding Card WS-F6700-CFC SAL1021PAZL 2.0 Ok
2 Centralized Forwarding Card WS-F6700-CFC SAL1019MFB6 2.0 Ok
3 Centralized Forwarding Card WS-F6700-CFC SAL1011G9XY 2.0 PwrDown
4 Centralized Forwarding Card WS-F6700-CFC SAL09391R9E 2.0 PwrDown
5 Policy Feature Card 3 WS-F6K-PFC3B SAL1020NATD 2.3 Ok
5 MSFC3 Daughterboard WS-SUP720 SAL1020MXCE 2.5 Ok
8 IDS 2 accelerator board WS-SVC-IDSUPG ADEI61601174 2.4 Ok

show log

-Process= "DiagCard8/-1", ipl= 0, pid= 62
-Traceback= 4017B2A8 4017E828 40883454 4086DAC8 40874484 4087237C 40872190 40193958 40193944
Mar 27 14:56:43: %SYS-SP-2-MALLOCFAIL: Memory allocation of 5768 bytes failed from 0x4088344C, alignment 8
Pool: Processor Free: 349312 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "DiagCard7/-1", ipl= 0, pid= 31
-Traceback= 4017B2A8 4017E828 40883454 4086DAC8 40874484 4087237C 40872190 40193958 40193944scp_fpoe_req: memory allocation error, subopcode=13, count=1
scp_fpoe_req: memory allocation error, subopcode=13, count=1
scp_fpoe_req: memory allocation error, subopcode=13, count=1
Mar 27 15:00:57: %SYS-SP-2-MALLOCFAIL: Memory allocation of 5768 bytes failed from 0x4088344C, alignment 8
Pool: Processor Free: 320136 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool

------------------ show process memory ------------------

Processor Pool Total: 396081504 Used: 72497944 Free: 323583560
I/O Pool Total: 67108864 Used: 10418768 Free: 56690096

PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 85845488 14694808 68388552 0 0 *Init*
0 0 3384 3608082936 3384 0 0 *Sched*
0 0 3603507336 4039919552 1503656 9841264 75552 *Dead*
1 0 4433280 4431792 35104 0 0 Chunk Manager
2 0 192 192 4024 0 0 Load Meter
3 0 179024 1827956488 21056 31416 1052744 OSPF Hello
4 0 0 17136 7040 0 0 DHCPD Timer
5 0 0 5128 7064 0 0 Check heaps
6 0 8443336 6991440 7024 2083620 1941296 Pool Manager
7 0 192 192 7024 0 0 Timers
8 0 2101377608 3184839112 1509960 0 2464 ARP Input
9 0 0 0 7024 0 0 AAA_SERVER_DEADT
10 0 192 192 7024 0 0 AAA high-capacit
11 0 16954336 14610312 2351416 0 0 Entity MIB API
12 0 0 0 7024 0 0 IFS Agent Manage
13 0 0 0 7024 0 0 IPC Dynamic Cach
14 0 48464 42784 12664 0 0 PF_Split Sync Pr
15 0 21384 0 7024 0 0 IPC Periodic Tim
16 0 0 0 7024 0 0 IPC Deferred Por
17 0 2212272 135280 125800 0 0 IPC Seat Manager
18 0 0 0 7024 0 0 IPC Stdby Update
19 0 192 192 7024 0 0 DDR Timers
20 0 192 192 13024 0 0 Dialer event
21 0 192 192 7024 0 0 Serial Backgroun
22 0 1568030944 1568076448 7024 0 0 EnvMon
23 0 0 0 7024 0 0 Compute SRP rate
24 0 10056 0 20000 0 0 EEM ED Syslog
25 0 168424 46352 86456 0 0 rf proxy rp agen
26 0 0 0 7024 0 0 HC Counter Timer
27 0 0 0 7024 0 0 SERIAL A'detect
28 0 0 0 7024 0 0 Snmp ICC Process
29 0 88568 27264 15984 0 0 Cat6k SNMP
30 0 3768 225864 10600 0 0 Cat6k SNMP Trap
31 0 0 0 7024 0 0 Dynamic ARP Insp
32 0 0 0 7024 0 0 Critical Bkgnd
33 0 1817424 29672 669312 0 0 Net Background
34 0 1744 192 13024 0 0 Logger
35 0 192 59400 10024 0 0 TTY Background
36 0 0 109508856 7024 0 0 Per-Second Jobs
37 0 0 2710528 7024 0 0 Per-minute Jobs
38 0 0 0 7024 0 0 PF Sync process
39 0 384 520 7024 0 0 RPC rf_rfmib_not
40 0 195336 1383593368 53952 0 0 Env action
41 0 192 192 7024 0 0 Flash ICC Server
42 0 168 0 7192 0 0 itasca
43 0 2200 2200 7024 0 0 EOBC Monitor Pro
44 0 43856 43456 7024 0 0 RPC pf-split-rp
45 0 160056 157720 10720 0 0 draco-oir-proces
46 0 26624 148024 17760 0 0 RTR Scheduler
47 0 0 0 7024 0 0 MWAM CONFIG ON S
48 0 141632 169296 7192 0 0 svclc
49 0 768 768 7024 0 0 RPC Sync
50 0 384 520 7024 0 0 RPC rpc-slave-15
51 0 0 1968 7024 0 0 SCP Multicast
52 0 0 0 7024 0 0 RF CWAN HA Proce
53 0 0 0 7024 0 0 CWAN HA VLAN SYN
54 0 0 0 7024 0 0 CWAN IF EVENT HA
55 0 0 0 7024 0 0 CWAN APS HA Proc
56 0 0 0 7024 0 0 CWAN SRP HA Proc
57 0 0 0 7024 0 0 CWAN LTL Mgr HA
58 0 26768 608 7024 5544 0 Heartbeat Proces
59 0 7544 18232 11976 0 0 Const ICC reques
60 0 0 0 7024 0 0 Const ICC event
61 0 0 0 7024 0 0 Const ICC except
62 0 0 0 7024 0 0 ICC Local Reques
63 0 0 68006160 7024 0 0 ICC Async mcast
64 0 0 0 7024 0 0 ICC_REQ Debug Pr
65 0 0 0 7024 0 0 ICC FC Process
66 0 0 0 7024 0 0 ICC FC Test Send
67 0 0 0 7024 0 0 ICC FC Test Low
68 0 0 0 4024 0 0 Inode Table Dest
69 0 0 0 7024 0 0 IfIndex Bulk Syn
70 0 0 0 7024 0 0 CWRP SPA TSM HA
71 0 0 0 7024 0 0 Mistral Reset Pr
72 0 0 0 7024 0 0 ECC correctable
73 0 68104 3896 35096 0 0 PF_Init Process
74 0 45056 24808 22296 0 0 RPC idprom-MP
75 0 0 0 7128 0 0 Net Input
76 0 192 192 7072 0 0 Compute load avg
77 0 3136 1544 7024 0 0 RTTYS Process
78 0 0 0 7024 0 0 BACK CHECK
79 0 10056 0 17080 0 0 chkpt message ha
80 0 0 0 7024 0 0 cpf_process_msg_
81 0 0 0 7024 0 0 cpf_process_ipcQ
82 0 0 0 7024 0 0 IPC Apps Task
83 0 60104 0 67184 0 0 RADIUS
84 0 0 8664 7096 0 0 AggMgr Process
85 0 1152 1640 7024 0 0 SFF8472
86 0 0 0 7024 0 0 Remote Console P
87 0 192 192 7024 0 0 CWAN port channe
88 0 0 0 7024 0 0 POS APS Event Pr
89 0 0 0 7024 0 0 CWAN OIR Handler
90 0 0 0 7024 0 0 DSX3MIB ll handl
91 0 0 0 7024 0 0 FPD Management P
92 0 0 0 7024 0 0 FPD Action Proce
93 0 0 0 7024 0 0 spa_oir_audit_pr
94 0 1808 384 8448 0 0 ACE_REQUESTS han
95 0 16688 0 41712 0 0 ACE Tunnel Task
96 0 0 0 25024 0 0 ACE VRF Task
97 0 0 0 7024 0 0 CWAN CHOCX PROCE
98 0 0 0 7024 0 0 HAL Mailbox
99 0 0 0 7024 0 0 CWAN CHDSX PROCE
100 0 0 0 7024 0 0 CT3 Mailbox
101 0 0 0 7024 0 0 CE3 Mailbox
102 0 192 192 7024 0 0 MIP Mailbox
103 0 0 0 7024 0 0 vcq_proc
104 0 7104 0 7104 0 0 Earl NDE Task
105 0 163924608 163577120 7024 50820 0 esw_vlan_stat_pr
106 0 2688 3064 7216 0 0 Gemini L3 forwar
107 0 0 0 7024 0 0 CWRP SPA ATM OIR
108 0 0 0 7056 0 0 ACE event
109 0 0 0 7024 0 0 ACE HAPI
110 0 0 0 7024 0 0 ACE policy loade
111 0 192 192 7024 0 0 AAA Server
112 0 0 0 7024 0 0 AAA ACCT Proc
113 0 0 0 7024 0 0 ACCT Periodic Pr
114 0 3562610112 3562392984 63328 0 0 CDP Protocol
115 0 747080 744104 12088 0 0 IP RTR Probe Max
116 0 168 0 7192 0 0 CEF Scanner
117 0 192 192 7024 0 0 AAA Dictionary R
118 0 0 0 7024 0 0 DHCP Snooping
119 0 56860656 797953784 869192 332108 36764 IP Input
120 0 192 192 7024 0 0 TurboACL
121 0 192 192 7024 0 0 MOP Protocols
122 0 0 0 7024 0 0 PPPATM Session d
123 0 192 192 13024 0 0 PASVC create VA
124 0 752664 749704 13096 0 0 IP RTR Probe Max
125 0 0 0 7024 0 0 HDLC HA
126 0 0 0 7024 0 0 ARP HA
127 0 408 216 7216 0 0 Spanning Tree
128 0 0 0 7024 0 0 AC Mgr
129 0 0 0 13024 0 0 SSS Manager
130 0 0 0 13024 0 0 SSS Test Client
131 0 519448 186128 354624 0 0 SW VLAN RP
132 0 96 88 7120 0 0 const ipfib per-
133 0 3240 272 9992 0 0 Const MPLS RP pr
134 0 7264 608 13120 0 0 Const Rate Limit
135 0 0 0 7024 0 0 CWAN LTL manager
136 0 82089144 82084864 17328 0 0 SSH Event handle
137 0 123928 515992 10312 0 0 CEF RP IPC Backg
138 0 467088 495768 71320 0 0 HWIF QoS Process
139 0 192 192 7024 0 0 IP Hdr Comp Proc
140 0 755960 752992 13072 0 0 IP RTR Probe Max
141 0 0 0 7024 0 0 MLS FIB RP
142 0 192 192 7024 0 0 mcast_rp_icc Pro
143 0 192 192 7024 0 0 QOS Stats Export
144 0 1487247328 1487247328 7024 0 0 QOS Stats Gather
145 0 392 280 7224 0 0 cfib v6 per pref
146 0 0 0 7024 0 0 LSP Tunnel FRR
147 0 0 0 7024 0 0 MPLS Auto-Tunnel
148 0 0 0 10024 0 0 IP IRDP
149 0 0 0 7024 0 0 PPP IP Add Route
150 0 0 0 7024 0 0 ATM HA
151 0 0 0 7024 0 0 SNMP Timers
152 0 0 0 7024 0 0 Inspect Timer
153 0 0 0 7024 0 0 Authentication P
154 0 14744 2736 11576 0 0 IP Background
155 0 84780544 71423280 142776 0 0 IP RIB Update
156 0 0 0 7024 0 0 CHKPT DevTest
157 0 0 0 7024 0 0 Socket Timers
158 0 3045325760 3045203408 33456 0 0 DHCPD Receive
159 0 0 13472 7024 0 0 IP Cache Ager
160 0 880 192 7024 0 0 Adj Manager
161 0 0 0 7024 0 0 FR HA
162 0 0 0 7024 0 0 HTTP Timer
163 0 6552 212299976 13208 0 0 TCP Timer
164 0 3648045496 208984 16072 0 0 TCP Protocols
165 0 73328 7608 43224 0 0 IPC LC Message H
166 0 0 0 7024 0 0 PPP HA
167 0 0 0 7024 0 0 SLB CF process
168 0 583872 480 569056 0 0 CEF process
169 0 384 192 7216 0 0 DVMRP Timers
170 0 0 0 25024 0 0 COPS
171 0 1091537256 1023641568 21496 0 0 L3 Manager
172 0 23501864 2012824240 869376 0 0 FM core
173 0 0 0 7024 0 0 Probe Input
174 0 0 0 7024 0 0 RARP Input
175 0 0 0 7024 0 0 PAD InCall
176 0 192 192 13024 0 0 X.25 Background
177 0 0 0 7024 0 0 X.25 Encaps Mana
178 0 0 0 7024 0 0 CWAN QinQ tasks
179 0 192 192 7024 0 0 HIDDEN VLAN Proc
180 0 0 0 7024 0 0 CWAN-RP SCP Proc
181 0 21408 0 14216 0 0 CWAN-RP SCP Inpu
182 0 616 320 7320 0 0 LSP Verification
183 0 0 0 7024 0 0 VFI Mgr
184 0 0 0 7024 0 0 AToM manager
185 0 192 17984 7024 0 0 AToM switching m
186 0 0 0 7024 0 0 AToM LDP manager
187 0 14736 192 11512 0 0 L2TRACE SERVER
188 0 1272 2304 7024 0 0 mfib-const-rp Pr
189 0 5144 248 12168 0 0 MLSM-CC Process
190 0 192 192 7024 0 0 MWAM_CONSOLE Tim
191 0 192 192 7024 0 0 CWPA QoS
192 0 192 192 61024 0 0 SIP10G QM Proces
193 0 656 272 7104 0 0 Const TCAM Mgr R
194 0 0 0 7024 0 0 Crypto HW Proc
195 0 0 0 7024 0 0 IPv6 CEF process
196 0 192 192 7024 0 0 ATIP_UDP_TSK
197 0 0 0 7024 0 0 IDS Timer
198 0 192 192 7024 0 0 Tag Input
199 0 0 0 7024 0 0 LFIB FRR IOS
200 0 0 0 7024 0 0 LDP Background
201 0 192 192 7056 0 0 AAA Cached Serve
202 0 192 192 7024 0 0 LOCAL AAA
203 0 0 0 7024 0 0 Crypto Support
204 0 0 0 13024 0 0 Encrypt Proc
205 0 2597247464 2597246328 10160 0 0 Key Proc
206 0 7744 1344 15424 0 0 Crypto CA
207 0 0 0 9024 0 0 Crypto PKI-CRL
208 0 344 0 7368 0 0 CCSWVOICE
209 0 0 0 7024 0 0 Laminar Icc Eve
210 0 192 192 13024 0 0 SLB Main CPU pro
211 0 176 488 25024 0 0 Crypto ACL
212 0 0 0 7024 0 0 CRYPTO QoS proce
213 0 24960 0 37984 0 0 Crypto Delete Ma
214 0 10728 192 23560 0 0 Crypto IKMP
215 0 183624 142784 53072 0 0 IPSEC key engine
216 0 0 0 7024 0 0 IPSEC manual key
217 0 192 192 7024 0 0 RADIUS TEST CMD
218 0 192 192 7024 0 0 AAA SEND STOP EV
219 0 0 0 7024 0 0 L3MM Process
220 0 0 0 7024 0 0 Syslog Traps
221 0 192 328 7080 0 0 RPC aapi_rp
222 0 138944 69360 76256 0 0 RPC capi_rp
223 0 192 328 7024 0 0 RPC chapi_rp
224 0 192 328 7024 0 0 RPC dot1x_aaa_rp
225 0 192 328 7024 0 0 RPC epld_sp2rp
226 0 27344 27480 7024 0 0 RPC idbman_sp2mp
227 0 168 0 7192 0 0 LC Registry Expo
228 0 2080 632 7024 0 0 Logger ICC Proce
229 0 2286185272 2286027320 103096 11396 0 RPC online_diag_
230 0 192 328 7024 0 0 RPC online_wan_d
231 0 424712 253640 158864 0 0 RPC pagp_switch_
232 0 192 328 7024 0 0 RPC pm_switch_sp
233 0 5248 328 12080 0 0 RPC psecure_rp
234 0 2336 2472 7024 0 0 RPC pt_switch_sp
235 0 192 328 7024 0 0 RPC qnq_switch_s
236 0 192 328 7024 0 0 RPC rf_cli_sp2rp
237 0 0 0 7024 0 0 SONET Traps
238 0 192 192 7024 0 0 VTPMIB EDIT BUFF
239 0 192 328 7024 0 0 RPC snmp_notify
240 0 8624 8568 8952 0 0 RPC snmp_stackli
241 0 192 328 7024 0 0 RPC span_switch_
242 0 40058792 40058928 7024 0 0 RPC stp_sp2mp_ap
243 0 168 0 7240 0 0 DHCPD Database
244 0 392 392 7088 0 0 Module enable-di
245 0 50488 23672 33976 0 0 RPC c6k_power_rp
246 0 203882824 3115251944 7024 17556 0 RPC c6k_rp_envir
247 0 0 0 7024 0 0 CONST AFL
248 0 0 0 7024 0 0 Const FIB MAC
249 0 20520 18808 7024 0 0 Constellation Ba
250 0 192 192 7024 0 0 Rommon upgrade p
251 0 192 192 7024 0 0 RP Error Detecti
252 0 192 328 7024 0 0 RPC error_detect
253 0 10224 0 17248 0 0 RP L2 Applicatio
254 0 0 0 7024 0 0 LACP-MP Process
255 0 67848 68352 7192 0 0 Online Diag EEM
256 0 5056 0 12080 0 0 PAgP-MP Process
257 0 192 328 7024 0 0 RPC pclc_g2_agsm
258 0 761320 761456 7024 0 0 RPC pclc_g2_fw_r
259 0 192 328 7024 0 0 RPC pikachu_rp
260 0 8152792 2605243888 360496 2156 0 IGMP Input
261 0 351025264 5728200 7264 0 0 RPC pm-mp
262 0 0 0 7024 0 0 PM Callback
263 0 1765825960 1765309208 289504 122584 0 Port manager per
264 0 0 0 7024 0 0 Max CPU utilizat
265 0 76408 151424 61384 0 0 QM Process on RP
266 0 192 2280 61024 0 0 QM Reply Process
267 0 60320 3408 7024 0 0 RPC lcc_sp
268 0 2496 2632 7024 0 0 RPC fabman_sp
269 0 192 192 7024 0 0 rp to lc clock u
270 0 0 0 10056 0 0 Connection Mgr
271 0 5520 5256 7288 0 0 EC-Card-Cfg
272 0 90608 51248 7024 0 0 RPC sw_vlan_rp
273 0 640 776 7024 0 0 RPC udld_rp
274 0 745992 520456 7024 0 0 RPC vtp_const_ap
275 0 10296 664 17320 0 0 EEM Server
276 0 9960 192 16128 0 0 EEM Policy Direc
277 0 9408 0 19432 0 0 EEM ED CLI
278 0 9408 0 19432 0 0 EEM ED Counter
279 0 9408 0 19480 0 0 EM ED GOLD
280 0 9408 0 19432 0 0 EEM ED Interface
281 0 9408 0 19432 0 0 EEM ED IOSWD
282 0 9408 0 19432 0 0 EEM ED Memory-th
283 0 9408 0 19432 0 0 EEM ED None
284 0 9752 384 19776 0 0 EM ED RF
285 0 9408 0 19432 0 0 EEM ED SNMP
286 0 9408 0 19432 0 0 EEM ED Timer
287 0 192 192 7024 0 0 CWAN Bridge task
288 0 0 0 7024 0 0 DHCP Snooping Sw
289 0 1135121184 603826784 97384 13552 0 PIM Process
290 0 168 0 7192 0 0 Online Diag CNS
291 0 168 0 7192 0 0 Online Diag CNS
292 0 0 96 7024 0 0 VACL Logger
293 0 168 2344988816 7192 0 0 Mwheel Process
294 0 0 0 7024 0 0 NetFlow Agg Task
295 0 0 0 7024 0 0 CWAN OIR IPC Rea
296 0 5840 3760 7024 0 0 RTTYS Process
297 0 81329648 81259480 691096 0 0 MLSM Process
298 0 0 0 7024 0 0 PF Clock Process
299 0 3631364152 3631354040 7024 0 0 MLSM Stat Proces
300 0 2204304 30128 23752 538076 6160 Standby (HSRP)
301 0 754816 751840 13000 0 0 IP RTR Probe Max
302 0 3451294192 3451056072 13912 0 0 IP SNMP
303 0 2487425416 76008 13024 0 0 PDU DISPATCHER
304 0 1937713720 129257784 139824 0 0 SNMP ENGINE
305 0 1310762040 1310852320 13480 0 0 SNMP ConfCopyPro
306 0 0 0 13024 0 0 SNMP Traps
307 0 3984 192 7512 0 0 NTP
308 0 1371217088 3843195224 51648 0 169400 OSPF Router 10
309 0 759968 756968 13032 0 0 IP RTR Probe Max
310 0 763400 760376 13088 0 0 IP RTR Probe Max
311 0 760312 757296 13104 0 0 IP RTR Probe Max
312 0 746264 743120 13256 0 0 IP RTR Probe Max
313 0 757624 754432 13328 0 0 IP RTR Probe Max
314 0 750072 746864 13288 0 0 IP RTR Probe Max
315 0 754232 751088 13184 0 0 IP RTR Probe Max
316 0 757464 754312 13280 0 0 IP RTR Probe Max
317 0 1012970184 1012956184 13184 0 0 IP RTR Probe Max
318 0 0 0 7064 0 0 draco-oir-proces
320 0 0 0 7160 0 0 draco-oir-proces
321 2 535448 501384 44800 0 0 SSH Process
322 3 7249928 7171696 81344 0 0 SSH Process

Boyan Sotirov · ‎04-01-2015

Memory allocation is usually a software problem. In your case it seems the system goes out of memory. It happens due to processess not freeing memory after it's being used.

Why this happens it's an interesting thing here...

Did you try with different interface cards if by any chance you have any spares?

The cards in slots 1-4 are the same model, but they're different hardware revisions though. And the one that are failing are with HW revision 2.2. It might be related. But we may also have a faulty interface card.

It would be also nice if you could test the interface cards in another chassis...