cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
3007
Views
0
Helpful
17
Replies

Switch WS-C3850-12XS IOS-XE version 03.07.05E is running out of memory

fasanchezv
Level 1
Level 1

Hi Everyone,

Switch WS-C3850-12XS is running out of memory and we must reload the switch to restore services, how can we solve this situation?

 

switch was upgraded but the problem happens with version 03.07.04E and also with version 03.07.05E

 

these is the log messages when the problem occurs:

 

Mar 29 09:02:31: %SYS-2-NOMEMORY: No memory available for failure to allocate from heap 14
Mar 29 09:02:31: %SYS-2-MALLOCFAIL: Memory allocation of 1500 bytes failed from 0x5954FD0C, alignment 0
Pool: Processor Free: 2593640 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "CMI IOSd task", ipl= 0, pid= 22
-Traceback= 1#00182b81fe0a9f14dcd569d8a3d6a0a5 :54D82000+2865014 :54D82000+47A9048 :54D82000+47B0F50 :54D82000+47CDD14 :54D82000+47A23C4 :54D82000+47CFFBC :54D82000+47F9294 :54D82000+FEF73C :54D82000+FEFCA4 learning:2C0E4000+13A58 learning:2C0E4000+648C
Mar 29 09:02:34: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for messages. No memory available -Process= "Chunk Manager", ipl= 0, pid= 1
-Traceback= 1#00182b81fe0a9f14dcd569d8a3d6a0a5 :54D82000+2865014 :54D82000+47A2B78 :54D82000+47A3118 :54D82000+4783EDC
Mar 29 09:02:45: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for messages. No memory available -Process= "Chunk Manager", ipl= 0, pid= 1
-Traceback= 1#00182b81fe0a9f14dcd569d8a3d6a0a5 :54D82000+2865014 :54D82000+47A2B78 :54D82000+47A3118 :54D82000+4783EDC
%% Low on memory; try again later

%% Low on memory; try again later

%% Low on memory; try again later

%% Low on memory; try again later

Mar 29 09:02:55: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for messages. No memory available -Process= "Chunk Manager", ipl= 0, pid= 1
-Traceback= 1#00182b81fe0a9f14dcd569d8a3d6a0a5 :54D82000+2865014 :54D82000+47A2B78 :54D82000+47A3118 :54D82000+4783EDC
%% Low on memory; try again later

%% Low on memory; try again later

%% Low on memory; try again later

Mar 29 09:03:01: %ENTROPY-0-ENTROPY_ERROR: Unable to collect sufficient entropy
Mar 29 09:03:01: %SYS-2-NOMEMORY: No memory available for failure to allocate from heap 14
Mar 29 09:03:01: %SYS-2-MALLOCFAIL: Memory allocation of 1500 bytes failed from 0x5954FD0C, alignment 0
Pool: Processor Free: 2592580 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "Chunk Manager", ipl= 0, pid= 1
-Traceback= 1#00182b81fe0a9f14dcd569d8a3d6a0a5 :54D82000+2865014 :54D82000+47A9048 :54D82000+47B0F50 :54D82000+47CDD14 :54D82000+47A23C4 :54D82000+47A259C :54D82000+47A3118 :54D82000+4783EDC
Mar 29 09:03:05: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for messages. No memory available -Process= "Chunk Manager", ipl= 0, pid= 1
-Traceback= 1#00182b81fe0a9f14dcd569d8a3d6a0a5 :54D8200

 

 

------------------ show version ------------------

Cisco IOS Software, IOS-XE Software, Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 03.07.05E RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2017 by Cisco Systems, Inc.
Compiled Fri 10-Feb-17 07:59 by prod_rel_team

 

Cisco IOS-XE software, Copyright (c) 2005-2015 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0.
(http://www.gnu.org/licenses/gpl-2.0.html) For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

 

ROM: IOS-XE ROMMON
BOOTLDR: CAT3K_CAA Boot Loader (CAT3K_CAA-HBOOT-M) Version 3.58, RELEASE SOFTWARE (P)

DIST_DC_USB uptime is 20 hours, 26 minutes
Uptime for this control processor is 20 hours, 29 minutes
System returned to ROM by Power Failure at 12:28:04 CO Fri Feb 1 2019
System restarted at 13:14:05 CO Fri Mar 22 2019
System image file is "flash:packages.conf"
Last reload reason: Power Failure

 

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

License Level: Ipbase
License Type: Permanent
Next reload license Level: Ipbase

cisco WS-C3850-12XS (MIPS) processor with 4194304K bytes of physical memory.
2 Virtual Ethernet interfaces
16 Ten Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
253984K bytes of Crash Files at crashinfo:.
3431232K bytes of Flash at flash:.
0K bytes of Dummy USB Flash at usbflash0:.
0K bytes of at webui:.


Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 16 WS-C3850-12XS 03.07.05E cat3k_caa-universalk9 INSTALL


Configuration register is 0x102

 

 

------------------ show process memory sorted ------------------

System memory : 3937228K total, 1633944K used, 2303284K free, 222140K kernel reserved
Lowest(b) : 1653156696
PID Text Data Stack Heap RSS Total Process
9338 102396 37768 112 288 246620 350392 iosd
6508 11544 290280 92 36212 232156 611872 fed
9333 26024 566144 88 12944 142808 683340 wcm
7018 4 109452 116 103780 112164 131520 idope.py
6510 836 98832 88 1136 74092 260340 stack-mgr
6509 708 95148 92 852 56784 236336 platform_mgr
7111 124 91936 88 8188 46872 133884 cli_agent
10559 4 83760 224 19268 46720 141452 wnweb_paster.py
7097 5080 77432 96 5532 39468 217344 ffm
7098 1580 181508 88 13428 29876 233428 eicored
7089 56 51248 88 424 23964 103304 pdsd
7095 44 21276 88 164 20856 68756 mgmte_tap
7107 28 17172 88 164 20812 64500 liin_tap
7114 888 117776 88 5312 15268 162368 snmp_subagent
7091 1032 49008 88 2036 14080 88120 osinfo-provider
7082 1148 74212 88 1508 9932 118936 obfld
9334 172 81212 88 1436 9648 123608 table_mgr
9339 252 52356 88 1216 9608 92192 ha_mgr
7096 328 125216 88 292 9188 166384 installer
7124 820 46980 88 132 8408 91268 vman
7087 48 46796 88 584 8216 84920 plogd
7094 60 31372 88 560 7556 74472 mem_mgmt
7090 164 21560 88 296 7052 62776 gold_slave
6237 1440 11052 88 808 7008 44092 system_mgr
7099 268 42532 88 784 6676 75948 dtmgr
9335 100 37944 88 268 6628 74012 cmm
9337 52 38820 88 292 6516 74180 iifd
7086 52 55144 88 296 6356 91432 system_mgr_cli
7100 212 37880 88 164 5944 77672 cpumemd
7088 88 38836 88 296 5896 135564 psdprov
7085 44 33732 88 296 5732 68088 csprovider
7092 180 33708 88 296 5704 66892 oscore_p
7122 36 33704 88 296 5688 66652 sysinfo_p
7117 120 33704 88 296 5632 66740 erm_p
7119 80 33772 88 296 5608 66772 ngdumper_provid
7080 84 33708 88 296 5592 66704 profiled
7083 40 33704 88 296 5528 66636 consoled
6466 140 43660 88 988 5052 60300 slproc
7084 16 13548 88 320 4748 41304 console_relay
7093 32 22372 88 164 4428 50756 netd
10560 20 1008 92 132 4024 25156 xlogger
7079 20 13056 88 164 3676 39588 ns_oir_proxy
9321 20 832 88 132 3280 20436 xlogger
9013 20 832 88 132 3272 20436 xlogger
8940 752 1968 88 1920 3224 4988 rollback_timer.
5575 12 780 88 132 2956 18296 epc_sm_liaison
7128 8 8956 88 164 2496 21528 fs_cmd
9012 752 1960 88 1912 2428 4980 rollback_timer.
7081 752 624 88 576 1896 3644 oom_poll.sh
1 324 452 88 344 1744 4672 init
9184 752 484 88 436 1744 3504 nova_sntp.sh
9336 752 404 88 356 1644 3424 wnweb.sh
4071 372 180 88 132 1376 3284 dbus-daemon
5572 752 108 88 60 1244 3128 sh
4455 24 880 88 784 1172 2612 klogd
4585 24 880 88 784 1172 2612 lockd
4535 220 364 88 132 1060 5996 sshd
4587 220 364 88 132 1060 5996 nfsd
7771 1188 180 88 0 992 7924 libvirtd
9320 752 484 88 436 944 3504 nova_sntp.sh
5005 752 436 88 388 928 3456 sh
4547 144 332 88 132 840 3568 xinetd
4551 144 332 88 132 840 3568 xinetd
4588 144 332 88 132 840 3568 nfsd
4589 144 332 88 132 840 3568 nfsd
9324 60 160 88 132 724 3260 sntp
1161 120 232 88 132 644 2312 udevd
2967 120 168 88 132 624 2248 udevd
2968 120 168 88 132 624 2248 udevd
9017 20 240 88 132 588 2064 inotifywait
5008 20 240 88 132 532 2064 inotifywait
22907 28 160 88 132 464 1896 sleep
4467 16 160 88 132 460 1884 portmap
4586 16 160 88 132 460 1884 nfsd
4599 76 188 88 132 332 2708 rpc.mountd
2 0 0 0 0 0 0 kthreadd

 

------------------ show buffers ------------------

PDS public buffers

Public buffer pools:
PDS Buffer Pool 0, 2048 bytes (total 2048, permanent 2048, transient 0
peak 1238 @ 1465238):
938 in free list (2048 max allowed)
20937551 hits, 0 misses, 20936441 free hits, 0 free misses
PDS Buffer Pool 1, 12288 bytes (total 341, permanent 341, transient 0):
341 in free list (341 max allowed)
0 hits, 0 misses, 0 free hits, 0 free misses
SM Buf Group TX, 516 bytes (total 32, permanent 32, transient 0
peak 5 @ 11214):
32 in free list (32 max allowed)
36724 hits, 0 misses, 36724 free hits, 0 free misses


Header pools:
Packet Header Memory, 184 bytes (total 4681, permanent 4681, transient 0
peak 146 @ 1465238):
4663 in free list (4681 max allowed)
28519699 hits, 0 misses, 28519681 free hits, 0 free misses
Buffer Header Memory, 88 bytes (total 16384, permanent 16384, transient 0
peak 1238 @ 1465238):
15274 in free list (16384 max allowed)
20974280 hits, 0 misses, 20973170 free hits, 0 free misses
Header pools:

Header pools:
Header pools:
Packet Header Memory, 184 bytes (total 4681, permanent 4681, transient 0
peak 146 @ 1465238):
4663 in free list (4681 max allowed)
28519702 hits, 0 misses, 28519684 free hits, 0 free misses
Buffer Header Memory, 88 bytes (total 16384, permanent 16384, transient 0
peak 1238 @ 1465238):
15274 in free list (16384 max allowed)
20974281 hits, 0 misses, 20973171 free hits, 0 free misses

IOSd private buffers:

Buffer elements:
2018 in free list
33992964 hits, 0 misses, 1242 created

Public buffer pools:
Small buffers, 104 bytes (total 600, permanent 600):
600 in free list (100 min, 1250 max allowed)
12157968 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Middle buffers, 600 bytes (total 450, permanent 450):
450 in free list (50 min, 1000 max allowed)
30630 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Big buffers, 1536 bytes (total 450, permanent 450, peak 451 @ 20:27:24):
450 in free list (25 min, 1500 max allowed)
82459 hits, 0 misses, 1 trims, 1 created
0 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 10, permanent 10, peak 11 @ 20:27:24):
10 in free list (0 min, 300 max allowed)
0 hits, 0 misses, 1 trims, 1 created
0 failures (0 no memory)
Large buffers, 5024 bytes (total 10, permanent 10, peak 11 @ 20:27:24):
10 in free list (0 min, 30 max allowed)
0 hits, 0 misses, 1 trims, 1 created
0 failures (0 no memory)
VeryLarge buffers, 8232 bytes (total 50, permanent 50):
50 in free list (0 min, 150 max allowed)
0 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Huge buffers, 18024 bytes (total 5, permanent 5, peak 6 @ 20:27:24):
5 in free list (0 min, 13 max allowed)
0 hits, 0 misses, 1 trims, 1 created
0 failures (0 no memory)

Interface buffer pools:
ng3k_rx32 buffers, 1 bytes (total 1, permanent 1):
1 in free list (0 min, 1 max allowed)
0 hits, 0 misses
ng3k_rx21 buffers, 5 bytes (total 5, permanent 5):
5 in free list (0 min, 5 max allowed)
0 hits, 0 misses
CF Small buffers, 104 bytes (total 101, permanent 100, peak 101 @ 20:27:35):
101 in free list (100 min, 200 max allowed)
0 hits, 0 misses, 1227 trims, 1228 created
0 failures (0 no memory)
BIPC small buffers, 128 bytes (total 251, permanent 250, peak 251 @ 20:27:36):
251 in free list (250 min, 250 max allowed)
23 hits, 0 misses, 1227 trims, 1228 created
0 failures (0 no memory)
Generic ED Pool buffers, 512 bytes (total 101, permanent 100, peak 101 @ 20:27:21):
101 in free list (100 min, 100 max allowed)
0 hits, 0 misses
BIPC middle buffers, 600 bytes (total 301, permanent 300, peak 301 @ 20:27:36):
301 in free list (300 min, 300 max allowed)
154 hits, 0 misses, 1227 trims, 1228 created
0 failures (0 no memory)
CF Middle buffers, 600 bytes (total 101, permanent 100, peak 101 @ 20:27:35):
101 in free list (100 min, 200 max allowed)
0 hits, 0 misses, 1227 trims, 1228 created
0 failures (0 no memory)
Syslog ED Pool buffers, 600 bytes (total 133, permanent 132, peak 133 @ 20:27:21):
101 in free list (132 min, 132 max allowed)
1164 hits, 0 misses
CF Big buffers, 1536 bytes (total 26, permanent 25, peak 26 @ 20:27:35):
26 in free list (25 min, 50 max allowed)
0 hits, 0 misses, 1227 trims, 1228 created
0 failures (0 no memory)
ng3k_rx6 buffers, 1600 bytes (total 512, permanent 512):
512 in free list (0 min, 512 max allowed)
0 hits, 0 misses
ng3k_rx7 buffers, 1600 bytes (total 512, permanent 512):
512 in free list (0 min, 512 max allowed)
0 hits, 0 misses
ng3k_rx8 buffers, 1600 bytes (total 512, permanent 512):
512 in free list (0 min, 512 max allowed)
0 hits, 0 misses
ng3k_rx9 buffers, 1600 bytes (total 512, permanent 512):
512 in free list (0 min, 512 max allowed)
0 hits, 0 misses
ng3k_rx10 buffers, 1600 bytes (total 96, permanent 96):
96 in free list (0 min, 96 max allowed)
0 hits, 0 misses
ng3k_rx11 buffers, 1600 bytes (total 96, permanent 96):
96 in free list (0 min, 96 max allowed)
0 hits, 0 misses
ng3k_rx12 buffers, 1600 bytes (total 96, permanent 96):
96 in free list (0 min, 96 max allowed)
0 hits, 0 misses
ng3k_rx13 buffers, 1600 bytes (total 96, permanent 96):
96 in free list (0 min, 96 max allowed)
0 hits, 0 misses
ng3k_rx14 buffers, 1600 bytes (total 96, permanent 96):
96 in free list (0 min, 96 max allowed)
0 hits, 0 misses
ng3k_rx15 buffers, 1600 bytes (total 96, permanent 96):
96 in free list (0 min, 96 max allowed)
0 hits, 0 misses
ng3k_rx17 buffers, 1600 bytes (total 768, permanent 768):
768 in free list (0 min, 768 max allowed)
5834739 hits, 0 misses
ng3k_rx18 buffers, 1600 bytes (total 512, permanent 512):
512 in free list (0 min, 512 max allowed)
80478 hits, 0 misses
ng3k_rx19 buffers, 1600 bytes (total 1024, permanent 1024):
1024 in free list (0 min, 1024 max allowed)
2760661 hits, 0 misses
ng3k_rx20 buffers, 1600 bytes (total 96, permanent 96):
96 in free list (0 min, 96 max allowed)
0 hits, 0 misses
ng3k_rx22 buffers, 1600 bytes (total 16, permanent 16):
16 in free list (0 min, 16 max allowed)
0 hits, 0 misses
ng3k_rx23 buffers, 1600 bytes (total 16, permanent 16):
16 in free list (0 min, 16 max allowed)
0 hits, 0 misses
ng3k_rx24 buffers, 1600 bytes (total 16, permanent 16):
16 in free list (0 min, 16 max allowed)
0 hits, 0 misses
ng3k_rx25 buffers, 1600 bytes (total 16, permanent 16):
16 in free list (0 min, 16 max allowed)
0 hits, 0 misses
ng3k_rx26 buffers, 1600 bytes (total 16, permanent 16):
16 in free list (0 min, 16 max allowed)
0 hits, 0 misses
ng3k_rx27 buffers, 1600 bytes (total 512, permanent 512):
512 in free list (0 min, 512 max allowed)
547145 hits, 0 misses
ng3k_rx28 buffers, 1600 bytes (total 512, permanent 512):
512 in free list (0 min, 512 max allowed)
5578 hits, 0 misses
ng3k_rx29 buffers, 1600 bytes (total 2048, permanent 2048):
2047 in free list (0 min, 2048 max allowed)
4057 hits, 0 misses
ng3k punt fallback pool buffers, 1600 bytes (total 1500, permanent 1500):
1500 in free list (0 min, 0 max allowed)
0 hits, 0 misses
ng3k_rx33 buffers, 1600 bytes (total 128, permanent 128):
128 in free list (0 min, 128 max allowed)
0 hits, 0 misses
ng3k_rx34 buffers, 1600 bytes (total 128, permanent 128):
128 in free list (0 min, 128 max allowed)
0 hits, 0 misses
ng3k_rx35 buffers, 1600 bytes (total 96, permanent 96):
96 in free list (0 min, 96 max allowed)
0 hits, 0 misses
ng3k_rx36 buffers, 1600 bytes (total 512, permanent 512):
512 in free list (0 min, 512 max allowed)
0 hits, 0 misses
ng3k_rx37 buffers, 1600 bytes (total 512, permanent 512):
512 in free list (0 min, 512 max allowed)
0 hits, 0 misses
ng3k_rx38 buffers, 1600 bytes (total 512, permanent 512):
512 in free list (0 min, 512 max allowed)
0 hits, 0 misses
ng3k_rx40 buffers, 1600 bytes (total 96, permanent 96):
96 in free list (0 min, 96 max allowed)
0 hits, 0 misses
ng3k_rx41 buffers, 1600 bytes (total 512, permanent 512):
512 in free list (0 min, 512 max allowed)
2 hits, 0 misses
ng3k_rx39 buffers, 2048 bytes (total 1024, permanent 1024):
1024 in free list (0 min, 1024 max allowed)
0 hits, 0 misses
BIPC buffers, 4096 bytes (total 2, permanent 2):
2 in free list (1 min, 8 max allowed)
2 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
CF VeryBig buffers, 4520 bytes (total 3, permanent 2, peak 3 @ 20:27:35):
3 in free list (2 min, 4 max allowed)
0 hits, 0 misses, 1227 trims, 1228 created
0 failures (0 no memory)
CF Large buffers, 5024 bytes (total 2, permanent 1, peak 2 @ 20:27:35):
2 in free list (1 min, 2 max allowed)
0 hits, 0 misses, 7 trims, 8 created
0 failures (0 no memory)
ng3k punt jumbo pool buffers, 9300 bytes (total 128, permanent 128):
128 in free list (0 min, 0 max allowed)
0 hits, 0 misses
BIPC Medium buffers, 16384 bytes (total 6, permanent 5, peak 6 @ 20:27:36):
6 in free list (5 min, 5 max allowed)
2 hits, 0 misses, 1227 trims, 1228 created
0 failures (0 no memory)
BIPC Large buffers, 65535 bytes (total 3, permanent 2, peak 3 @ 20:27:36):
3 in free list (2 min, 2 max allowed)
0 hits, 0 misses, 1227 trims, 1228 created
0 failures (0 no memory)
IPC small buffers, 128 bytes (total 1200, permanent 1200):
1198 in free list (400 min, 1200 max allowed)
2 hits, 0 fallbacks, 0 trims, 0 created
0 failures (0 no memory)
0 max cache size, 0 in cache
0 hits in cache, 0 misses in cache
IPC middle buffers, 600 bytes (total 1200, permanent 1200):
1200 in free list (400 min, 1200 max allowed)
0 hits, 0 fallbacks, 0 trims, 0 created
0 failures (0 no memory)
0 max cache size, 0 in cache
0 hits in cache, 0 misses in cache
IPC Emergency buffers, 4096 bytes (total 301, permanent 300, peak 301 @ 20:27:36):
301 in free list (300 min, 300 max allowed)
0 hits, 0 misses, 1227 trims, 1228 created
0 failures (0 no memory)
0 max cache size, 0 in cache
0 hits in cache, 0 misses in cache
IPC buffers, 4096 bytes (total 1200, permanent 1200):
1199 in free list (400 min, 1200 max allowed)
1 hits, 0 fallbacks, 0 trims, 0 created
0 failures (0 no memory)
0 max cache size, 0 in cache
0 hits in cache, 0 misses in cache
IPC Medium buffers, 16384 bytes (total 40, permanent 40):
40 in free list (20 min, 40 max allowed)
0 hits, 0 fallbacks, 0 trims, 0 created
0 failures (0 no memory)
0 max cache size, 0 in cache
0 hits in cache, 0 misses in cache
Private Huge IPC buffers, 18024 bytes (total 1, permanent 0, peak 1 @ 20:27:36):
1 in free list (0 min, 4 max allowed)
0 hits, 0 misses, 1227 trims, 1228 created
0 failures (0 no memory)
Private Huge buffers, 65280 bytes (total 1, permanent 0, peak 1 @ 20:27:36):
1 in free list (0 min, 4 max allowed)
0 hits, 0 misses, 1227 trims, 1228 created
0 failures (0 no memory)
IPC Large buffers, 65535 bytes (total 104, permanent 104, peak 105 @ 20:27:36):
104 in free list (20 min, 104 max allowed)
0 hits, 0 misses, 1 trims, 1 created
0 failures (0 no memory)
0 max cache size, 0 in cache
0 hits in cache, 0 misses in cache

Header pools:
Header buffers, 0 bytes (total 267, permanent 256, peak 267 @ 20:25:35):
11 in free list (10 min, 512 max allowed)
253 hits, 3 misses, 9807 trims, 9818 created
0 failures (0 no memory)
256 max cache size, 256 in cache
0 hits in cache, 0 misses in cache

Particle Clones:
1024 clones, 0 hits, 0 misses

Public particle pools:
F/S buffers, 256 bytes (total 384, permanent 384):
128 in free list (128 min, 1024 max allowed)
256 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
256 max cache size, 256 in cache
0 hits in cache, 0 misses in cache
Normal buffers, 512 bytes (total 512, permanent 512):
384 in free list (128 min, 1024 max allowed)
128 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
128 max cache size, 128 in cache
0 hits in cache, 0 misses in cache

 

1 Accepted Solution

Accepted Solutions

Thanks to everyone who replay to this post, finally the solution was given by using the command:

ipv6 mld snooping

this command enable the MLD snooping, so, a per-VLAN IPv6 multicast address table is constructed in software and hardware. Switch then performs IPv6 multicast-address based bridging in hardware, which prevents these packets to be processed by software.

View solution in original post

17 Replies 17

Richard Burts
Hall of Fame
Hall of Fame

If the switch is running out of memory then it sounds like the version of code that it is running has some problem like a memory leak. Also I see some traceback messages in your logs. traceback is always an indicator of a software problem. My suggestion is that you address these problems by upgrading to a different version of the software.

 

HTH

 

Rick

HTH

Rick

Hi Richard, switch was upgraded but the problem happens with version 03.07.04E and also with version 03.07.05E

Reza Sharifi
Hall of Fame
Hall of Fame

Hi,

Tracebacks are usually a sign of software bug and crashes.  I would open a ticket with Cisco and send them the info for analysis.  Most likely, the fix is to upgrade to a different version.

HTH

Hi Reza, switch was upgraded but the problem happens with version 03.07.04E and also with version 03.07.05E

Hi,

I would still open a ticket. With tracebacks, sometimes when multiple IOS versions have the same issue, it is actually a hardware problem.

HTH

Deepak Kumar
VIP Alumni
VIP Alumni

Hi,

As usual, try to upgrade the Firmware and book a ticket with TAC.

 

Regards,

Deepak Kumar

Regards,
Deepak Kumar,
Don't forget to vote and accept the solution if this comment will help you!

Leo Laohoo
Hall of Fame
Hall of Fame
If the switch needs to be reloaded very regularly, then Rick, Reza and Deepak's recommendation is the ONLY way to go. Upgrade to 16.X.X.
NOTE: Open a TAC Case so the memory leak can be analyzed and identified.

fasanchezv
Level 1
Level 1

Hi Everyone,

 

thanks for all your answers and recommendations, due to this we are highly inclined to re-upgrade the software, however we have some additional information that maybe can be helpful to address any other solutions:

 

we noted that fed process has increased dramatically since the last reload:

 

just after the reload (3 days ago)

DIST_DC_USB#show processes memory sorted
System memory : 3937228K total, 1637932K used, 2299296K free, 222140K kernel reserved
--More-- ######### #########Lowest(b) : 1656821584
PID Text Data Stack Heap RSS Total Process
9343 102396 37768 112 288 245764 350392 iosd
6511 11544 303380 92 20640 245360 624972 fed
9338 26024 565236 88 8964 142372 682432 wcm
7021 4 109452 116 103780 112164 131520 idope.py
6513 836 103952 88 1136 73700 265460 stack-mgr

 

today

DIST_DC_USB#sh processes memory sorted
System memory : 3937228K total, 2391540K used, 1545688K free, 222140K kernel reserved
Lowest(b) : 876961032
PID Text Data Stack Heap RSS Total Process
6511 11544 1002172 92 57928 943464 1323764 fed
9343 102396 103312 112 288 290296 415936 iosd
9338 26024 566260 88 8964 142800 683456 wcm
7021 4 109452 116 103780 112164 131520 idope.py

 

DIST_DC_USB#show memory detailed process fed  
System memory : 3937228K total, 2391324K used, 1545904K free, 222140K kernel reserved
Lowest(b) : 876961032

Process fed, type L, PID = 6511
1323764K total, 11544K text, 1002172K data, 92K stack, 57928K heap

 

DIST_DC_USB#show platform fed-spi-fc statistics | i LEARNING
LEARNING | 0| 0| 0| 0|8504658| 0| 0| 0| 0| 0| 0
DIST_DC_USB#show platform fed-spi-fc statistics | i LEARNING
LEARNING | 0| 0| 0| 0|8504709| 0| 0| 0| 0| 0| 0

 

and logs shows a lot of mac moves like these ones

 

Apr 1 12:29:41: %SW_MATM-4-MACFLAP_NOTIF: Host 0015.5d86.0a26 in vlan 134 is flapping between port Te1/0/5 and port Te1/0/6
Apr 1 12:29:41: %SW_MATM-4-MACFLAP_NOTIF: Host 4437.e6d6.e62b in vlan 134 is flapping between port Te1/0/5 and port Te1/0/6
Apr 1 12:29:41: %SW_MATM-4-MACFLAP_NOTIF: Host 001a.64c1.c488 in vlan 134 is flapping between port Te1/0/5 and port Te1/0/6
Apr 1 12:30:42: %SW_MATM-4-MACFLAP_NOTIF: Host 0021.5e26.59f8 in vlan 134 is flapping between port Te1/0/5 and port Te1/0/6
Apr 1 12:49:42: %SW_MATM-4-MACFLAP_NOTIF: Host 001a.64c1.c488 in vlan 134 is flapping between port Te1/0/5 and port Te1/0/6
Apr 1 12:50:42: %SW_MATM-4-MACFLAP_NOTIF: Host 001a.64c1.c488 in vlan 134 is flapping between port Te1/0/5 and port Te1/0/6
Apr 1 12:50:56: %SW_MATM-4-MACFLAP_NOTIF: Host 0015.5d86.0a28 in vlan 134 is flapping between port Te1/0/5 and port Te1/0/6
Apr 1 12:51:41: %SW_MATM-4-MACFLAP_NOTIF: Host 0015.5d86.0a29 in vlan 134 is flapping between port Te1/0/5 and port Te1/0/6
Apr 1 12:51:41: %SW_MATM-4-MACFLAP_NOTIF: Host 0015.5d86.0a2c in vlan 134 is flapping between port Te1/0/5 and port Te1/0/6
Apr 1 12:51:41: %SW_MATM-4-MACFLAP_NOTIF: Host 0015.5d86.0a24 in vlan 134 is flapping between port Te1/0/5 and port Te1/0/6

 

is fed process related to spanning tree calculations? it could be the cause of this problem?

 

thanks for your comments

MAC Flapping can cause high CPU.
This to me means Ten 1/0/5 and Ten 1/0/6 isn't configured for Etherchannel.

Hi Leo,

 

Do you mean that the mac flapping shouldn`t affect the memory usage and it should affect only the CPU utilization?

If so, we haven't seen high CPU utilization just high memory utilization, so can we dismiss mac flapping as one of the causes of the increase of memory utilization we have seen along these days?

 

Regarding your question about interfaces, the interface Ten1/0/5 and Ten 1/0/6 have two independent connections to two different switches.


@fasanchezv wrote:

the interface Ten1/0/5 and Ten 1/0/6 have two independent connections to two different switches.


Well, this means that someone has installed an un-managed switch somewhere in the network and has bridged the two (or more) VLANs.  

I'd go and disconnect that switch and see if it improves the CPU. 

jalejand
Cisco Employee
Cisco Employee
The traceback you are currently facing is related to numerous high CPU utilization cases, there is no specific memory leak related to such bug, but as you have mac flapping in logs, STP process or a possible broadcast storm could be hitting your CPU utilization to abnormal levels.

How is the CPU level of this switch? #show process cpu sor | ex 0.0
Between Te1/0/5 and 6 flapping exists, can you shut down one of these interfaces?

Hi jalejand,

 

this is the CPU level

 

DIST_DC_USB#sh proc cpu sort
Core 0: CPU utilization for five seconds: 9%; one minute: 11%; five minutes: 11%
Core 1: CPU utilization for five seconds: 4%; one minute: 3%; five minutes: 3%
Core 2: CPU utilization for five seconds: 2%; one minute: 2%; five minutes: 2%
Core 3: CPU utilization for five seconds: 1%; one minute: 2%; five minutes: 2%
Core 4: CPU utilization for five seconds: 1%; one minute: 2%; five minutes: 2%
Core 5: CPU utilization for five seconds: 0%; one minute: 3%; five minutes: 3%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
6511 3442031 88611382 42 1.81 2.25 2.14 1088 fed
9343 30781 92519416 371 1.22 1.29 1.18 0 iosd
6513 2027500 10171095 199 0.13 0.11 0.10 0 stack-mgr
9338 1011130 11598961 8 0.10 0.07 0.07 0 wcm
82 100 272237 0 0.07 0.07 0.07 0 events/1
7090 34840 1364089 25 0.03 0.03 0.03 0 plogd
7103 350640 563241 622 0.03 0.02 0.03 0 cpumemd
1 1310 1270 1031 0.00 0.00 0.00 0 init

 

and this is the memory

 

DIST_DC_USB#sh proc memory sorted
System memory : 3937228K total, 1770300K used, 2166928K free, 222140K kernel reserved
Lowest(b) : 873569632
PID Text Data Stack Heap RSS Total Process
6511 11544 1002176 92 57928 309472 1323768 fed
9343 102396 103312 112 288 301436 415936 iosd
9338 26024 566260 88 8964 142800 683456 wcm
7021 4 109452 116 103780 112164 131520 idope.py
6513 836 103952 88 1136 74088 265484 stack-mgr
6512 708 95152 92 856 56844 236340 platform_mgr
7115 124 92112 88 8364 47108 134060 cli_agent
10553 4 83692 224 19200 46644 141384 wnweb_paster.py
7100 5080 77396 100 5496 39428 217312 ffm
7094 1032 44228 88 1352 30520 101468 osinfo-provider

 

I think the high utilization is only seen at memory level not al cpu level, what do you think?

we have to schedule a MW in order to shutdown one of these interfaces. if we shutdown one of these interfaces besides the mac flapping should we see a decrease in memory utilization?

Richard Burts
Hall of Fame
Hall of Fame

In my initial response in this discussion I mentioned the tracebacks and indicated that they represent some software issue. I assumed that the tracebacks were related to the initial problem report of low memory. Now there is discussion that the tracebacks are related to CPU utilization and are not related to low memory. We should investigate the CPU issue. But we should not lose sight of the original problem report of low memory.

Mar 29 09:02:31: %SYS-2-NOMEMORY: No memory available for failure to allocate from heap 14
Mar 29 09:02:31: %SYS-2-MALLOCFAIL: Memory allocation of 1500 bytes failed from 0x5954FD0C, alignment 0
Pool: Processor Free: 2593640 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool

 

Mar 29 09:03:01: %ENTROPY-0-ENTROPY_ERROR: Unable to collect sufficient entropy
Mar 29 09:03:01: %SYS-2-NOMEMORY: No memory available for failure to allocate from heap 14
Mar 29 09:03:01: %SYS-2-MALLOCFAIL: Memory allocation of 1500 bytes failed from 0x5954FD0C, alignment 0
Pool: Processor Free: 2592580 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "Chunk Manager", ipl= 0, pid= 1

 

My sense of the 2 issues is that the low memory is a more significant issue than the high CPU tracebacks. 

 

HTH

 

Rick

HTH

Rick
Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Review Cisco Networking products for a $25 gift card