Switch WS-C3850-12XS IOS-XE version 03.07.05E is running out of memory

[fasanchezv]

Hi Everyone,

Switch WS-C3850-12XS is running out of memory and we must reload the switch to restore services, how can we solve this situation?

switch was upgraded but the problem happens with version 03.07.04E and also with version 03.07.05E

these is the log messages when the problem occurs:

Mar 29 09:02:31: %SYS-2-NOMEMORY: No memory available for failure to allocate from heap 14
Mar 29 09:02:31: %SYS-2-MALLOCFAIL: Memory allocation of 1500 bytes failed from 0x5954FD0C, alignment 0
Pool: Processor Free: 2593640 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "CMI IOSd task", ipl= 0, pid= 22
-Traceback= 1#00182b81fe0a9f14dcd569d8a3d6a0a5 :54D82000+2865014 :54D82000+47A9048 :54D82000+47B0F50 :54D82000+47CDD14 :54D82000+47A23C4 :54D82000+47CFFBC :54D82000+47F9294 :54D82000+FEF73C :54D82000+FEFCA4 learning:2C0E4000+13A58 learning:2C0E4000+648C
Mar 29 09:02:34: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for messages. No memory available -Process= "Chunk Manager", ipl= 0, pid= 1
-Traceback= 1#00182b81fe0a9f14dcd569d8a3d6a0a5 :54D82000+2865014 :54D82000+47A2B78 :54D82000+47A3118 :54D82000+4783EDC
Mar 29 09:02:45: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for messages. No memory available -Process= "Chunk Manager", ipl= 0, pid= 1
-Traceback= 1#00182b81fe0a9f14dcd569d8a3d6a0a5 :54D82000+2865014 :54D82000+47A2B78 :54D82000+47A3118 :54D82000+4783EDC
%% Low on memory; try again later

%% Low on memory; try again later

%% Low on memory; try again later

%% Low on memory; try again later

Mar 29 09:02:55: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for messages. No memory available -Process= "Chunk Manager", ipl= 0, pid= 1
-Traceback= 1#00182b81fe0a9f14dcd569d8a3d6a0a5 :54D82000+2865014 :54D82000+47A2B78 :54D82000+47A3118 :54D82000+4783EDC
%% Low on memory; try again later

%% Low on memory; try again later

%% Low on memory; try again later

Mar 29 09:03:01: %ENTROPY-0-ENTROPY_ERROR: Unable to collect sufficient entropy
Mar 29 09:03:01: %SYS-2-NOMEMORY: No memory available for failure to allocate from heap 14
Mar 29 09:03:01: %SYS-2-MALLOCFAIL: Memory allocation of 1500 bytes failed from 0x5954FD0C, alignment 0
Pool: Processor Free: 2592580 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "Chunk Manager", ipl= 0, pid= 1
-Traceback= 1#00182b81fe0a9f14dcd569d8a3d6a0a5 :54D82000+2865014 :54D82000+47A9048 :54D82000+47B0F50 :54D82000+47CDD14 :54D82000+47A23C4 :54D82000+47A259C :54D82000+47A3118 :54D82000+4783EDC
Mar 29 09:03:05: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for messages. No memory available -Process= "Chunk Manager", ipl= 0, pid= 1
-Traceback= 1#00182b81fe0a9f14dcd569d8a3d6a0a5 :54D8200

------------------ show version ------------------

Cisco IOS Software, IOS-XE Software, Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 03.07.05E RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2017 by Cisco Systems, Inc.
Compiled Fri 10-Feb-17 07:59 by prod_rel_team

Cisco IOS-XE software, Copyright (c) 2005-2015 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0.
(http://www.gnu.org/licenses/gpl-2.0.html) For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: IOS-XE ROMMON
BOOTLDR: CAT3K_CAA Boot Loader (CAT3K_CAA-HBOOT-M) Version 3.58, RELEASE SOFTWARE (P)

DIST_DC_USB uptime is 20 hours, 26 minutes
Uptime for this control processor is 20 hours, 29 minutes
System returned to ROM by Power Failure at 12:28:04 CO Fri Feb 1 2019
System restarted at 13:14:05 CO Fri Mar 22 2019
System image file is "flash:packages.conf"
Last reload reason: Power Failure

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

License Level: Ipbase
License Type: Permanent
Next reload license Level: Ipbase

cisco WS-C3850-12XS (MIPS) processor with 4194304K bytes of physical memory.
2 Virtual Ethernet interfaces
16 Ten Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
253984K bytes of Crash Files at crashinfo:.
3431232K bytes of Flash at flash:.
0K bytes of Dummy USB Flash at usbflash0:.
0K bytes of at webui:.

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 16 WS-C3850-12XS 03.07.05E cat3k_caa-universalk9 INSTALL

Configuration register is 0x102

------------------ show process memory sorted ------------------

System memory : 3937228K total, 1633944K used, 2303284K free, 222140K kernel reserved
Lowest(b) : 1653156696
PID Text Data Stack Heap RSS Total Process
9338 102396 37768 112 288 246620 350392 iosd
6508 11544 290280 92 36212 232156 611872 fed
9333 26024 566144 88 12944 142808 683340 wcm
7018 4 109452 116 103780 112164 131520 idope.py
6510 836 98832 88 1136 74092 260340 stack-mgr
6509 708 95148 92 852 56784 236336 platform_mgr
7111 124 91936 88 8188 46872 133884 cli_agent
10559 4 83760 224 19268 46720 141452 wnweb_paster.py
7097 5080 77432 96 5532 39468 217344 ffm
7098 1580 181508 88 13428 29876 233428 eicored
7089 56 51248 88 424 23964 103304 pdsd
7095 44 21276 88 164 20856 68756 mgmte_tap
7107 28 17172 88 164 20812 64500 liin_tap
7114 888 117776 88 5312 15268 162368 snmp_subagent
7091 1032 49008 88 2036 14080 88120 osinfo-provider
7082 1148 74212 88 1508 9932 118936 obfld
9334 172 81212 88 1436 9648 123608 table_mgr
9339 252 52356 88 1216 9608 92192 ha_mgr
7096 328 125216 88 292 9188 166384 installer
7124 820 46980 88 132 8408 91268 vman
7087 48 46796 88 584 8216 84920 plogd
7094 60 31372 88 560 7556 74472 mem_mgmt
7090 164 21560 88 296 7052 62776 gold_slave
6237 1440 11052 88 808 7008 44092 system_mgr
7099 268 42532 88 784 6676 75948 dtmgr
9335 100 37944 88 268 6628 74012 cmm
9337 52 38820 88 292 6516 74180 iifd
7086 52 55144 88 296 6356 91432 system_mgr_cli
7100 212 37880 88 164 5944 77672 cpumemd
7088 88 38836 88 296 5896 135564 psdprov
7085 44 33732 88 296 5732 68088 csprovider
7092 180 33708 88 296 5704 66892 oscore_p
7122 36 33704 88 296 5688 66652 sysinfo_p
7117 120 33704 88 296 5632 66740 erm_p
7119 80 33772 88 296 5608 66772 ngdumper_provid
7080 84 33708 88 296 5592 66704 profiled
7083 40 33704 88 296 5528 66636 consoled
6466 140 43660 88 988 5052 60300 slproc
7084 16 13548 88 320 4748 41304 console_relay
7093 32 22372 88 164 4428 50756 netd
10560 20 1008 92 132 4024 25156 xlogger
7079 20 13056 88 164 3676 39588 ns_oir_proxy
9321 20 832 88 132 3280 20436 xlogger
9013 20 832 88 132 3272 20436 xlogger
8940 752 1968 88 1920 3224 4988 rollback_timer.
5575 12 780 88 132 2956 18296 epc_sm_liaison
7128 8 8956 88 164 2496 21528 fs_cmd
9012 752 1960 88 1912 2428 4980 rollback_timer.
7081 752 624 88 576 1896 3644 oom_poll.sh
1 324 452 88 344 1744 4672 init
9184 752 484 88 436 1744 3504 nova_sntp.sh
9336 752 404 88 356 1644 3424 wnweb.sh
4071 372 180 88 132 1376 3284 dbus-daemon
5572 752 108 88 60 1244 3128 sh
4455 24 880 88 784 1172 2612 klogd
4585 24 880 88 784 1172 2612 lockd
4535 220 364 88 132 1060 5996 sshd
4587 220 364 88 132 1060 5996 nfsd
7771 1188 180 88 0 992 7924 libvirtd
9320 752 484 88 436 944 3504 nova_sntp.sh
5005 752 436 88 388 928 3456 sh
4547 144 332 88 132 840 3568 xinetd
4551 144 332 88 132 840 3568 xinetd
4588 144 332 88 132 840 3568 nfsd
4589 144 332 88 132 840 3568 nfsd
9324 60 160 88 132 724 3260 sntp
1161 120 232 88 132 644 2312 udevd
2967 120 168 88 132 624 2248 udevd
2968 120 168 88 132 624 2248 udevd
9017 20 240 88 132 588 2064 inotifywait
5008 20 240 88 132 532 2064 inotifywait
22907 28 160 88 132 464 1896 sleep
4467 16 160 88 132 460 1884 portmap
4586 16 160 88 132 460 1884 nfsd
4599 76 188 88 132 332 2708 rpc.mountd
2 0 0 0 0 0 0 kthreadd

------------------ show buffers ------------------

PDS public buffers

Public buffer pools:
PDS Buffer Pool 0, 2048 bytes (total 2048, permanent 2048, transient 0
peak 1238 @ 1465238):
938 in free list (2048 max allowed)
20937551 hits, 0 misses, 20936441 free hits, 0 free misses
PDS Buffer Pool 1, 12288 bytes (total 341, permanent 341, transient 0):
341 in free list (341 max allowed)
0 hits, 0 misses, 0 free hits, 0 free misses
SM Buf Group TX, 516 bytes (total 32, permanent 32, transient 0
peak 5 @ 11214):
32 in free list (32 max allowed)
36724 hits, 0 misses, 36724 free hits, 0 free misses

Header pools:
Packet Header Memory, 184 bytes (total 4681, permanent 4681, transient 0
peak 146 @ 1465238):
4663 in free list (4681 max allowed)
28519699 hits, 0 misses, 28519681 free hits, 0 free misses
Buffer Header Memory, 88 bytes (total 16384, permanent 16384, transient 0
peak 1238 @ 1465238):
15274 in free list (16384 max allowed)
20974280 hits, 0 misses, 20973170 free hits, 0 free misses
Header pools:

Header pools:
Header pools:
Packet Header Memory, 184 bytes (total 4681, permanent 4681, transient 0
peak 146 @ 1465238):
4663 in free list (4681 max allowed)
28519702 hits, 0 misses, 28519684 free hits, 0 free misses
Buffer Header Memory, 88 bytes (total 16384, permanent 16384, transient 0
peak 1238 @ 1465238):
15274 in free list (16384 max allowed)
20974281 hits, 0 misses, 20973171 free hits, 0 free misses

IOSd private buffers:

Buffer elements:
2018 in free list
33992964 hits, 0 misses, 1242 created

Public buffer pools:
Small buffers, 104 bytes (total 600, permanent 600):
600 in free list (100 min, 1250 max allowed)
12157968 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Middle buffers, 600 bytes (total 450, permanent 450):
450 in free list (50 min, 1000 max allowed)
30630 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Big buffers, 1536 bytes (total 450, permanent 450, peak 451 @ 20:27:24):
450 in free list (25 min, 1500 max allowed)
82459 hits, 0 misses, 1 trims, 1 created
0 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 10, permanent 10, peak 11 @ 20:27:24):
10 in free list (0 min, 300 max allowed)
0 hits, 0 misses, 1 trims, 1 created
0 failures (0 no memory)
Large buffers, 5024 bytes (total 10, permanent 10, peak 11 @ 20:27:24):
10 in free list (0 min, 30 max allowed)
0 hits, 0 misses, 1 trims, 1 created
0 failures (0 no memory)
VeryLarge buffers, 8232 bytes (total 50, permanent 50):
50 in free list (0 min, 150 max allowed)
0 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Huge buffers, 18024 bytes (total 5, permanent 5, peak 6 @ 20:27:24):
5 in free list (0 min, 13 max allowed)
0 hits, 0 misses, 1 trims, 1 created
0 failures (0 no memory)

Interface buffer pools:
ng3k_rx32 buffers, 1 bytes (total 1, permanent 1):
1 in free list (0 min, 1 max allowed)
0 hits, 0 misses
ng3k_rx21 buffers, 5 bytes (total 5, permanent 5):
5 in free list (0 min, 5 max allowed)
0 hits, 0 misses
CF Small buffers, 104 bytes (total 101, permanent 100, peak 101 @ 20:27:35):
101 in free list (100 min, 200 max allowed)
0 hits, 0 misses, 1227 trims, 1228 created
0 failures (0 no memory)
BIPC small buffers, 128 bytes (total 251, permanent 250, peak 251 @ 20:27:36):
251 in free list (250 min, 250 max allowed)
23 hits, 0 misses, 1227 trims, 1228 created
0 failures (0 no memory)
Generic ED Pool buffers, 512 bytes (total 101, permanent 100, peak 101 @ 20:27:21):
101 in free list (100 min, 100 max allowed)
0 hits, 0 misses
BIPC middle buffers, 600 bytes (total 301, permanent 300, peak 301 @ 20:27:36):
301 in free list (300 min, 300 max allowed)
154 hits, 0 misses, 1227 trims, 1228 created
0 failures (0 no memory)
CF Middle buffers, 600 bytes (total 101, permanent 100, peak 101 @ 20:27:35):
101 in free list (100 min, 200 max allowed)
0 hits, 0 misses, 1227 trims, 1228 created
0 failures (0 no memory)
Syslog ED Pool buffers, 600 bytes (total 133, permanent 132, peak 133 @ 20:27:21):
101 in free list (132 min, 132 max allowed)
1164 hits, 0 misses
CF Big buffers, 1536 bytes (total 26, permanent 25, peak 26 @ 20:27:35):
26 in free list (25 min, 50 max allowed)
0 hits, 0 misses, 1227 trims, 1228 created
0 failures (0 no memory)
ng3k_rx6 buffers, 1600 bytes (total 512, permanent 512):
512 in free list (0 min, 512 max allowed)
0 hits, 0 misses
ng3k_rx7 buffers, 1600 bytes (total 512, permanent 512):
512 in free list (0 min, 512 max allowed)
0 hits, 0 misses
ng3k_rx8 buffers, 1600 bytes (total 512, permanent 512):
512 in free list (0 min, 512 max allowed)
0 hits, 0 misses
ng3k_rx9 buffers, 1600 bytes (total 512, permanent 512):
512 in free list (0 min, 512 max allowed)
0 hits, 0 misses
ng3k_rx10 buffers, 1600 bytes (total 96, permanent 96):
96 in free list (0 min, 96 max allowed)
0 hits, 0 misses
ng3k_rx11 buffers, 1600 bytes (total 96, permanent 96):
96 in free list (0 min, 96 max allowed)
0 hits, 0 misses
ng3k_rx12 buffers, 1600 bytes (total 96, permanent 96):
96 in free list (0 min, 96 max allowed)
0 hits, 0 misses
ng3k_rx13 buffers, 1600 bytes (total 96, permanent 96):
96 in free list (0 min, 96 max allowed)
0 hits, 0 misses
ng3k_rx14 buffers, 1600 bytes (total 96, permanent 96):
96 in free list (0 min, 96 max allowed)
0 hits, 0 misses
ng3k_rx15 buffers, 1600 bytes (total 96, permanent 96):
96 in free list (0 min, 96 max allowed)
0 hits, 0 misses
ng3k_rx17 buffers, 1600 bytes (total 768, permanent 768):
768 in free list (0 min, 768 max allowed)
5834739 hits, 0 misses
ng3k_rx18 buffers, 1600 bytes (total 512, permanent 512):
512 in free list (0 min, 512 max allowed)
80478 hits, 0 misses
ng3k_rx19 buffers, 1600 bytes (total 1024, permanent 1024):
1024 in free list (0 min, 1024 max allowed)
2760661 hits, 0 misses
ng3k_rx20 buffers, 1600 bytes (total 96, permanent 96):
96 in free list (0 min, 96 max allowed)
0 hits, 0 misses
ng3k_rx22 buffers, 1600 bytes (total 16, permanent 16):
16 in free list (0 min, 16 max allowed)
0 hits, 0 misses
ng3k_rx23 buffers, 1600 bytes (total 16, permanent 16):
16 in free list (0 min, 16 max allowed)
0 hits, 0 misses
ng3k_rx24 buffers, 1600 bytes (total 16, permanent 16):
16 in free list (0 min, 16 max allowed)
0 hits, 0 misses
ng3k_rx25 buffers, 1600 bytes (total 16, permanent 16):
16 in free list (0 min, 16 max allowed)
0 hits, 0 misses
ng3k_rx26 buffers, 1600 bytes (total 16, permanent 16):
16 in free list (0 min, 16 max allowed)
0 hits, 0 misses
ng3k_rx27 buffers, 1600 bytes (total 512, permanent 512):
512 in free list (0 min, 512 max allowed)
547145 hits, 0 misses
ng3k_rx28 buffers, 1600 bytes (total 512, permanent 512):
512 in free list (0 min, 512 max allowed)
5578 hits, 0 misses
ng3k_rx29 buffers, 1600 bytes (total 2048, permanent 2048):
2047 in free list (0 min, 2048 max allowed)
4057 hits, 0 misses
ng3k punt fallback pool buffers, 1600 bytes (total 1500, permanent 1500):
1500 in free list (0 min, 0 max allowed)
0 hits, 0 misses
ng3k_rx33 buffers, 1600 bytes (total 128, permanent 128):
128 in free list (0 min, 128 max allowed)
0 hits, 0 misses
ng3k_rx34 buffers, 1600 bytes (total 128, permanent 128):
128 in free list (0 min, 128 max allowed)
0 hits, 0 misses
ng3k_rx35 buffers, 1600 bytes (total 96, permanent 96):
96 in free list (0 min, 96 max allowed)
0 hits, 0 misses
ng3k_rx36 buffers, 1600 bytes (total 512, permanent 512):
512 in free list (0 min, 512 max allowed)
0 hits, 0 misses
ng3k_rx37 buffers, 1600 bytes (total 512, permanent 512):
512 in free list (0 min, 512 max allowed)
0 hits, 0 misses
ng3k_rx38 buffers, 1600 bytes (total 512, permanent 512):
512 in free list (0 min, 512 max allowed)
0 hits, 0 misses
ng3k_rx40 buffers, 1600 bytes (total 96, permanent 96):
96 in free list (0 min, 96 max allowed)
0 hits, 0 misses
ng3k_rx41 buffers, 1600 bytes (total 512, permanent 512):
512 in free list (0 min, 512 max allowed)
2 hits, 0 misses
ng3k_rx39 buffers, 2048 bytes (total 1024, permanent 1024):
1024 in free list (0 min, 1024 max allowed)
0 hits, 0 misses
BIPC buffers, 4096 bytes (total 2, permanent 2):
2 in free list (1 min, 8 max allowed)
2 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
CF VeryBig buffers, 4520 bytes (total 3, permanent 2, peak 3 @ 20:27:35):
3 in free list (2 min, 4 max allowed)
0 hits, 0 misses, 1227 trims, 1228 created
0 failures (0 no memory)
CF Large buffers, 5024 bytes (total 2, permanent 1, peak 2 @ 20:27:35):
2 in free list (1 min, 2 max allowed)
0 hits, 0 misses, 7 trims, 8 created
0 failures (0 no memory)
ng3k punt jumbo pool buffers, 9300 bytes (total 128, permanent 128):
128 in free list (0 min, 0 max allowed)
0 hits, 0 misses
BIPC Medium buffers, 16384 bytes (total 6, permanent 5, peak 6 @ 20:27:36):
6 in free list (5 min, 5 max allowed)
2 hits, 0 misses, 1227 trims, 1228 created
0 failures (0 no memory)
BIPC Large buffers, 65535 bytes (total 3, permanent 2, peak 3 @ 20:27:36):
3 in free list (2 min, 2 max allowed)
0 hits, 0 misses, 1227 trims, 1228 created
0 failures (0 no memory)
IPC small buffers, 128 bytes (total 1200, permanent 1200):
1198 in free list (400 min, 1200 max allowed)
2 hits, 0 fallbacks, 0 trims, 0 created
0 failures (0 no memory)
0 max cache size, 0 in cache
0 hits in cache, 0 misses in cache
IPC middle buffers, 600 bytes (total 1200, permanent 1200):
1200 in free list (400 min, 1200 max allowed)
0 hits, 0 fallbacks, 0 trims, 0 created
0 failures (0 no memory)
0 max cache size, 0 in cache
0 hits in cache, 0 misses in cache
IPC Emergency buffers, 4096 bytes (total 301, permanent 300, peak 301 @ 20:27:36):
301 in free list (300 min, 300 max allowed)
0 hits, 0 misses, 1227 trims, 1228 created
0 failures (0 no memory)
0 max cache size, 0 in cache
0 hits in cache, 0 misses in cache
IPC buffers, 4096 bytes (total 1200, permanent 1200):
1199 in free list (400 min, 1200 max allowed)
1 hits, 0 fallbacks, 0 trims, 0 created
0 failures (0 no memory)
0 max cache size, 0 in cache
0 hits in cache, 0 misses in cache
IPC Medium buffers, 16384 bytes (total 40, permanent 40):
40 in free list (20 min, 40 max allowed)
0 hits, 0 fallbacks, 0 trims, 0 created
0 failures (0 no memory)
0 max cache size, 0 in cache
0 hits in cache, 0 misses in cache
Private Huge IPC buffers, 18024 bytes (total 1, permanent 0, peak 1 @ 20:27:36):
1 in free list (0 min, 4 max allowed)
0 hits, 0 misses, 1227 trims, 1228 created
0 failures (0 no memory)
Private Huge buffers, 65280 bytes (total 1, permanent 0, peak 1 @ 20:27:36):
1 in free list (0 min, 4 max allowed)
0 hits, 0 misses, 1227 trims, 1228 created
0 failures (0 no memory)
IPC Large buffers, 65535 bytes (total 104, permanent 104, peak 105 @ 20:27:36):
104 in free list (20 min, 104 max allowed)
0 hits, 0 misses, 1 trims, 1 created
0 failures (0 no memory)
0 max cache size, 0 in cache
0 hits in cache, 0 misses in cache

Header pools:
Header buffers, 0 bytes (total 267, permanent 256, peak 267 @ 20:25:35):
11 in free list (10 min, 512 max allowed)
253 hits, 3 misses, 9807 trims, 9818 created
0 failures (0 no memory)
256 max cache size, 256 in cache
0 hits in cache, 0 misses in cache

Particle Clones:
1024 clones, 0 hits, 0 misses

Public particle pools:
F/S buffers, 256 bytes (total 384, permanent 384):
128 in free list (128 min, 1024 max allowed)
256 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
256 max cache size, 256 in cache
0 hits in cache, 0 misses in cache
Normal buffers, 512 bytes (total 512, permanent 512):
384 in free list (128 min, 1024 max allowed)
128 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
128 max cache size, 128 in cache
0 hits in cache, 0 misses in cache

[fasanchezv]

Yes Richard, I agree with you, memory leak is the real problem, discusion about cpu was initiated because we wanted to clear if mac flapping affected the cpu or memory, and if mac flapping could be the root cause of memory leak, but the problem to solve here is the switch`s high memory utilization.

look at this:

Fed process has increased dramatically in the lap of three days:

first register:

DIST_DC_USB#show processes memory sorted
System memory : 3937228K total, 1637932K used, 2299296K free, 222140K kernel reserved
Lowest(b) : 1656821584
PID Text Data Stack Heap RSS Total Process
9343 102396 37768 112 288 245764 350392 iosd
6511 11544 303380 92 20640 245360 624972 fed
9338 26024 565236 88 8964 142372 682432 wcm
7021 4 109452 116 103780 112164 131520 idope.py
6513 836 103952 88 1136 73700 265460 stack-mgr

three days later register:

DIST_DC_USB#sh processes memory sorted
System memory : 3937228K total, 2391540K used, 1545688K free, 222140K kernel reserved
Lowest(b) : 876961032
PID Text Data Stack Heap RSS Total Process
6511 11544 1002172 92 57928 943464 1323764 fed
9343 102396 103312 112 288 290296 415936 iosd
9338 26024 566260 88 8964 142800 683456 wcm
7021 4 109452 116 103780 112164 131520 idope.py

[Richard Burts]

Thanks for the update. It is interesting to see how the memory for fed has increased. If you have a support contract with Cisco I would suggest opening a case with Cisco TAC and have them help identify what is causing this.

HTH

Rick

[fasanchezv · Accepted solution]

Thanks to everyone who replay to this post, finally the solution was given by using the command:

ipv6 mld snooping

this command enable the MLD snooping, so, a per-VLAN IPv6 multicast address table is constructed in software and hardware. Switch then performs IPv6 multicast-address based bridging in hardware, which prevents these packets to be processed by software.