06-15-2016 08:07 AM - edited 03-08-2019 06:13 AM
We have deployed 35 Catalyst 3850 24 and 48 port switches as Layer 2 IDF closet switches, and we are monitoring with a standard tool. We received alerts from two of the 3850-48p switches that memory went over 90%. When I checked them with command "show proc memory" everything looked fine with 2.4GB available. I did a little more searching and found our tool was actually checking the memory under "show mem det proc iosd":
<switch>#sh mem det process iosd
System memory : 3930840K total, 1508124K used, 2422716K free, 222004K kernel reserved
Lowest(b) : 1915232736
Process iosd, type L, PID = 8576
338932K total, 72368K text, 34372K data, 104K stack, 288K heap
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 32A00000 536870912 483406000 53464912 52336864 52060720
IOS Proce 52A00008 16777216 8920180 7857036 7745736 6619672
<SNIP>
483406000 div 536870912 = .9004 or 90.04%
I checked the other switches and most had this memory allocation around or under 27%, only a couple of switch stacks were at 29%. We decided to reboot the two switch and memory went back down under 27%. However, just two weeks later, both of the switches are showing the memory is increasing, and one switch went back over 30% utilization:
<switch>#show mem det proce iosd
System memory : 3930840K total, 1420868K used, 2509972K free, 222004K kernel reserved
Lowest(b) : 2008849460
Process iosd, type L, PID = 8569
337904K total, 72368K text, 33348K data, 104K stack, 288K heap
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 32C00000 536870912 178395428 358475484 357196608 357332804
IOS Proce 52C00008 16777216 8882996 7894220 7813456 6693400
<SNIP>
178395428 div 536870912 = .3323 or 33.23%
I was thinking a complete power off and power on of the two switches in case it was some sort of transient error. Is there anything else I should check for, or should I open a ticket with TAC?
Thank you.
06-15-2016 08:15 AM
Hi,
Powering off the switches or rebooting them is not going to be a permanent solution as, the memory will leak again and the same thing will happen again. I recommend you open a ticket with TAC, as this maybe a bug in the IOS you are running. Most likely they will recommend an IOS upgrade.
HTH
06-15-2016 11:25 PM
Yes, reach TAC.
3.6.5E is the code you should go with if you are running anything below. In fact 3.6.4E fixed few memory related bugs and I am running that code for more than 12 months without major issues.
HTH
Rasika
06-20-2016 06:45 AM
Thanks guys. Yeah, we upgrade a while back to version 03.03.05SE for a vulnerability. Everything was fine until I found these two.
06-20-2016 07:21 AM
Joe,
This version (Version 03.07.02.E) is also pretty good. At the end of the day, its all about the feature set you are running but it strange to see 2 switches out of 35 with issues.
HTH
06-15-2016 06:18 PM
What code are you running?
06-20-2016 06:36 AM
Running version 03.03.05SE
Sorry for the delay, got side tracked.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide