Catalyst 3850 High Memory Used

[jluis_alves]

Hi,

We have some catalyst 3850 switchs with more than 80% Memory usage.

Looking on memory allocated we can see that arp input have highest memory usage.

show processes memory sorted allocated
Processor Pool Total: 886455648 Used: 364614224 Free: 521841424
lsmpi_io Pool Total: 6295128 Used: 6294296 Free: 832

PID TTY Allocated Freed Holding Getbufs Retbufs Process
35 0 219477763952 219476780384 1037120 0 0 ARP Input
527 0 103328177000 111161034504 123504 0 0 SNMP ENGINE
526 0 60720782024 53033593312 119096 0 0 PDU DISPATCHER
10 0 43242937440 43280730664 1684264 41208919058 41212352574 Pool Manager
520 0 28481858096 28481857952 69144 0 0 mdns Timer Proce
191 0 17112531160 7484608760 89464 0 0 CDP Protocol
203 0 15545194768 9467824 72208 61600 0 IP Input
0 0 11601240976 11249616384 5052280 17486679 0 *Dead*
117 0 10562472296 10241322360 636536 1474704 0 IOSXE-RP Punt Se
36 0 10348925360 10349181064 45000 0 0 ARP Background

Is this behavior normal?

[Reza Sharifi]

Hi,

No, this is not normal. Although the 3850 series run high on memory, the average should be between 40 to 50% and not higher. So, having it at 80% is maybe because an IOS bug in the version you are running.

I would open a ticket with TAC and have them analyze and possibly recommend an upgrade.

HTH

[Leo Laohoo]

What firmware is currently being used?

[jluis_alves · Accepted solution]

Hi,

There is a bug on this firmware - 16.3.2

Due support recomendation we will upgrade to version - 16.3.5b

Thanks.

[Leo Laohoo]

---

@jluis_alves wrote:

Due support recomendation we will upgrade to version - 16.3.5b

---

That is not a good recommendation from TAC.  16.3.6S is the way to go.  

[takats_nandor]

Hello Leo,

Where did you see the cisco recomends 16.3.6S. Right now I`m having issues on 16.3.5b. 

Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor FF9FAE7010 885692992 343531196 542161796 539350872 541706148
lsmpi_io FF9F2A81A8 6295128 6294304 824 824 412

Thanks a lot,

Nandor

[Leo Laohoo]

---

@takats_nandor wrote:

Where did you see the cisco recomends 16.3.6S. Right now I`m having issues on 16.3.5b. 

---

That response was way back 2018.  

I'd look into 16.6.6.

[Roman_Shilenko]

Looks like 16.3.5b has this bug too:

EdgeRouter#sh ver
Cisco IOS Software [Denali], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.3.5b, RELEASE SOFTWARE (fc1)

EdgeRouter#sh mem
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor FF9F370010 885692992 341081240 544611752 533380304 536595852
lsmpi_io FF9EB311A8 6295128 6294304 824 824 412

[FilipHansen]

Downgrading / upgrading to 16.3.5b did the trick for me. It was also what was recommended on the Cisco support download page. I think now its 16.3.6 which has the recommendation and personally I would go for that. It is as previously mentioned a bug. It is supposedly to have been solved but... still

If u can find the caveat for it there is also a temp work around if I remember correctly

What does your logs say btw u should see something there to? It also helps to turn of vstack if that is not in use

[Roman_Shilenko]

After reading multiple articles, including one from a Cisco engineer, I've changed my mind. It looks like having lsmpi_io memory utilization at 99.9% is totally normal as it is allocated only once at the moment of reboot. More than that, too much free lsmpi_io memory can be a sign of a problem.

But my switch isn't feeling well and its log is full of messages like this:

087561: Aug  9 19:03:41 PST: %PLATFORM-3-ELEMENT_CRITICAL:Switch 1 R0/0: smand:  1/RP/0: Used Memory value 96% exceeds critical level 95%

I think I have no other option but to upgrade to 16.3.6.

[jasonm002]

Don't go to 16.3.6, go to 16.3.7. 16.3.6 has catastrophic bug CSCvj49476.

And also be aware that 16.x enables device tracking if you have DHCP snooping applied, so if you are doing that you will want to create device-tracking profiles and apply them to at least your trunk links so ARP and NDP from every switch in the network isn't snooped.

Also 16.x enables control plane policing (copp) by default so you will want to be aware of this as well.

[Roman_Shilenko]

Thank you for the information, jasonm002 !

[Zolboo92]

How can i download this version?

[jasonm002]

Well now this post is extremely old so 16.3.7 is no longer the thing to use. You should be fine at this point with whatever the recommended (i.e. "Gold Star") release is for your switch on https://software.cisco.com/download/home