High memory utilization of 3560 switch after upgrade the IOS

[Abdo Ahmed Saleh Al-Kaf]

Hi All,

I upgrade the IOS of my switch 3560 from   C3560-IPBASE-M), Version 12.2(25)SEB4,  To   C3560-IPBASEK9-M), Version 12.2(55)SE5

after that the utilizatio of the memory increase to 80%

I attached the output of show memory statistics history

We need help to resolve this issue.

Thanks

Adullah

[InayathUlla Sharieff]

Hi Abdo,

Please provide the following outputs from the device:

- Show tech

--show clock

--show process memory sorted

--show process memory summary

--show proc memory >>>>POrocess ID which you see occupied more process.

--show memory allocating-process totals

--show memory dead /show memory dead total

--show memory dead totals

--show alignment

--show buffer summary

Regards

Inayath.

[Abdo Ahmed Saleh Al-Kaf]

Hi Inayath,

I took the output of the commands that you asked.

I didn't find the attach file there is only for photo, video and links, So I upload it on the mean post, please check the file Show Commands.

Thanks

Abdo

[InayathUlla Sharieff]

Hi Abdo,

Thanks for the data. I analysed the attach file but could nt found any issue with the memory as of now.

------------------ show memory statistics ------------------

                Head    Total(b)     Used(b)     Free(b)   Lowest(b)  Largest(b)

Processor    3D9F690    78231664    18948552    59283112    44697888    19762476

      I/O    6800000     8388608     3599156     4789452     4620800     4786104

Driver te    2C00000     1048576          44     1048532     1048532     1048532

As per the memory showing above the total memory on the device is 78 MB and the

used is 18 MB , which is left us with 59 MB which is quite good enough.

#############################################################################

2-

0   0   27607808    6876512   17632684          0          0 *Init*

The command above showing that most of the memory is consumed by the *Init*

process , this process is responsible of holding a memory for the IOS image after

Extracting it , and its keep  its value (doesn’t increase) .

Not quite sure what process was holding the memory when you saw the memory utilization high.

3-

I check the buffer pool and it looks good to me. And in the logs there is no msg displaying about the memory .

4-Please find some detail on memory below:

There are two types of memory in IOS: process memory and IO memory.

When a feature is enable on an IOS device (e.g. PIM, HSRP, and etc), IOS allocates process memory for the process.

IO memory is used when a software switched traffic hits the CPU. The CPU allocates IO memory to store the frame temporarily.

Troubleshooting Methodology

Introduction

There are two main reasons why a switch runs out of memory.

1. A process does not free up the memory after the memory is no longer needed. This is generally known as memory leak.

2. A process does not limit the amount of memory it allocates. Eventually, the process uses up all process memory on the switch.

The goal is to identify the misbehaving process. There are two approaches.

Show process memory debug leak  <<<<< Only to be run during MW or when the device is not in production .

This command is handy to identify any process leaking memory. However, it does not help if the memory problem is caused by a process grabs all memory (i.e. problem #2 above).

HTH

Regards

Inayath

*Plz rate if this is helpfull.

[Abdo Ahmed Saleh Al-Kaf]

Hi Inayath,

I appreciate your help, I did not understand what do you mean by:

Show process memory debug leak  <<<<< Only to be run during MW or when the device is not in production .

???

Thanks

[InayathUlla Sharieff]

Hi Ahmed,

Sorry the cmd is: show memroy debug leaks this is process intensive command hence suggested not to run during live environment.

More about the same:

To display detected memory leaks, use the show memory debug leaks command in privileged EXEC mode.

Cisco IOS software

show memory debug leaks [chunks | largest | lowmem | summary]

Cisco Catalyst 4500e Series Switches running IOS XE software

show memory debug leak

Syntax Description

chunks	(Optional) Displays the memory leaks in chunks.
largest	(Optional) Displays the top ten leaking allocator_pcs based on size, and the total amount of memory they have leaked.
lowmem	(Optional) Forces the memory leak detector to work in low memory mode, making no memory allocations.
summary	(Optional) Reports summarized memory leaks based on allocator_pc and size of the memory block.

Command Modes

Privileged EXEC

Command History

Release	Modification
12.3(8)T1	This command was introduced.
12.2(25)S	This command was integrated into Cisco IOS Release 12.2(25)S.
12.2(33)SRA	This command was integrated into Cisco IOS Release 12.2(33)SRA.
Cisco IOS XE Release 3.1.0.SG	This command was introduced on the Cisco Catalyst 4500e Serfies Switches to display per-process memory leak ammounts.

Usage Guidelines

If no optional keywords are specified, the show memory debug leaks command invokes normal mode memory leak detection and does not look for memory leaks in chunks.

The show memory debug leaks chunks command invokes normal mode memory leak detection and looks for leaks in chunks as well.

The show memory debug leaks largest command displays the top ten leaking allocator_pcs and the total amount of memory that they have leaked. Additionally, each time this command is invoked it remembers the previous invocation's report and compares it to the current invocation's report. If there are new entries in the current report they are tagged as "inconclusive." If the same entry appears in the previous invocation's report and the current invocation's report, the inconclusive tag is not added. It would be beneficial to run memory leak detection more than once and to consider only the consistently reported leaks.

The show memory debug leaks lowmem command forces memory leak detection to work in low memory mode. The amount of time taken for analysis is considerably greater than that of normal mode. The output for this command is similar to the show memory debug leaks command. You can use this command when you already know that normal mode memory leak detection will fail (perhaps by an unsuccessful previous attempt to invoke normal mode memory leak detection).

The show memory debug leaks summary command reports memory leaks based on allocator_pc and then on the size of the block.

---

Note All show memory debug commands must be used on customer networks only to diagnose the router for memory leaks when memory depletion is observed. These CLI's will have high CPU utilization and might result in time sensitive protocols to flap. These CLI's are recommended for customer use, only in the maintenance window when the router is not in a scaled condition.

---

---

Note All memory leak detection commands invoke normal mode memory leak detection, except when the low memory option is specifically invoked by use of the lowmem keyword. In normal mode, if memory leak detection determines that there is insufficient memory to proceed in normal mode, it will display an appropriate message and switch to low memory mode.

HTH

Regards

Inayath

*Plz rate the usefull posts.