Catalyst 2950 high memory utilization problem

[toma.nikolov96]

I have a problem with a Cisco Catalyst 2950 (WS-C2950G-24-EI)
I am observing throu our management system(Solarwinds), a constant growth of the memory utilization of the device. It reached arround 96-97% and then dropped down to a state of 36% by now. 

I managed to observe that when the device was with the highest level of memory utilization it had several proccesses using the memory(found after the show proccesses memory command): 
37162572 bytes used by the Pool Manager, 10605020 bytes IP Multicast Dat and 682316  bytes for Net Input proccesses. 

After returning to it's normal memory usage state, without any change of configuration or restarting the device, these processes usage of memory were nearly lowerred to zero and eventually the devices overal memory utilization dropped down to 36%. 

The swithc is used as an access swithch and all of the ports are configured as follows: 

interface FastEthernet0/XX
description XXXX
switchport access vlan XXX
switchport mode access
no ip address
mls qos trust cos
spanning-tree portfast

Here is the other configuration of the device: 

ip domain-name xxxx
ip name-server 1xxxx
ip dhcp-server 1xxxx
vtp mode transparent
!
spanning-tree mode pvst
no spanning-tree optimize bpdu transmission
spanning-tree extend system-id
!
!
vlan 1xxxx
name Mxxxx
!
vlan 1xxxx
name Hxxxx

interface GigabitEthernet0/xx
description xxxxxx
switchport mode trunk
no ip address
!
interface GigabitEthernet0/xx
no ip address
!
interface Vlan1xxxx
ip address 1xxxx 2xxxx
no ip route-cache
shutdown
!
interface Vlan1xxx
description xxxxx
ip address 1xxxx 2xxxxx
no ip route-cache
!
ip default-gateway 1xxxxxx
ip http server
ip http authentication local
!
logging source-interface Vlan1xxxxx
logging 1xxxx
access-list 1 permit 1xxxxx
access-list 1 permit 1xxxxx
snmp-server engineID local xxxxxxx
snmp-server community xxxxx
snmp-server location xxxx
snmp-server contact xxxx

 
Can please someone advice me with an explanation about these proccesses and the reason for their high level of memowy usage?

Thank you in advance!

[Diana Karolina Rojas]

Hello good morning!

Can you provide the output of this command: show process cpu sorted?

Thanks,

[Mark Malone]

As Diana said you need to collect the show proc cpu sorted as its occuring to see whihc process was causing it as its live

did you see anything in logs like spanning gtree alerts or cpu hog etc , or anything else unusual ?

These switches are old i cant remember if they support eem but if they do you could use this script to automatically capture the output as its high when it reaches pover 70% save you having to monitor it for the issue

event manager applet High_CPU

    event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.4.1 get-type exact entry-op ge entry-val "70" exit-time 10 poll-interval 5

    action 0.1 syslog msg "CPU Utilization is high"

    action 0.2 cli command "enable"

    action 0.4 cli command "show log | append flash:CPU_Profile.txt"

    action 0.5 cli command "show process cpu sorted | append flash:CPU_Profile.txt"

    action 0.6 cli command "show interfaces | append flash:CPU_Profile.txt"

    action 0.7 cli command " show ip cef switching stat | append flash:CPU_Profile.txt"

    action 0.8 cli command " show ip traffic | append flash:CPU_Profile.txt"

    action 0.9 cli command " show int switching | append flash:CPU_Profile.txt"

    action 1.0 cli command "no event manager applet High_CPU"

[Julio E. Moisa]

Hi

Please provide the output requested previously by Diana, I have had experiences where the CPU is high but the processes are normal, in this case we have detected some frames passing through undefinely through the switches, the solution is block the mac address to drop the packets.

You could use the following debug, (note: you should use it few seconds, 3 or 5 seconds, also you could delete the logs and increase the buffer) 

debug platform cpu-queues software-fwd-q

to disable

u all (press enter)

It is not common but also take in consideration the switch 2950 was manufactured a long time ago and a bug could arise. 

Hope it is useful

:-)

[toma.nikolov96]

Greetings,  

Actually the show proccesses cpu sorted command is not available - anly show cpu proccess  from which the ooutput is attached to the message.

Thank youi for the support. 

[Mark Malone]

for that for yourself you can use this command then to shorten it when trying to see cpu hitters if proc sorted is unavailable

sh proc cpu | ex 0.00

anywat thats a clean output in cpu if it was taken during the issue as it needs to be captured real time ,  so you may have triggered a bug , very old software running on it

You may have hit a memory leak like below in the pool manager , when softwrae gets older its a lot easier for it too happen , reboot will clear it , an upgarde should fix it , thenot uncommon ihit oine in an 4331 router with  new ios on it the other day , upgraded the software its fixed now

The other process is for multicast tafffic in memory

Memory leak in Pool Manager process

CSCtg23115

Description

Symptoms: A memory leak occurs in the Pool Manager process.

Conditions: This symptom is observed with Cisco IOS Release 12.4(24)T. It is
not yet known what other IOS releases may be affected. This symptom is seen
with multicast enabled.

Workaround: There is no workaround to prevent the leak. Once the memory has
leaked, reloading the router will temporarily free the memory.

[Julio E. Moisa]

Hi

It looks like a bug, check the following image, usually the first field (5%) should be always higher than the second field (0%), now if you have the inverse, it could represent a problem as I mentioned previously. Now should be good take a screenshot once the behavior is present, probably a specific process is generating high CPU.

[toma.nikolov96]

Thank you Julio. 

This is verry helpfull for me. 
I will wait for the problem to occur again and I will troubleshoot it during the high utilization time. 
I managed to observe that the memory has dropped suddenly to its normaml condition of utilization and for the last couple of days it begins again to increase the utilization slowly. 

Kind regards, 

[SergeyTimofeev8340]

I had this problem too. On all of 2950. Switches had a lot of vlans that it got by VTP. I heard this model can't work with more than 70 vlans. Actually, I had issue with deleting vlans on one of my 2950 switches with VTP mode transparent and then I found info - no more 70 vlans or you will get FFFFed up vlan.dat and there will be high processor memory utilization. OK, I deleted vlan.dat, rebooted and then I created only useful vlans (about 5) and processor memory utilization became admissible - something about 65 percent. On other switches there was no problem with deleting so all that was needed was deleting uneuseful vlans