08-22-2017
06:41 AM
- last edited on
03-25-2019
04:45 PM
by
ciscomoderator
I have a problem with a Cisco Catalyst 2950 (WS-C2950G-24-EI)
I am observing throu our management system(Solarwinds), a constant growth of the memory utilization of the device. It reached arround 96-97% and then dropped down to a state of 36% by now.
I managed to observe that when the device was with the highest level of memory utilization it had several proccesses using the memory(found after the show proccesses memory command):
37162572 bytes used by the Pool Manager, 10605020 bytes IP Multicast Dat and 682316 bytes for Net Input proccesses.
After returning to it's normal memory usage state, without any change of configuration or restarting the device, these processes usage of memory were nearly lowerred to zero and eventually the devices overal memory utilization dropped down to 36%.
The swithc is used as an access swithch and all of the ports are configured as follows:
interface FastEthernet0/XX
description XXXX
switchport access vlan XXX
switchport mode access
no ip address
mls qos trust cos
spanning-tree portfast
Here is the other configuration of the device:
ip domain-name xxxx
ip name-server 1xxxx
ip dhcp-server 1xxxx
vtp mode transparent
!
spanning-tree mode pvst
no spanning-tree optimize bpdu transmission
spanning-tree extend system-id
!
!
vlan 1xxxx
name Mxxxx
!
vlan 1xxxx
name Hxxxx
interface GigabitEthernet0/xx
description xxxxxx
switchport mode trunk
no ip address
!
interface GigabitEthernet0/xx
no ip address
!
interface Vlan1xxxx
ip address 1xxxx 2xxxx
no ip route-cache
shutdown
!
interface Vlan1xxx
description xxxxx
ip address 1xxxx 2xxxxx
no ip route-cache
!
ip default-gateway 1xxxxxx
ip http server
ip http authentication local
!
logging source-interface Vlan1xxxxx
logging 1xxxx
access-list 1 permit 1xxxxx
access-list 1 permit 1xxxxx
snmp-server engineID local xxxxxxx
snmp-server community xxxxx
snmp-server location xxxx
snmp-server contact xxxx
Can please someone advice me with an explanation about these proccesses and the reason for their high level of memowy usage?
Thank you in advance!
08-22-2017 07:58 AM
Hello good morning!
Can you provide the output of this command: show process cpu sorted?
Thanks,
08-22-2017 08:12 AM
As Diana said you need to collect the show proc cpu sorted as its occuring to see whihc process was causing it as its live
did you see anything in logs like spanning gtree alerts or cpu hog etc , or anything else unusual ?
These switches are old i cant remember if they support eem but if they do you could use this script to automatically capture the output as its high when it reaches pover 70% save you having to monitor it for the issue
event manager applet High_CPU
event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.4.1 get-type exact entry-op ge entry-val "70" exit-time 10 poll-interval 5
action 0.1 syslog msg "CPU Utilization is high"
action 0.2 cli command "enable"
action 0.4 cli command "show log | append flash:CPU_Profile.txt"
action 0.5 cli command "show process cpu sorted | append flash:CPU_Profile.txt"
action 0.6 cli command "show interfaces | append flash:CPU_Profile.txt"
action 0.7 cli command " show ip cef switching stat | append flash:CPU_Profile.txt"
action 0.8 cli command " show ip traffic | append flash:CPU_Profile.txt"
action 0.9 cli command " show int switching | append flash:CPU_Profile.txt"
action 1.0 cli command "no event manager applet High_CPU"
08-22-2017 01:57 PM - edited 08-22-2017 01:59 PM
Hi
Please provide the output requested previously by Diana, I have had experiences where the CPU is high but the processes are normal, in this case we have detected some frames passing through undefinely through the switches, the solution is block the mac address to drop the packets.
You could use the following debug, (note: you should use it few seconds, 3 or 5 seconds, also you could delete the logs and increase the buffer)
debug platform cpu-queues software-fwd-q
to disable
u all (press enter)
It is not common but also take in consideration the switch 2950 was manufactured a long time ago and a bug could arise.
Hope it is useful
:-)
08-23-2017 05:33 AM
08-23-2017 05:41 AM
for that for yourself you can use this command then to shorten it when trying to see cpu hitters if proc sorted is unavailable
sh proc cpu | ex 0.00
anywat thats a clean output in cpu if it was taken during the issue as it needs to be captured real time , so you may have triggered a bug , very old software running on it
You may have hit a memory leak like below in the pool manager , when softwrae gets older its a lot easier for it too happen , reboot will clear it , an upgarde should fix it , thenot uncommon ihit oine in an 4331 router with new ios on it the other day , upgraded the software its fixed now
The other process is for multicast tafffic in memory
08-23-2017 06:13 AM
Hi
It looks like a bug, check the following image, usually the first field (5%) should be always higher than the second field (0%), now if you have the inverse, it could represent a problem as I mentioned previously. Now should be good take a screenshot once the behavior is present, probably a specific process is generating high CPU.
08-24-2017 06:27 AM
Thank you Julio.
This is verry helpfull for me.
I will wait for the problem to occur again and I will troubleshoot it during the high utilization time.
I managed to observe that the memory has dropped suddenly to its normaml condition of utilization and for the last couple of days it begins again to increase the utilization slowly.
Kind regards,
08-08-2019 08:29 AM - edited 08-08-2019 08:44 AM
I had this problem too. On all of 2950. Switches had a lot of vlans that it got by VTP. I heard this model can't work with more than 70 vlans. Actually, I had issue with deleting vlans on one of my 2950 switches with VTP mode transparent and then I found info - no more 70 vlans or you will get FFFFed up vlan.dat and there will be high processor memory utilization. OK, I deleted vlan.dat, rebooted and then I created only useful vlans (about 5) and processor memory utilization became admissible - something about 65 percent. On other switches there was no problem with deleting so all that was needed was deleting uneuseful vlans
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide