Re: Switches 2950G-48 Memory Issues

erickflamenco · ‎10-24-2011

Hi Everyone,

I have some old switches 2950G-48 with some memory allocation errors, when I see the show version I see a different size of memory as reported by the CNA and the show memory stats.

Question:

Why they don´t match?

Show version

cisco WS-C2950G-48-EI (RC32300) processor (revision L0) with 19918K bytes of memory.

Processor board ID FOC0808Y029

Last reset from system-reset

Running Enhanced Image

48 FastEthernet/IEEE 802.3 interface(s)

2 Gigabit Ethernet/IEEE 802.3 interface(s)

Show memory

#sh memory

Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)

Processor 80C84FA0 3891296 3737636 153660 0 46496

I/O A0A6FDE0 2183584 1806852 376732 167048 11880

3891296 +

2183584

6074880

With CNA Heath/Health Details/Memory Utilization

Hostname	Utilization %	Total (MB)	Used (MB)	Free (MB)
SW2950	80	5.79	4.63	1.17

Some answers?

Thanks a lot for your help!!!

Eugene Khabarov · ‎10-27-2011

5,79 = 6074880/1024/1024

1M = 1024K

1K = 1024B

19918K is a packet-memory. That's make your confusion.

Packet memory, is either:

•The total physical input/output (I/O) memory (or "Fast memory") installed on the router (Cisco 4000, 4500, 4700, and 7500 series), or

•The amount of "shared memory" used for packet buffering. In the shared memory scheme (Cisco 2500, 2600, 3600, and 7200 Series), a percentage of DRAM is used for packet buffering by the router's network interfaces.

Note The terms "I/O memory" or "iomem"; "shared memory"; "Fast memory" and "PCI memory" all refer to "Packet Memory". Packet memory is either separate physical RAM or shared DRAM.

___

HTH. Please rate this post if it was helpful. If this solves your problem, please mark this post as "Correct Answer"

erickflamenco · ‎10-27-2011

Hi Eugine,

Thanks a lot for your response, Now I have found the match between the CNA report and the show memory summary, but the doubt still remains about the 19.45 MB reported by the show version command output.

The switch still has:

19.45MB -

5.79MB

13.66MB of memory available for processor and I/O.

Where are they?

What must be done to make available this memory and overcome the MEM-ALLOC failures due to the lack of memory resources?

Erick

Eugene Khabarov · ‎10-27-2011

Can you please provide me with full "sh ver" output.

___

HTH. Please rate this post if it was helpful. If this solves your problem, please mark this post as "Correct Answer"

erickflamenco · ‎10-27-2011

Of course,

#sh ver
Cisco Internetwork Operating System Software
IOS (tm) C2950 Software (C2950-I6K2L2Q4-M), Version 12.1(22)EA4, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2005 by cisco Systems, Inc.
Compiled Wed 23-Mar-05 17:25 by yenanh
Image text-base: 0x80010000, data-base: 0x80672000

ROM: Bootstrap program is C2950 boot loader

SWITCH uptime is 4 hours, 56 minutes
System returned to ROM by error - Signal 10, Exception code (0x0008)!, PC 0x800844DC
System restarted at 11:50:52 Wed Oct 26 2011
System image file is "flash:c2950-i6k2l2q4-mz.121-22.EA4.bin"

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C2950G-48-EI (RC32300) processor (revision K0) with 19994K bytes of memory.
Processor board ID FOC0736X1Z9
Last reset from system-reset
Running Enhanced Image
48 FastEthernet/IEEE 802.3 interface(s)
2 Gigabit Ethernet/IEEE 802.3 interface(s)

32K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address: 00:0D:BC:64:F7:40
Motherboard assembly number: 73-7409-12
Power supply part number: 34-0965-01
Motherboard serial number: FOC07351KZJ
Power supply serial number: PHI07260F3Q
Model revision number: K0
Motherboard revision number: A0
Model number: WS-C2950G-48-EI
System serial number: FOC0736X1Z9
Configuration register is 0xF

Eugene Khabarov · ‎10-28-2011

Here is really serious problems:

This device was last restarted by an error. The reload cause was:

'error - Signal 10, Exception code (0x0008)!, PC 0x800844DC'

This indicates a software error or less frequently, a hardware problem.

Paste the output of the 'show stacks' command for further analysis.

Also output interpreter confirms that:

This device has 19994 K available for main memory.

This device has a total of 19.5253906 MB of RAM installed.

And data sheet says, so that is strange:

8 MB memory architecture shared by all ports

Up to 16 MB SDRAM and 8 MB Flash memory

Please consider to upgrade to

c2950-i6k2l2q4-mz.121-22.EA14.bin

Also output interpreter says about your show memory output:

Processor memory utilization is 96.05119%. This is considered to be very

high.

Processor memory or main memory stores the running configuration and routing tables.

The Cisco IOS software executes from main memory.

The amount of processor memory required by the router is affected by the Cisco IOS

version used, the size of the network and by the access list configurations.

Also please paste here "show processes memory" for analysys.

___

HTH. Please rate this post if it was helpful. If this solves your problem, please mark this post as "Correct Answer"

erickflamenco · ‎10-28-2011

Hi Eugene,

Here is the output

------------------ show stacks ------------------

Minimum process stacks:

Free/Size Name

4748/6000 calhoun_flash init

3812/6000 SaveCrashBuffer

5564/6000 vqpc_shim_create_addr_tbl

5548/6000 SPAN Subsystem

5532/6000 platform SPAN Subsystem

8720/12000 Init

5252/6000 RADIUS INITCONFIG

9512/12000 HTTP Server

34584/36000 TCP Command

8376/12000 SSH Process

9848/12000 Virtual Exec

3768/6000 Calhoun CrashBuffer

7432/9000 DHCP Client

Interrupt level stacks:

Level Called Unused/Size Name

3 2463021 7504/9000 StrataSwitch

4 29245 8700/9000 EIC intr

7 4432751 8752/9000 Timer Interrupt Handler

Spurious interrupts: 26

System was restarted by error - Signal 10, Exception code (0x0008)!, PC 0x800844DC

C2950 Software (C2950-I6K2L2Q4-M), Version 12.1(22)EA4, RELEASE SOFTWARE (fc1)

Compiled Wed 23-Mar-05 17:25 by yenanh (current version)

Image text-base: 0x80010000, data-base: 0x80672000

Stack trace from system failure:

FP: 0xA0C6DB00, RA: 0x800844DC

FP: 0xA0C6DB38, RA: 0x80154848

FP: 0xA0C6DB58, RA: 0x8014F5B4

FP: 0xA0C6DB70, RA: 0x8008AA2C

FP: 0xA0C6DC08, RA: 0x80085CF0

FP: 0xA0C6DC30, RA: 0x80088838

FP: 0xA0C6DC58, RA: 0x80088CA8

FP: 0xA0C6DC78, RA: 0x80088E80

***************************************************

******* Information of Last System Crash **********

***************************************************

Using flash:/crashinfo/crashinfo_1.

C2950 Software (C2950-I6K2L2Q4-M), Version 12.1(22)EA4, RELEASE SOFTWARE (fc1)

Compiled Wed 23-Mar-05 17:25 by yenanh

Signal = 10, Code = 0x8, Uptime 2w2d

r0: 00000000, AT: 00000000, v0: 800844B8, v1: A0C6DC88

a0: A0C6DC88, a1: 00000000, a2: 11800000, a3: 8014AC50

t0: 00000018, t1: 1000E701, t2: 1000E100, t3: FFFF00FF

t4: 802D0DD8, t5: 80CA3550, t6: 80CA354C, t7: 80CA3548

s0: 00000001, s1: 80CA2380, s2: A0C6DC88, s3: 00000000

s4: A0C6DC88, s5: 00000000, s6: 00000000, s7: 00000000

t8: 80CA35B8, t9: 00000000, k0: 00000000, k1: 00000000

gp: 8092CD20, sp: A0C6DB00, s8: 00000000, ra: 80154848

EPC: 800844DC, ErrEPC: 9FC02DF8, BadVA: 0x00000000, SREG: 1000E703

Cause: 0x00000008 (code 2): TLB (load or ifetch) exception

Signal 10, Exception code (0x0008)!

Frame 00: SP = 0xA0C6DB00 PC = 0x800844DC

Frame 01: SP = 0xA0C6DB38 PC = 0x80154848

Frame 02: SP = 0xA0C6DB58 PC = 0x8014F5B4

Frame 03: SP = 0xA0C6DB70 PC = 0x8008AA2C

Frame 04: SP = 0xA0C6DC08 PC = 0x80085CF0

Frame 05: SP = 0xA0C6DC30 PC = 0x80088838

Frame 06: SP = 0xA0C6DC58 PC = 0x80088CA8

Frame 07: SP = 0xA0C6DC78 PC = 0x80088E80

Frame 08: SP = 0xA0C6DD98 PC = 0x801BC034

Frame 09: SP = 0xA0C6DDB0 PC = 0x801BC020

-Traceback= 800844DC 80154848 8014F5B4 8008AA2C 80085CF0 80088838 80088CA8 80088E80 801BC034 801BC020

Regards,

erickflamenco · ‎10-28-2011

Hi Eugene,

Here is the show precess output

------------------ show process memory ------------------

Total: 6200704, Used: 5974272, Free: 226432

PID TTY Allocated Freed Holding Getbufs Retbufs Process

0 0 121548 5044 3483900 0 0 *Init*

0 0 1164 4506856 1164 0 0 *Sched*

0 0 10508152 6782348 226712 984380 0 *Dead*

1 0 12384 12384 6808 0 0 Chunk Manager

2 0 184 184 3808 0 0 Load Meter

3 0 69952 93856 6808 0 0 SpanTree Helper

4 0 0 0 6808 0 0 Check heaps

5 0 8833992 76608 186732 6138384 113756 Pool Manager

6 0 184 184 6808 0 0 Timers

7 0 2120356 1875352 575596 0 159696 Entity MIB API

8 0 0 0 6808 0 0 HC Counter Timer

9 0 140 140 6948 0 0 ARP Input

10 0 0 54076 6808 0 679268 Net Input

11 0 0 0 6808 0 0 Critical Bkgnd

12 0 20688 0 13072 0 0 Net Background

13 0 184 184 12808 0 0 Logger

14 0 184 952 6808 0 800 TTY Background

15 0 0 0 9808 0 0 Per-Second Jobs

16 0 184 184 6808 0 0 Compute load avg

17 0 0 0 6808 0 0 Per-minute Jobs

18 0 0 0 6808 0 0 AggMgr Process

19 1 643744 563520 90968 4536 0 SSH Process

20 0 12228 184 18852 0 0 HMATM Learn proc

21 0 340 184 3964 0 0 Calhoun Storm Co

22 0 0 0 6808 0 0 Calhoun LED Proc

23 0 3518996 3518996 3808 0 0 Calhoun Statisti

24 0 467580 94144 31276 0 0 CalhounPM

25 0 184 184 6808 0 0 Link Status Proc

26 0 0 0 6808 0 0 CalhounInsIpopti

27 0 0 0 3808 0 0 Calhoun DHCP Sno

28 0 83584 71288 11872 5040 0 CDP Protocol

29 0 0 0 6808 0 0 DHCP Snooping

30 0 410364 8400 13980 0 1600 IP Input

31 0 140952 5308 102176 0 304 DTP Protocol

32 0 184 184 6808 0 0 Ethchnl

33 0 654788 70780 163416 102180 0 VLAN Manager

34 0 184 184 6808 0 0 PI MATM Aging Pr

35 0 393480 266400 400288 0 0 VMATM Callback

36 0 861560 925676 6960 156520 412084 Spanning Tree

37 0 412 184 7036 0 0 UDLD

38 0 29444 0 10240 0 0 Port-Security

39 0 114064 84960 22296 0 0 DHCPD Receive

40 0 0 0 6808 0 0 HTTP Timer

41 0 0 0 6808 0 0 RARP Input

42 0 136 370652 12808 0 0 TCP Timer

43 0 5115920 7032 14460 0 0 TCP Protocols

44 0 0 0 6808 0 0 Socket Timers

45 0 107904 147728 114712 0 0 L2MM

46 0 508 508 6808 0 0 MRD

47 0 0 3420 6808 0 1520 IGMPSN

48 0 0 0 3808 0 0 IGMP Snooping Pr

49 0 0 0 3808 0 0 IGMP Snooping Re

50 0 0 1923204 3808 0 5205644 IP Multicast Dat

51 0 2720 184 9344 0 0 L2TRACE SERVER

52 0 184 184 6808 0 0 Cluster L2

53 0 0 0 6808 0 0 Cluster RARP

54 0 300 184 6924 0 0 Cluster Base

56 0 0 0 6808 0 0 DHCPD Timer

57 0 0 68 9808 0 0 Router Autoconf

58 0 2182452 2192084 9776 0 0 Key Proc

59 0 81364 70044 20128 0 0 Crypto CA

60 0 0 0 6808 0 0 Syslog Traps

61 0 184 184 6808 0 0 VTPMIB EDIT BUFF

62 0 240320 496828 11492 0 0 PM Callback

63 0 5228 9292 11852 0 0 VTP Trap Process

64 0 152 0 6960 0 0 DHCPD Database

65 0 184 184 3808 0 0 Calhoun EnvMon P

66 0 184 184 3808 0 0 STP STACK TOPOLO

67 0 184 184 3808 0 0 STP FAST TRANSIT

68 0 184 184 3808 0 0 STP STACK TOPOLO

69 0 184 184 3808 0 0 CSRT RAPID TRANS

70 0 3292 272 6808 0 0 TACACS+

71 0 15484 7004 6828 0 0 SSH Event handle

72 0 0 0 6808 0 0 SNMP Timers

73 0 1340488 1328140 13716 9352 0 IP SNMP

74 0 3561092 0 12808 0 0 PDU DISPATCHER

75 0 8953028 12514344 13216 0 0 SNMP ENGINE

76 0 0 0 12852 0 0 SNMP ConfCopyPro

77 0 0 0 12808 0 0 SNMP Traps

78 0 0 0 6808 0 0 RADIUS

79 0 7876 6768 7916 0 0 NTP

80 0 0 0 6808 0 0 Periodic Account

81 0 28204 30064 7024 0 0 AAA Accounting

5968432 Total

------------------ show process cpu ------------------

CPU utilization for five seconds: 99%/5%; one minute: 38%; five minutes: 25%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

1 0 17 0 0.00% 0.00% 0.00% 0 Chunk Manager

2 76 3543 21 0.00% 0.00% 0.00% 0 Load Meter

3 1084 116 9344 0.00% 0.01% 0.00% 0 SpanTree Helper

4 19580 2470 7927 0.00% 0.06% 0.07% 0 Check heaps

5 2364 924 2558 0.47% 0.10% 0.02% 0 Pool Manager

6 0 2 0 0.00% 0.00% 0.00% 0 Timers

7 1712 11 155636 0.00% 0.00% 0.00% 0 Entity MIB API

8 464 4426 104 0.00% 0.00% 0.00% 0 HC Counter Timer

9 10848 19834 546 0.00% 0.01% 0.00% 0 ARP Input

10 76 471 161 0.00% 0.00% 0.00% 0 Net Input

11 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd

12 23288 27213 855 0.00% 0.09% 0.06% 0 Net Background

13 56 327 171 0.00% 0.00% 0.00% 0 Logger

14 436 17684 24 0.00% 0.00% 0.00% 0 TTY Background

15 716 17697 40 0.00% 0.00% 0.00% 0 Per-Second Jobs

16 3648 3542 1029 0.00% 0.01% 0.00% 0 Compute load avg

17 3616 296 12216 0.00% 0.01% 0.00% 0 Per-minute Jobs

18 0 3 0 0.00% 0.00% 0.00% 0 AggMgr Process

19 15692 171 91766 84.08% 17.17% 4.15% 1 SSH Process

20 14892 87690 169 0.00% 0.01% 0.00% 0 HMATM Learn proc

21 0 2 0 0.00% 0.00% 0.00% 0 Calhoun Storm Co

22 1850316 390203 4741 3.31% 8.82% 9.86% 0 Calhoun LED Proc

23 285516 34724 8222 1.10% 1.37% 1.43% 0 Calhoun Statisti

24 13480 284727 47 0.15% 0.11% 0.06% 0 CalhounPM

25 34440 88110 390 0.86% 0.29% 0.17% 0 Link Status Proc

26 0 1 0 0.00% 0.00% 0.00% 0 CalhounInsIpopti

27 0 1 0 0.00% 0.00% 0.00% 0 Calhoun DHCP Sno

28 10240 4751 2155 0.07% 0.05% 0.01% 0 CDP Protocol

29 4 148 27 0.00% 0.00% 0.00% 0 DHCP Snooping

30 16704 28485 586 0.23% 0.07% 0.01% 0 IP Input

31 1696 1189 1426 0.07% 0.00% 0.00% 0 DTP Protocol

32 12 1773 6 0.00% 0.00% 0.00% 0 Ethchnl

33 7956 5930 1341 0.00% 0.03% 0.00% 0 VLAN Manager

34 0 2 0 0.00% 0.00% 0.00% 0 PI MATM Aging Pr

35 460 650 707 0.00% 0.00% 0.00% 0 VMATM Callback

36 1144216 1293774 884 3.54% 3.68% 3.99% 0 Spanning Tree

37 600 17711 33 0.15% 0.02% 0.00% 0 UDLD

38 528 392 1346 0.00% 0.00% 0.00% 0 Port-Security

39 252 381 661 0.00% 0.00% 0.00% 0 DHCPD Receive

40 0 1 0 0.00% 0.00% 0.00% 0 HTTP Timer

41 0 1 0 0.00% 0.00% 0.00% 0 RARP Input

42 116 1118 103 0.00% 0.00% 0.00% 0 TCP Timer

43 892 577 1545 0.00% 0.00% 0.00% 0 TCP Protocols

44 0 1 0 0.00% 0.00% 0.00% 0 Socket Timers

45 280 366 765 0.00% 0.00% 0.00% 0 L2MM

46 32 62 516 0.00% 0.00% 0.00% 0 MRD

47 300 503 596 0.00% 0.00% 0.00% 0 IGMPSN

48 0 1 0 0.00% 0.00% 0.00% 0 IGMP Snooping Pr

49 196 517 379 0.00% 0.00% 0.00% 0 IGMP Snooping Re

50 11048 31097 355 0.15% 0.05% 0.03% 0 IP Multicast Dat

51 4 2 2000 0.00% 0.00% 0.00% 0 L2TRACE SERVER

52 4 224 17 0.00% 0.00% 0.00% 0 Cluster L2

53 60 1772 33 0.00% 0.00% 0.00% 0 Cluster RARP

54 72 2216 32 0.00% 0.00% 0.00% 0 Cluster Base

56 8 148 54 0.00% 0.00% 0.00% 0 DHCPD Timer

57 0 2 0 0.00% 0.00% 0.00% 0 Router Autoconf

58 16740 79 211898 0.00% 0.00% 0.00% 0 Key Proc

59 724 6 120666 0.00% 0.00% 0.00% 0 Crypto CA

60 0 1 0 0.00% 0.00% 0.00% 0 Syslog Traps

61 0 2 0 0.00% 0.00% 0.00% 0 VTPMIB EDIT BUFF

62 5180 110473 46 0.15% 0.04% 0.01% 0 PM Callback

63 20 131 152 0.00% 0.00% 0.00% 0 VTP Trap Process

64 128 5000 25 0.00% 0.00% 0.00% 0 DHCPD Database

65 160 17687 9 0.00% 0.00% 0.00% 0 Calhoun EnvMon P

66 4 2 2000 0.00% 0.00% 0.00% 0 STP STACK TOPOLO

67 0 2 0 0.00% 0.00% 0.00% 0 STP FAST TRANSIT

68 0 2 0 0.00% 0.00% 0.00% 0 STP STACK TOPOLO

69 4 2 2000 0.00% 0.00% 0.00% 0 CSRT RAPID TRANS

70 40 70 571 0.00% 0.00% 0.00% 0 TACACS+

71 56 203 275 0.00% 0.00% 0.00% 0 SSH Event handle

72 0 10 0 0.00% 0.00% 0.00% 0 SNMP Timers

73 7480 8175 914 0.00% 0.04% 0.00% 0 IP SNMP

74 2936 4090 717 0.00% 0.02% 0.00% 0 PDU DISPATCHER

75 13764 4094 3361 0.00% 0.08% 0.06% 0 SNMP ENGINE

76 0 1 0 0.00% 0.00% 0.00% 0 SNMP ConfCopyPro

77 0 1 0 0.00% 0.00% 0.00% 0 SNMP Traps

78 0 1 0 0.00% 0.00% 0.00% 0 RADIUS

79 720 17950 40 0.00% 0.00% 0.00% 0 NTP

80 4 1 4000 0.00% 0.00% 0.00% 0 Periodic Account

81 76 44 1727 0.00% 0.00% 0.00% 0 AAA Accounting

It could be something wrong with the SSH...

Eugene Khabarov · ‎10-30-2011

Hi! First of all try to upgrade to the latest software as already mentioned.

Second, if it is not helpful try to disable ssh ('crypto key zerosize rsa').

BTW here is helpful document about troubleshooting high CPU utilization:

http://www.cisco.com/en/US/products/hw/switches/ps607/products_tech_note09186a0080094e78.shtml

___

HTH. Please rate this post if it was helpful. If this solves your problem, please mark this post as "Correct Answer"

nkarpysh · ‎10-31-2011

Hi,

SSH is not a problem - it just spike when you performed some commands - I guess you do it through SSH - thus it took some CPU. It is not priority process and will not affect switch performance.

Agree with upgrade needed. Seems to be a memory leak. So you need to do "show proc mem sorted" regularly to see which process is continiously growing. But please note -ending up without no memory is not a good idea - leading to crash. So at particular moment you can consider reload and upgrade simultanenously.

Nik

HTH,
Niko

Giovanni Ceci · ‎09-15-2014

I'm having this problem too. I noticed that this problem occurs on any switches running a crypto-supported IOS. I reported the problem in this post:

https://supportforums.cisco.com/discussion/12292911/cisco-2950-switch-crypto-ios-image-reporting-1mb-less-total-memory-switch-non

We have other switches that are not having the low memory problem that are running older non-crypto IOS versions. I have yet to test with the different newer IOS versions but I arranging to get a test switch and test it out. Will let you know how it goes. If you have the opportunity, I would give that a shot too.

John