ā10-24-2011 05:23 PM - edited ā03-07-2019 03:02 AM
Hi Everyone,
I have some old switches 2950G-48 with some memory allocation errors, when I see the show version I see a different size of memory as reported by the CNA and the show memory stats.
Question:
Why they donĀ“t match?
Show version
cisco WS-C2950G-48-EI (RC32300) processor (revision L0) with 19918K bytes of memory.
Processor board ID FOC0808Y029
Last reset from system-reset
Running Enhanced Image
48 FastEthernet/IEEE 802.3 interface(s)
2 Gigabit Ethernet/IEEE 802.3 interface(s)
Show memory
#sh memory
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 80C84FA0 3891296 3737636 153660 0 46496
I/O A0A6FDE0 2183584 1806852 376732 167048 11880
3891296 +
2183584
6074880
With CNA Heath/Health Details/Memory Utilization
Hostname | Utilization % | Total (MB) | Used (MB) | Free (MB) |
---|---|---|---|---|
SW2950 | 80 | 5.79 | 4.63 | 1.17 |
Some answers?
Thanks a lot for your help!!!
ā10-27-2011 05:18 AM
5,79 = 6074880/1024/1024
1M = 1024K
1K = 1024B
19918K is a packet-memory. That's make your confusion.
Packet memory, is either:
ā¢
The total physical input/output (I/O) memory (or "Fast memory") installed on the router (Cisco 4000, 4500, 4700, and 7500 series), or
ā¢
The amount of "shared memory" used for packet buffering. In the shared memory scheme (Cisco 2500, 2600, 3600, and 7200 Series), a percentage of DRAM is used for packet buffering by the router's network interfaces.
Note
The terms "I/O memory" or "iomem"; "shared memory"; "Fast memory" and "PCI memory" all refer to "Packet Memory". Packet memory is either separate physical RAM or shared DRAM.
___
HTH. Please rate this post if it was helpful. If this solves your problem, please mark this post as "Correct Answer"
ā10-27-2011 09:28 AM
Hi Eugine,
Thanks a lot for your response, Now I have found the match between the CNA report and the show memory summary, but the doubt still remains about the 19.45 MB reported by the show version command output.
The switch still has:
19.45MB -
5.79MB
13.66MB of memory available for processor and I/O.
Where are they?
What must be done to make available this memory and overcome the MEM-ALLOC failures due to the lack of memory resources?
Erick
ā10-27-2011 11:22 AM
Can you please provide me with full "sh ver" output.
___
HTH. Please rate this post if it was helpful. If this solves your problem, please mark this post as "Correct Answer"
ā10-27-2011 02:50 PM
Of course,
#sh ver
Cisco Internetwork Operating System Software
IOS (tm) C2950 Software (C2950-I6K2L2Q4-M), Version 12.1(22)EA4, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2005 by cisco Systems, Inc.
Compiled Wed 23-Mar-05 17:25 by yenanh
Image text-base: 0x80010000, data-base: 0x80672000
ROM: Bootstrap program is C2950 boot loader
SWITCH uptime is 4 hours, 56 minutes
System returned to ROM by error - Signal 10, Exception code (0x0008)!, PC 0x800844DC
System restarted at 11:50:52 Wed Oct 26 2011
System image file is "flash:c2950-i6k2l2q4-mz.121-22.EA4.bin"
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
If you require further assistance please contact us by sending email to
export@cisco.com.
cisco WS-C2950G-48-EI (RC32300) processor (revision K0) with 19994K bytes of memory.
Processor board ID FOC0736X1Z9
Last reset from system-reset
Running Enhanced Image
48 FastEthernet/IEEE 802.3 interface(s)
2 Gigabit Ethernet/IEEE 802.3 interface(s)
32K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address: 00:0D:BC:64:F7:40
Motherboard assembly number: 73-7409-12
Power supply part number: 34-0965-01
Motherboard serial number: FOC07351KZJ
Power supply serial number: PHI07260F3Q
Model revision number: K0
Motherboard revision number: A0
Model number: WS-C2950G-48-EI
System serial number: FOC0736X1Z9
Configuration register is 0xF
ā10-28-2011 12:18 AM
Here is really serious problems:
This device was last restarted by an error. The reload cause was:
'error - Signal 10, Exception code (0x0008)!, PC 0x800844DC'
This indicates a software error or less frequently, a hardware problem.
Paste the output of the 'show stacks' command for further analysis.
Also output interpreter confirms that:
This device has 19994 K available for main memory.
This device has a total of 19.5253906 MB of RAM installed.
And data sheet says, so that is strange:
8 MB memory architecture shared by all ports
Up to 16 MB SDRAM and 8 MB Flash memory
Please consider to upgrade to
c2950-i6k2l2q4-mz.121-22.EA14.bin
Also output interpreter says about your show memory output:
Processor memory utilization is 96.05119%. This is considered to be very
high.
Processor memory or main memory stores the running configuration and routing tables.
The Cisco IOS software executes from main memory.
The amount of processor memory required by the router is affected by the Cisco IOS
version used, the size of the network and by the access list configurations.
Also please paste here "show processes memory" for analysys.
___
HTH. Please rate this post if it was helpful. If this solves your problem, please mark this post as "Correct Answer"
ā10-28-2011 08:50 AM
Hi Eugene,
Here is the output
------------------ show stacks ------------------
ć
Minimum process stacks:
Free/Size Name
4748/6000 calhoun_flash init
3812/6000 SaveCrashBuffer
5564/6000 vqpc_shim_create_addr_tbl
5548/6000 SPAN Subsystem
5532/6000 platform SPAN Subsystem
8720/12000 Init
5252/6000 RADIUS INITCONFIG
9512/12000 HTTP Server
34584/36000 TCP Command
8376/12000 SSH Process
9848/12000 Virtual Exec
3768/6000 Calhoun CrashBuffer
7432/9000 DHCP Client
Interrupt level stacks:
Level Called Unused/Size Name
3 2463021 7504/9000 StrataSwitch
4 29245 8700/9000 EIC intr
7 4432751 8752/9000 Timer Interrupt Handler
Spurious interrupts: 26
System was restarted by error - Signal 10, Exception code (0x0008)!, PC 0x800844DC
C2950 Software (C2950-I6K2L2Q4-M), Version 12.1(22)EA4, RELEASE SOFTWARE (fc1)
Compiled Wed 23-Mar-05 17:25 by yenanh (current version)
Image text-base: 0x80010000, data-base: 0x80672000
ć
Stack trace from system failure:
FP: 0xA0C6DB00, RA: 0x800844DC
FP: 0xA0C6DB38, RA: 0x80154848
FP: 0xA0C6DB58, RA: 0x8014F5B4
FP: 0xA0C6DB70, RA: 0x8008AA2C
FP: 0xA0C6DC08, RA: 0x80085CF0
FP: 0xA0C6DC30, RA: 0x80088838
FP: 0xA0C6DC58, RA: 0x80088CA8
FP: 0xA0C6DC78, RA: 0x80088E80
ć
***************************************************
******* Information of Last System Crash **********
***************************************************
ć
Using flash:/crashinfo/crashinfo_1.
C2950 Software (C2950-I6K2L2Q4-M), Version 12.1(22)EA4, RELEASE SOFTWARE (fc1)
Compiled Wed 23-Mar-05 17:25 by yenanh
Signal = 10, Code = 0x8, Uptime 2w2d
r0: 00000000, AT: 00000000, v0: 800844B8, v1: A0C6DC88
a0: A0C6DC88, a1: 00000000, a2: 11800000, a3: 8014AC50
t0: 00000018, t1: 1000E701, t2: 1000E100, t3: FFFF00FF
t4: 802D0DD8, t5: 80CA3550, t6: 80CA354C, t7: 80CA3548
s0: 00000001, s1: 80CA2380, s2: A0C6DC88, s3: 00000000
s4: A0C6DC88, s5: 00000000, s6: 00000000, s7: 00000000
t8: 80CA35B8, t9: 00000000, k0: 00000000, k1: 00000000
gp: 8092CD20, sp: A0C6DB00, s8: 00000000, ra: 80154848
EPC: 800844DC, ErrEPC: 9FC02DF8, BadVA: 0x00000000, SREG: 1000E703
Cause: 0x00000008 (code 2): TLB (load or ifetch) exception
Signal 10, Exception code (0x0008)!
Frame 00: SP = 0xA0C6DB00 PC = 0x800844DC
Frame 01: SP = 0xA0C6DB38 PC = 0x80154848
Frame 02: SP = 0xA0C6DB58 PC = 0x8014F5B4
Frame 03: SP = 0xA0C6DB70 PC = 0x8008AA2C
Frame 04: SP = 0xA0C6DC08 PC = 0x80085CF0
Frame 05: SP = 0xA0C6DC30 PC = 0x80088838
Frame 06: SP = 0xA0C6DC58 PC = 0x80088CA8
Frame 07: SP = 0xA0C6DC78 PC = 0x80088E80
Frame 08: SP = 0xA0C6DD98 PC = 0x801BC034
Frame 09: SP = 0xA0C6DDB0 PC = 0x801BC020
ć
-Traceback= 800844DC 80154848 8014F5B4 8008AA2C 80085CF0 80088838 80088CA8 80088E80 801BC034 801BC020
Regards,
ā10-28-2011 01:37 PM
Hi Eugene,
Here is the show precess output
------------------ show process memory ------------------
ć
Total: 6200704, Used: 5974272, Free: 226432
PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 121548 5044 3483900 0 0 *Init*
0 0 1164 4506856 1164 0 0 *Sched*
0 0 10508152 6782348 226712 984380 0 *Dead*
1 0 12384 12384 6808 0 0 Chunk Manager
2 0 184 184 3808 0 0 Load Meter
3 0 69952 93856 6808 0 0 SpanTree Helper
4 0 0 0 6808 0 0 Check heaps
5 0 8833992 76608 186732 6138384 113756 Pool Manager
6 0 184 184 6808 0 0 Timers
7 0 2120356 1875352 575596 0 159696 Entity MIB API
8 0 0 0 6808 0 0 HC Counter Timer
9 0 140 140 6948 0 0 ARP Input
10 0 0 54076 6808 0 679268 Net Input
11 0 0 0 6808 0 0 Critical Bkgnd
12 0 20688 0 13072 0 0 Net Background
13 0 184 184 12808 0 0 Logger
14 0 184 952 6808 0 800 TTY Background
15 0 0 0 9808 0 0 Per-Second Jobs
16 0 184 184 6808 0 0 Compute load avg
17 0 0 0 6808 0 0 Per-minute Jobs
18 0 0 0 6808 0 0 AggMgr Process
19 1 643744 563520 90968 4536 0 SSH Process
20 0 12228 184 18852 0 0 HMATM Learn proc
21 0 340 184 3964 0 0 Calhoun Storm Co
22 0 0 0 6808 0 0 Calhoun LED Proc
23 0 3518996 3518996 3808 0 0 Calhoun Statisti
24 0 467580 94144 31276 0 0 CalhounPM
25 0 184 184 6808 0 0 Link Status Proc
26 0 0 0 6808 0 0 CalhounInsIpopti
27 0 0 0 3808 0 0 Calhoun DHCP Sno
28 0 83584 71288 11872 5040 0 CDP Protocol
29 0 0 0 6808 0 0 DHCP Snooping
30 0 410364 8400 13980 0 1600 IP Input
31 0 140952 5308 102176 0 304 DTP Protocol
32 0 184 184 6808 0 0 Ethchnl
33 0 654788 70780 163416 102180 0 VLAN Manager
34 0 184 184 6808 0 0 PI MATM Aging Pr
35 0 393480 266400 400288 0 0 VMATM Callback
36 0 861560 925676 6960 156520 412084 Spanning Tree
37 0 412 184 7036 0 0 UDLD
38 0 29444 0 10240 0 0 Port-Security
39 0 114064 84960 22296 0 0 DHCPD Receive
40 0 0 0 6808 0 0 HTTP Timer
41 0 0 0 6808 0 0 RARP Input
42 0 136 370652 12808 0 0 TCP Timer
43 0 5115920 7032 14460 0 0 TCP Protocols
44 0 0 0 6808 0 0 Socket Timers
45 0 107904 147728 114712 0 0 L2MM
46 0 508 508 6808 0 0 MRD
47 0 0 3420 6808 0 1520 IGMPSN
48 0 0 0 3808 0 0 IGMP Snooping Pr
49 0 0 0 3808 0 0 IGMP Snooping Re
50 0 0 1923204 3808 0 5205644 IP Multicast Dat
51 0 2720 184 9344 0 0 L2TRACE SERVER
52 0 184 184 6808 0 0 Cluster L2
53 0 0 0 6808 0 0 Cluster RARP
54 0 300 184 6924 0 0 Cluster Base
56 0 0 0 6808 0 0 DHCPD Timer
57 0 0 68 9808 0 0 Router Autoconf
58 0 2182452 2192084 9776 0 0 Key Proc
59 0 81364 70044 20128 0 0 Crypto CA
60 0 0 0 6808 0 0 Syslog Traps
61 0 184 184 6808 0 0 VTPMIB EDIT BUFF
62 0 240320 496828 11492 0 0 PM Callback
63 0 5228 9292 11852 0 0 VTP Trap Process
64 0 152 0 6960 0 0 DHCPD Database
65 0 184 184 3808 0 0 Calhoun EnvMon P
66 0 184 184 3808 0 0 STP STACK TOPOLO
67 0 184 184 3808 0 0 STP FAST TRANSIT
68 0 184 184 3808 0 0 STP STACK TOPOLO
69 0 184 184 3808 0 0 CSRT RAPID TRANS
70 0 3292 272 6808 0 0 TACACS+
71 0 15484 7004 6828 0 0 SSH Event handle
72 0 0 0 6808 0 0 SNMP Timers
73 0 1340488 1328140 13716 9352 0 IP SNMP
74 0 3561092 0 12808 0 0 PDU DISPATCHER
75 0 8953028 12514344 13216 0 0 SNMP ENGINE
76 0 0 0 12852 0 0 SNMP ConfCopyPro
77 0 0 0 12808 0 0 SNMP Traps
78 0 0 0 6808 0 0 RADIUS
79 0 7876 6768 7916 0 0 NTP
80 0 0 0 6808 0 0 Periodic Account
81 0 28204 30064 7024 0 0 AAA Accounting
5968432 Total
------------------ show process cpu ------------------
ć
CPU utilization for five seconds: 99%/5%; one minute: 38%; five minutes: 25%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 0 17 0 0.00% 0.00% 0.00% 0 Chunk Manager
2 76 3543 21 0.00% 0.00% 0.00% 0 Load Meter
3 1084 116 9344 0.00% 0.01% 0.00% 0 SpanTree Helper
4 19580 2470 7927 0.00% 0.06% 0.07% 0 Check heaps
5 2364 924 2558 0.47% 0.10% 0.02% 0 Pool Manager
6 0 2 0 0.00% 0.00% 0.00% 0 Timers
7 1712 11 155636 0.00% 0.00% 0.00% 0 Entity MIB API
8 464 4426 104 0.00% 0.00% 0.00% 0 HC Counter Timer
9 10848 19834 546 0.00% 0.01% 0.00% 0 ARP Input
10 76 471 161 0.00% 0.00% 0.00% 0 Net Input
11 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd
12 23288 27213 855 0.00% 0.09% 0.06% 0 Net Background
13 56 327 171 0.00% 0.00% 0.00% 0 Logger
14 436 17684 24 0.00% 0.00% 0.00% 0 TTY Background
15 716 17697 40 0.00% 0.00% 0.00% 0 Per-Second Jobs
16 3648 3542 1029 0.00% 0.01% 0.00% 0 Compute load avg
17 3616 296 12216 0.00% 0.01% 0.00% 0 Per-minute Jobs
18 0 3 0 0.00% 0.00% 0.00% 0 AggMgr Process
19 15692 171 91766 84.08% 17.17% 4.15% 1 SSH Process
20 14892 87690 169 0.00% 0.01% 0.00% 0 HMATM Learn proc
21 0 2 0 0.00% 0.00% 0.00% 0 Calhoun Storm Co
22 1850316 390203 4741 3.31% 8.82% 9.86% 0 Calhoun LED Proc
23 285516 34724 8222 1.10% 1.37% 1.43% 0 Calhoun Statisti
24 13480 284727 47 0.15% 0.11% 0.06% 0 CalhounPM
25 34440 88110 390 0.86% 0.29% 0.17% 0 Link Status Proc
26 0 1 0 0.00% 0.00% 0.00% 0 CalhounInsIpopti
27 0 1 0 0.00% 0.00% 0.00% 0 Calhoun DHCP Sno
28 10240 4751 2155 0.07% 0.05% 0.01% 0 CDP Protocol
29 4 148 27 0.00% 0.00% 0.00% 0 DHCP Snooping
30 16704 28485 586 0.23% 0.07% 0.01% 0 IP Input
31 1696 1189 1426 0.07% 0.00% 0.00% 0 DTP Protocol
32 12 1773 6 0.00% 0.00% 0.00% 0 Ethchnl
33 7956 5930 1341 0.00% 0.03% 0.00% 0 VLAN Manager
34 0 2 0 0.00% 0.00% 0.00% 0 PI MATM Aging Pr
35 460 650 707 0.00% 0.00% 0.00% 0 VMATM Callback
36 1144216 1293774 884 3.54% 3.68% 3.99% 0 Spanning Tree
37 600 17711 33 0.15% 0.02% 0.00% 0 UDLD
38 528 392 1346 0.00% 0.00% 0.00% 0 Port-Security
39 252 381 661 0.00% 0.00% 0.00% 0 DHCPD Receive
40 0 1 0 0.00% 0.00% 0.00% 0 HTTP Timer
41 0 1 0 0.00% 0.00% 0.00% 0 RARP Input
42 116 1118 103 0.00% 0.00% 0.00% 0 TCP Timer
43 892 577 1545 0.00% 0.00% 0.00% 0 TCP Protocols
44 0 1 0 0.00% 0.00% 0.00% 0 Socket Timers
45 280 366 765 0.00% 0.00% 0.00% 0 L2MM
46 32 62 516 0.00% 0.00% 0.00% 0 MRD
47 300 503 596 0.00% 0.00% 0.00% 0 IGMPSN
48 0 1 0 0.00% 0.00% 0.00% 0 IGMP Snooping Pr
49 196 517 379 0.00% 0.00% 0.00% 0 IGMP Snooping Re
50 11048 31097 355 0.15% 0.05% 0.03% 0 IP Multicast Dat
51 4 2 2000 0.00% 0.00% 0.00% 0 L2TRACE SERVER
52 4 224 17 0.00% 0.00% 0.00% 0 Cluster L2
53 60 1772 33 0.00% 0.00% 0.00% 0 Cluster RARP
54 72 2216 32 0.00% 0.00% 0.00% 0 Cluster Base
56 8 148 54 0.00% 0.00% 0.00% 0 DHCPD Timer
57 0 2 0 0.00% 0.00% 0.00% 0 Router Autoconf
58 16740 79 211898 0.00% 0.00% 0.00% 0 Key Proc
59 724 6 120666 0.00% 0.00% 0.00% 0 Crypto CA
60 0 1 0 0.00% 0.00% 0.00% 0 Syslog Traps
61 0 2 0 0.00% 0.00% 0.00% 0 VTPMIB EDIT BUFF
62 5180 110473 46 0.15% 0.04% 0.01% 0 PM Callback
63 20 131 152 0.00% 0.00% 0.00% 0 VTP Trap Process
64 128 5000 25 0.00% 0.00% 0.00% 0 DHCPD Database
65 160 17687 9 0.00% 0.00% 0.00% 0 Calhoun EnvMon P
66 4 2 2000 0.00% 0.00% 0.00% 0 STP STACK TOPOLO
67 0 2 0 0.00% 0.00% 0.00% 0 STP FAST TRANSIT
68 0 2 0 0.00% 0.00% 0.00% 0 STP STACK TOPOLO
69 4 2 2000 0.00% 0.00% 0.00% 0 CSRT RAPID TRANS
70 40 70 571 0.00% 0.00% 0.00% 0 TACACS+
71 56 203 275 0.00% 0.00% 0.00% 0 SSH Event handle
72 0 10 0 0.00% 0.00% 0.00% 0 SNMP Timers
73 7480 8175 914 0.00% 0.04% 0.00% 0 IP SNMP
74 2936 4090 717 0.00% 0.02% 0.00% 0 PDU DISPATCHER
75 13764 4094 3361 0.00% 0.08% 0.06% 0 SNMP ENGINE
76 0 1 0 0.00% 0.00% 0.00% 0 SNMP ConfCopyPro
77 0 1 0 0.00% 0.00% 0.00% 0 SNMP Traps
78 0 1 0 0.00% 0.00% 0.00% 0 RADIUS
79 720 17950 40 0.00% 0.00% 0.00% 0 NTP
80 4 1 4000 0.00% 0.00% 0.00% 0 Periodic Account
81 76 44 1727 0.00% 0.00% 0.00% 0 AAA Accounting
It could be something wrong with the SSH...
ā10-30-2011 10:41 PM
Hi! First of all try to upgrade to the latest software as already mentioned.
Second, if it is not helpful try to disable ssh ('crypto key zerosize rsa').
BTW here is helpful document about troubleshooting high CPU utilization:
http://www.cisco.com/en/US/products/hw/switches/ps607/products_tech_note09186a0080094e78.shtml
___
HTH. Please rate this post if it was helpful. If this solves your problem, please mark this post as "Correct Answer"
ā10-31-2011 08:27 AM
Hi,
SSH is not a problem - it just spike when you performed some commands - I guess you do it through SSH - thus it took some CPU. It is not priority process and will not affect switch performance.
Agree with upgrade needed. Seems to be a memory leak. So you need to do "show proc mem sorted" regularly to see which process is continiously growing. But please note -ending up without no memory is not a good idea - leading to crash. So at particular moment you can consider reload and upgrade simultanenously.
Nik
ā09-15-2014 10:14 PM
I'm having this problem too. I noticed that this problem occurs on any switches running a crypto-supported IOS. I reported the problem in this post:
We have other switches that are not having the low memory problem that are running older non-crypto IOS versions. I have yet to test with the different newer IOS versions but I arranging to get a test switch and test it out. Will let you know how it goes. If you have the opportunity, I would give that a shot too.
John
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide