11-02-2016 02:18 AM - edited 03-08-2019 08:01 AM
Hello everybody,
The switch in question is:
WS-C3560CG-8PC-S
IOS 152-1.E3
RAM 130 MB
FLASH 57 MB
My problem is this:
the switch is in production environment and the ping is always ok but sometimes is impossible access in SSH/TELNET.
The output is Connection timed out.
I have checked more times the configuration and is all ok. The same configuration of AAA ecc is span on more switch and the problems is only this model of switch.
The CPU and RAM is monitored by SNMP but when I have succeed to access on SSH I typed "show tech" "show memory statistics history" "show processes cpu history" and I noticed that the free ram is only 1,9 MB. Is true?
------------------ show memory statistics ------------------
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 4ED21E8 43263032 41276128 1986904 671744 1675528
I/O 6800000 8388608 6695596 1693012 1640240 1682456
Driver te 3000000 4194304 44 4194260 4194260 4194260
------------------ show process cpu history ------------------
333111111111111111111111111122222111112222211111222221111111
111222226666655555555553333388888333333333366666666662222266
100
90
80
70
60
50
40
30 *** ***** *****
20 *** *************** ***** ***************
10 **********************************************************
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per second (last 60 seconds)
222222382333222223223333222232222233222333223222322323323322
866797268213849680955650987667755210798201972877848270061888
100
90 *
80 *
70 *
60 *
50 *
40 * *** * * *
30 ************* ******************* *************** ********
20 ##########################################################
10 ##########################################################
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
434944444434333333334334434434433333344444333434333333333334334343333333
090243503271538485791732171194279769943711679093587357698985431739474668
100
90 *
80 *
70 *
60 *
50 * * * *
40 ************* * ****** **************************** ******** **** * *
30 **********************************************************************
20 ###########***********############*************#**********************
10 ######################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%
I want to try to upgrade the IOS version to 152-2.E5.bin (Cisco release recommended) and see if the problem will be solved.
Do you have some advice for my problem?
Thank you
Solved! Go to Solution.
11-03-2016 11:27 AM
Hi Leo
Your only option is to move away from 15.2(1)E3 to either 15.0(2)SE10 OR 15.2(2)E5. I personally prefer the 15.0(2)SE10 release.
Please read further for technical explanation for the above recommendation.
Here is the baseline info:
15.2(1)E3
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 4ED21E8 43263032 40392092 2870940 2139028 2493280
15.0(2)SE10
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 479AC7C 50830008 28935652 21894356 21020848 16768820
15.2(2)E5
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 4E4B42C 43815400 35757380 8058020 8012656 8021680
All of the above are taken from a WS-C3560CG-8PC-S without any features configured. Look at that the "Used" column.
It is the most for 15.2(1)E3 and it is the least for 15.0(2)SE10. 15.2(2)E5 is newer than 15.2(1)E3 and yet, its "Used" is less than 15.2(1)E3. This is the single/main reason why Cisco "suggests" to use 15.0(2)SE10 (I prefer this) OR 15.2(2)E5 (I will use this if I need features not available in 15.0(2)SE10).
The hw platform has 128MB DRAM and it cannot be upgraded. IOS images tend to grow over a period of time, which is what happened from 15.0(2)SE to 15.2(1)E trains. Then, users started reporting that the switch is pretty much useless because they cannot configure any features and in spite of configuring the least, basic features like SSh were not working. This feedback to 15.2(2) train where Cisco made conscious efforts to optimize the "Used" portion and release as much as possible for "free".
I hope this helps. As always, feel free to ask questions/clarifications and I will do my best to address them.
Kind regards ... Palani
11-02-2016 03:20 AM
Hi did you check when this is happening is there free ports in ---- show users
yes the mem is very low 1.9mb is there any logs showing mem issues or malloc failures ?
did you collect the show tech when the issue was occurring ?
if your hitting a bug an upgrade may help but if your just over utilizing it you may need more powerful switch
11-02-2016 03:46 AM
Thank for your answer.
I checked if it was a problem of free ports but it wasn't.
For example, I was in the device but my co-worker could not enter.
I checked log, but there wasn't nothing of interesting.
Yes I even collected the show tech when the problem was occurring.
Leo
11-02-2016 04:01 AM
it really sounds as if there was not enough mem to process your connections , can you post the show tech if you have it , to see the rest of the memory outputs
11-02-2016 04:09 AM
11-02-2016 05:02 AM
mem is very low , what was the show user count when this show tech was taken it should be in the output as well , reason I ask is cpu is seeing 16% just on ssh traffic , that's high -------------------------------------------------------------------------------------------- --134 10950 2380 4600 16.07% 1.58% 0.90% 1 SSH Process
Currently I have 4 ssh connections open to a switch as a test and the cpu is only showing .007% ssh
I would move to the safe harbour image that you have chosen above to start with see if it relieves some of the memory , everything else looks ok form the output nothing irregular
when you reboot how long does it take before the issue re-occurs , as reboot should wipe mem , what memory does it show as having after the reboot when running normally
11-03-2016 06:08 AM
Leo
You do not have sufficient memory, for sure. This is your problem, not the CPU. The momentary spike to 31% is simply because you typed show tech. To confirm memory is the problem, try connecting via console. It is likely you will see "low on memory, try again later" message.
We need to investigate why your memory is low. After IOS is loaded and config is applied, you are left with very little memory. This is not a sign of a software bug. So, a reload will not fix this.
The file named show tech is not really the show tech. Please familiarize yourself with
https://www.packet6.com/saving-your-putty-sessions-to-a-log-file/
This will capture the cmds you type to a log file. Please upload this file.
Your hw is C3560CG. This should not be running 15.1(2)E release. So, when you get a chance, please send the complete show tech. If you are unable to, please upload the output of show ver, show inv and show run.
Kind regards ... Palani
11-03-2016 06:41 AM
11-03-2016 06:41 AM
Hi Leonardo
The outputs you provided told us few things:
CPU is NOT your problem
Over 80% of your memory are used up right after boot
Your initial message mentioned you are runnign 15.2(1)E3 which left me scratching my head, asking "What's going on?".
This is the background for requesting additional information. I hope this clarifies my intent.
I will wait to see what I find from the show tech, once it is uploaded.
Kind regards ... Palani
11-03-2016 06:49 AM
Excuse me but the extension of files was .log and it wasn't allowed upload. Now it's all ok. Look up. Leonardo
11-03-2016 11:27 AM
Hi Leo
Your only option is to move away from 15.2(1)E3 to either 15.0(2)SE10 OR 15.2(2)E5. I personally prefer the 15.0(2)SE10 release.
Please read further for technical explanation for the above recommendation.
Here is the baseline info:
15.2(1)E3
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 4ED21E8 43263032 40392092 2870940 2139028 2493280
15.0(2)SE10
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 479AC7C 50830008 28935652 21894356 21020848 16768820
15.2(2)E5
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 4E4B42C 43815400 35757380 8058020 8012656 8021680
All of the above are taken from a WS-C3560CG-8PC-S without any features configured. Look at that the "Used" column.
It is the most for 15.2(1)E3 and it is the least for 15.0(2)SE10. 15.2(2)E5 is newer than 15.2(1)E3 and yet, its "Used" is less than 15.2(1)E3. This is the single/main reason why Cisco "suggests" to use 15.0(2)SE10 (I prefer this) OR 15.2(2)E5 (I will use this if I need features not available in 15.0(2)SE10).
The hw platform has 128MB DRAM and it cannot be upgraded. IOS images tend to grow over a period of time, which is what happened from 15.0(2)SE to 15.2(1)E trains. Then, users started reporting that the switch is pretty much useless because they cannot configure any features and in spite of configuring the least, basic features like SSh were not working. This feedback to 15.2(2) train where Cisco made conscious efforts to optimize the "Used" portion and release as much as possible for "free".
I hope this helps. As always, feel free to ask questions/clarifications and I will do my best to address them.
Kind regards ... Palani
11-04-2016 01:36 AM
Thank a lot for your answer. It's very clear and complete.
I'll try to change the IOS at 15.0(2)SE10 and I will see if the problem will be resolved.
Leonardo
11-03-2016 10:27 AM
If you can get in do a show users and see if there are any sessions hung and if so clear them otherwise to clear this issue you will probably have to reload the switch to free up resources .
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide