Solved: Problem SSH on WS-C3560CG-8PC-S

leofalconi · ‎11-02-2016

Hello everybody,

The switch in question is:

WS-C3560CG-8PC-S

IOS 152-1.E3

RAM 130 MB

FLASH 57 MB

My problem is this:

the switch is in production environment and the ping is always ok but sometimes is impossible access in SSH/TELNET.

The output is Connection timed out.

I have checked more times the configuration and is all ok. The same configuration of AAA ecc is span on more switch and the problems is only this model of switch.

The CPU and RAM is monitored by SNMP but when I have succeed to access on SSH I typed "show tech" "show memory statistics history" "show processes cpu history" and I noticed that the free ram is only 1,9 MB. Is true?

------------------ show memory statistics ------------------

                         Head               Total(b)     Used(b)    Free(b)     Lowest(b)    Largest(b)
Processor        4ED21E8    43263032    41276128      1986904      671744     1675528
      I/O              6800000     8388608     6695596         1693012     1640240     1682456
Driver te           3000000     4194304          44            4194260     4194260     4194260

------------------ show process cpu history ------------------


      333111111111111111111111111122222111112222211111222221111111
      111222226666655555555553333388888333333333366666666662222266
100
   90
   80
   70
   60
   50
   40
   30 ***                         *****               *****
   20 ***     ***************     *****     ***************
   10 **********************************************************
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               CPU% per second (last 60 seconds)




      222222382333222223223333222232222233222333223222322323323322
      866797268213849680955650987667755210798201972877848270061888
100
   90        *
   80        *
   70        *
   60        *
   50        *
   40        *            ***     *                   *        *
   30 ************* ******************* *************** ********
   20 ##########################################################
   10 ##########################################################
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%




      434944444434333333334334434434433333344444333434333333333334334343333333
      090243503271538485791732171194279769943711679093587357698985431739474668
100
   90    *
   80    *
   70    *
   60    *
   50    * *                                *                   *
   40 ************* * ****** **************************** ******** **** * *
   30 **********************************************************************
   20 ###########***********############*************#**********************
   10 ######################################################################
     0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
               0    5    0    5    0    5    0    5    0    5    0    5    0
                   CPU% per hour (last 72 hours)
                  * = maximum CPU%   # = average CPU%

I want to try to upgrade the IOS version to 152-2.E5.bin (Cisco release recommended) and see if the problem will be solved.

Do you have some advice for my problem?

Thank you

Palani Mohan · ‎11-03-2016

Hi Leo

Your only option is to move away from 15.2(1)E3 to either 15.0(2)SE10 OR 15.2(2)E5. I personally prefer the 15.0(2)SE10 release.

Please read further for technical explanation for the above recommendation.

Here is the baseline info:

15.2(1)E3
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 4ED21E8 43263032 40392092 2870940 2139028 2493280

15.0(2)SE10

                Head    Total(b)     Used(b)     Free(b)   Lowest(b)  Largest(b)
Processor    479AC7C    50830008    28935652    21894356    21020848    16768820

15.2(2)E5

                Head    Total(b)     Used(b)     Free(b)   Lowest(b)  Largest(b)
Processor    4E4B42C    43815400    35757380     8058020     8012656     8021680

All of the above are taken from a WS-C3560CG-8PC-S without any features configured. Look at that the "Used" column.

It is the most for 15.2(1)E3 and it is the least for 15.0(2)SE10. 15.2(2)E5 is newer than 15.2(1)E3 and yet, its "Used" is less than 15.2(1)E3. This is the single/main reason why Cisco "suggests" to use 15.0(2)SE10 (I prefer this) OR 15.2(2)E5 (I will use this if I need features not available in 15.0(2)SE10).

The hw platform has 128MB DRAM and it cannot be upgraded. IOS images tend to grow over a period of time, which is what happened from 15.0(2)SE to 15.2(1)E trains. Then, users started reporting that the switch is pretty much useless because they cannot configure any features and in spite of configuring the least, basic features like SSh were not working. This feedback to 15.2(2) train where Cisco made conscious efforts to optimize the "Used" portion and release as much as possible for "free".

I hope this helps. As always, feel free to ask questions/clarifications and I will do my best to address them.

Kind regards ... Palani

View solution in original post

Mark Malone · ‎11-02-2016

Hi did you check when this is happening is there free ports in ---- show users

yes the mem is very low 1.9mb is there any logs showing mem issues or malloc failures ?

did you collect the show tech when the issue was occurring ?

if your hitting a bug an upgrade may help but if your just over utilizing it you may need more powerful switch

leofalconi · ‎11-02-2016

Thank for your answer.

I checked if it was a problem of free ports but it wasn't.

For example, I was in the device but my co-worker could not enter.

I checked log, but there wasn't nothing of interesting.

Yes I even collected the show tech when the problem was occurring.

Leo

Mark Malone · ‎11-02-2016

it really sounds as if there was not enough mem to process your connections , can you post the show tech if you have it , to see the rest of the memory outputs

leofalconi · ‎11-02-2016

Yes, I have attached the file.

Thank you

Mark Malone · ‎11-02-2016

mem is very low , what was the show user count when this show tech was taken it should be in the output as well , reason I ask is cpu is seeing 16% just on ssh traffic , that's high -------------------------------------------------------------------------------------------- --134 10950 2380 4600 16.07% 1.58% 0.90% 1 SSH Process

Currently I have 4 ssh connections open to a switch as a test and the cpu is only showing .007% ssh

I would move to the safe harbour image that you have chosen above to start with see if it relieves some of the memory , everything else looks ok form the output nothing irregular

when you reboot how long does it take before the issue re-occurs , as reboot should wipe mem , what memory does it show as having after the reboot when running normally

Palani Mohan · ‎11-03-2016

Leo

You do not have sufficient memory, for sure. This is your problem, not the CPU. The momentary spike to 31% is simply because you typed show tech. To confirm memory is the problem, try connecting via console. It is likely you will see "low on memory, try again later" message.

We need to investigate why your memory is low. After IOS is loaded and config is applied, you are left with very little memory. This is not a sign of a software bug. So, a reload will not fix this.

The file named show tech is not really the show tech. Please familiarize yourself with

https://www.packet6.com/saving-your-putty-sessions-to-a-log-file/

This will capture the cmds you type to a log file. Please upload this file.

Your hw is C3560CG. This should not be running 15.1(2)E release. So, when you get a chance, please send the complete show tech. If you are unable to, please upload the output of show ver, show inv and show run.

Kind regards ... Palani

leofalconi · ‎11-03-2016

Thank for your answer.

I know that these it isn't the complete "show tech", but I thought that those show was enough.

Anyway I have attached the entire files.

Leonardo

Palani Mohan · ‎11-03-2016

Hi Leonardo

The outputs you provided told us few things:

CPU is NOT your problem

Over 80% of your memory are used up right after boot

Your initial message mentioned you are runnign 15.2(1)E3 which left me scratching my head, asking "What's going on?".

This is the background for requesting additional information. I hope this clarifies my intent.

I will wait to see what I find from the show tech, once it is uploaded.

Kind regards ... Palani

leofalconi · ‎11-03-2016

Excuse me but the extension of files was .log and it wasn't allowed upload. Now it's all ok. Look up. Leonardo

Palani Mohan · ‎11-03-2016

Hi Leo

Your only option is to move away from 15.2(1)E3 to either 15.0(2)SE10 OR 15.2(2)E5. I personally prefer the 15.0(2)SE10 release.

Please read further for technical explanation for the above recommendation.

Here is the baseline info:

15.2(1)E3
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 4ED21E8 43263032 40392092 2870940 2139028 2493280

15.0(2)SE10

                Head    Total(b)     Used(b)     Free(b)   Lowest(b)  Largest(b)
Processor    479AC7C    50830008    28935652    21894356    21020848    16768820

15.2(2)E5

                Head    Total(b)     Used(b)     Free(b)   Lowest(b)  Largest(b)
Processor    4E4B42C    43815400    35757380     8058020     8012656     8021680

All of the above are taken from a WS-C3560CG-8PC-S without any features configured. Look at that the "Used" column.

It is the most for 15.2(1)E3 and it is the least for 15.0(2)SE10. 15.2(2)E5 is newer than 15.2(1)E3 and yet, its "Used" is less than 15.2(1)E3. This is the single/main reason why Cisco "suggests" to use 15.0(2)SE10 (I prefer this) OR 15.2(2)E5 (I will use this if I need features not available in 15.0(2)SE10).

The hw platform has 128MB DRAM and it cannot be upgraded. IOS images tend to grow over a period of time, which is what happened from 15.0(2)SE to 15.2(1)E trains. Then, users started reporting that the switch is pretty much useless because they cannot configure any features and in spite of configuring the least, basic features like SSh were not working. This feedback to 15.2(2) train where Cisco made conscious efforts to optimize the "Used" portion and release as much as possible for "free".

I hope this helps. As always, feel free to ask questions/clarifications and I will do my best to address them.

Kind regards ... Palani

leofalconi · ‎11-04-2016

Thank a lot for your answer. It's very clear and complete.

I'll try to change the IOS at 15.0(2)SE10 and I will see if the problem will be resolved.

Leonardo

glen.grant · ‎11-03-2016

If you can get in do a show users and see if there are any sessions hung and if so clear them otherwise to clear this issue you will probably have to reload the switch to free up resources .