Can't access the switch (memory errors)

[ryancisco01]

Hi so we have a 3750G stack of 3, can ping the device, can snmp push and poll the device and syslogs are flowing. However We cannot ssh, telnet, http or https (all ports time out except 80/443 which actively refuse). I am trying to ssh/telnet from another device on the same subnet, can see its arp, its directly attached off this other switch but all the connections timeout, even trying to telnet on port 22 for example just times out (if it was an auth issue, port would still be open right?). There is no acl on the vty line, ip ssh version 2 is enabled.

I don't have physical access to the device, but I have looked at its syslog and its reporting this:

Alternate Pool: None Free: 0 Cause: No Alternate pool

Pool: Processor Free: 846348 Cause: Memory fragmentation

-Traceback= 67A0E0z 22FE124z 230611Cz 2F73958z 219FCD0z 21A20B4z 251DA10z 251DAF0z 5CDF24z 5C8830z 685A5Cz 688D58z 688E90z 278F028z 2D94A9Cz 2D8EFD4z

-Process= "hulc running config", ipl= 0, pid= 389

However these messages are not frequent it last appeared 12 hours prior to my trying to login. It has a single Ip address and a default gateway

software is -

version: 15.2(1)E2

line vty 0 15
password redacted
login local
transport input all

Any ideas? I am sure a reboot will probably fix it but there are lot of links in use so not easy to arrange

[Joseph W. Doherty]

Probably a reboot is needed but might try limiting to just the current active master.

I suspect that's an early IOS release.  You might consider updating to a later/latest release in that IOS train.

[MHM Cisco World]

Before reboot 

Check which process eat cpu and memory' 

Snmp most process eat memory in ios devices 

MHM

[Joseph W. Doherty]

That's a great recommendation but if I understood OP correctly, there's no over the wire or console access.

[MHM Cisco World]

He can use console instead of ssh/telent 

MHM

[Richard Burts]

Using the console would be nice, but the original post is quite clear "I don't have physical access to the device". 

I see in the original post that there are traceback messages being sent in syslog. Traceback is always an indication of a software issue. So this switch is running software that has a problem. A short term solution would be to reboot the switch. The long term solution would be to upgrade the software running on the switch. But both of these require access to the switch which at present seems not possible.

Is there someone on site that could power cycle that switch? If the switch is power cycled it should run at least for a while, giving you opportunity to upgrade the software.

[Joseph W. Doherty]

"A short term solution would be to reboot the switch."

Yup, but as OP notes it's a stack of 3, you may only need to reboot the stack master.  Depending on your stack configuration, and how devices are connected to the stack, that might be the least disrupted method to restore full stack operation, although, it may require power cycling the whole stack.

Rick is correct, tracebacks generally indicate bugs, and yours might be one of the memory leak variety.  Which is why we've both suggested updating your IOS (which can be tricky for a 3750G as they are EoL).

I recall @Leo Laohoo has, in the past, mentioned one of the later Catalyst switch IOSs can be used on the earlier 3750 models.

[Leo Laohoo]

3750G can support 12.2(55)SE12 or 15.0(2)SE11.

Both versions are well known amongst the industry as the most stable firmware and highly recommended.  Do not make any attempts to use any other trains other than 12.2(55)SE or 15.0(2)SE.

[Joseph W. Doherty]

---

@MHM Cisco World wrote:

He can use console instead of ssh/telent 

MHM

---

OP also noted, no console access (as it's remote).

[ryancisco01]

Thanks all for the suggestions!

I can arrange for remote hands to plug a console into it, but my feeling is I probably won't be able to achieve much by doing so unless I have an outage window to try a reboot, I think the best course of action will be to plan a firmware upgrade in an approved window and use remote hands with a console cable.