2960X lost contact with switch but working

[Sazwon]

I have a 2960X that i lost monitoring on yesterday.  When i went to look at the core switch it's connected to i saw activity on the port and when i look at the core it shows the port up up but when i run cdp neigh it doesn't show the edge switch and I can no longer ping the edge switch or otherwise remote into it.  I went to look at the edge switch and the port is dark no activity light however everything connected to the switch is still functioning and responding to pings.  I did replace the glc on the fiber port and got the same situation, working devices but nothing on the port. no activity or contact with the switch remotely.  There wasn't a power outage or anything that would have made the switch reboot and the current configuration has been on the switch for over a year with multiple restarts meaning losing it's config if changes had been made and not saved.  I'm really the only person who admin's this switch.  I haven't been able to console into the switch yet to look at the config that's something i'll do on a weekend, but was wondering if anyone else had seem a symptom like this and had some thoughts.

[marce1000]

- Usually network-admins also have simple managerial ip-access to switches, to verify it's operation.

 M

[johnd2310]

Hi,

What ios version is running on that switch? Could be a bug. Have you restarted the switch or looked at the logs?

Thanks

John

[Sazwon]

running 15.2(7)E1, RELEASE SOFTWARE (fc4) 

[Sazwon]

haven't restarted yet, the switch is in production and taking it down at this time would cause major issues... the log up until the moment monitoring ended showed cpu memory usage at 27% and memory at 21%

[Leo Laohoo]

What you've just describe fits the description of a memory leak which is about to crash.
Passing traffic from one port to another is the most basic function of a switch, however, all the management portion sits on the CPU. IF the CPU or memory runs out of resources, the first thing that will "give up" are the management portion (ping, telnet/ssh, console, etc.).

[Deepak Kumar]

Hi,

Seems like a bug or memory leak issue. What is the IOS version? Are you collecting logs on any Syslog or other servers? Check the last few hour's logs for any sign high CPU or other resources.

[Sazwon]

15.2(7)E1, RELEASE SOFTWARE (fc4)  and i'll go back through log and see what i can see for cpu usage

[Joseph W. Doherty]

Yea, I've encountered that from time-to-time. Something (as noted by others) has caused the issue. The only short term fix I've found, is a power off/on cycle of the device. As the switch might continue to properly forward traffic, you may be able to defer the power recycle to a non-peak time, but I wouldn't defer that longer that really necessary.

It's also been my experience, that without any further changes to the device, you may not see the issue repeat. This because, it might be a really rare set of circumstances that causes the issue.

Of course, if the same device starts to keeping showing this behavior, you'll want to try to find a long term way to eliminate the issue. For example, if the problem is caused by low memory, not from a memory leak, adjusting some features of the device can reduce memory usage, such as decreasing the size of the RAM system log. Understand, IOS generally does not have a "garbage collector", so running low on free RAM, causing lots of fragmentation, but not actually using all of it, can cause all kinds of problems (as further memory allocation requests fail).