Hi List,
we are running a lot of 3750 stacks and in the last few weeks we are faced with a strange phenomenon that now has affected the fifth switch stack in series. All affected switches had in common that they had an uptime of almost two years and an IOS 12.2(44)SE.
It starts with the switch complaining "%SNMP-3-INPUT_QFULL_ERR" on our syslog server for no reason (the switch gets the same snmp requests as every other switch on our network). If we ssh to the affected switch and do a "show interfaces status" it shows a couple of interfaces of the first switch, then the ssh session crashes. The same with "show etherchannel summary" .
If we reconnect to the switch again a "show users" lists the broken connection but a "clear line vty ..." does not reset it.
If we do a "show tech-support | redirect tftp:..." the ssh session from which we are doing this also crashes. The file on the tftserver ends with the interface at which "sho int statu" breaks.
At this stage the stack still seems to forward traffic but if we do a reload in a maintenance window things get worse: The switch that has been the stack master instantly crashes and does not recover. Forwarding stops and the management interface does not come back. Even the console is unusable. The only thing to remedy the situation is to unplug the mains cables. After that the switch comes back as though nothing had happened.
Has anyone heard of such a situation? Is there a way to predict that a switch will show this behaviour in the future? Is it safe to do a firmware upgrade on a switch that will run into this soon? Is there a way to prevent this remotely (without manually unplugging the stack)?
Thanks in advance,
Sebastian.
