Hi. On a big industrial plant we've replaced an old HP switch with a brand new couple of C2960x switches in stack configuration and ever since then, every 6/8 hours or so, the two fiber optics links of switch #2 go down at once. These are connected to a ring of 3 similar other access switches, that never detect their interfaces going down, so the issue is on the rx side from the point of view of switch #2. Also note that the links on switch #1, connected to the main network, never go down.
Traffic is on average very low as the switches are connected to industrial equipment that requires very little bandwidth. No errors on the interfaces, the fiber links have been checked and they are in good condition. IoS is the latest recommended. Both CPU and memory usages are low. Optical tx and rx powers are at good levels. There are no processes that stop running after the event occurs.
We've replaced the SFP modules. We've replaced switch #2. We've installed an UPS and checked the power source. I've been looking at several kind of debug outputs but so far nothing seems to be of any help, I can't find anything unusual. Note that the *only* way to make the links come up again is to reload switch #2. Shut/no shut does nothing. Disconnecting and reconnecting the cables does nothing. Removing and reinserting the SFP does nothing.
It seems that the whole fiber optic controller stops working for some external cause, like there's something that troubles the controller to the point that it just hangs after a while, after a buffer overflows or something. We can't disconnect the hosts on the ethernet ports as a test, as the connected equipment needs to run 24x7. None of this ever happened with the previous HP switch, it seems that whatever is going on with the network didn't bother it one bit.
We've been plagued by this issue for months and nothing seems to solve it. How do I troubleshoot it further?
