I've got 2 Nexus 3548 switches in 2 DCs. A door sensor is connected to each, via RJ45 over an SFP Module.
At some point in the past, both of them stopped responding. I didn't immediately notice, but now suspect they stopped when I updated the OS of the switches (it is now NXOS: version 9.3(8)).
I originally thought something with the sensor devices had gone wrong - a filled up cache inside of them or similar. Having them restarted (power cable pulled) did not work, so I left it alone
(the sensor devices luckily aren't critical for anything, just a security measure that isn't working).
This is a remote location for me, I am very rarely there. Recently I visited the data center though, and used that chance to take a closer look, planning to factory reset the sensor devices.
So I played around with them a bit, and found something unexpected:
- The link LEDs on the Nexus Switch and on the sensor devices are dark, so neither side recognizes they are connected to anything
- If I connect both the sensor device and my laptop to a cheap switch I had with me, I can connect to them! So it's not actually the sensor devices that broke.
- Exchanging the SFP Module, the Switchport the device connects to, or the RJ45 cable makes no difference.
- I can connect my laptop to the switches, but only if I don't put the cheap switch in between (if I do, the link LED stays dark) - so it's not actually just the sensor devices that get refused!
At that point I was out of ideas and out of devices to experiment with. Googling afterwards didn't turn anything up, neither did any diagnostics I could think of.
And so I am presenting to you a case of either me missing something simple (e.g.: You can't connect devices to the switch that don't support technology xy),
or something akin to a murder mystery: All the clues are there, you just have to thing around some corners to determine the culprit.
- We can eliminate one-off problems as culprits: We have two sensor devices and a cheap switch that all don't get a connection, to two different Nexus Switches
- We can eliminate the SFP Module, Switchport and cable as problems - exchanging them made no difference, and they work, if I use them to connect my Laptop
- We can eliminate networking problems as culprit - my laptop was in the same subnet as the devices, and the cheap switch is just passing through anyways
- It worked in the past. That makes certain explanations, like the SFP Module not passing through data for some devices, impossible
My only remaining idea is that the switches got more picky in which devices they accept through the software update.
But by which logic? Why can't I find anybody else having a problem like that?
What could I reasonably test here (please remember I am remote, making experiments requiring my presence difficult).
This is the brief status of the interface on the Nexus, though I don't it matters for this riddle:
Eth1/16 SERVER_MGMT notconnec 10 auto auto 1000base-T
Anybody willing to share a clue I would be most thankful to!
