SOLVED:
Today, we discovered gradually increasing packet loss to one of our remote network switches located at an unmanned and locked facility. The degredation resulted in the loss of voice and data communications. Remote management was also lost. Local monitor event logs, syslogs and graphs revealed the degredation had been increasing over a period of hours. The remote site's router appeared unaffected and was still reachable. The following troubleshooting commands were issued on the router:
- show ip interfaces brief (all physical and virtual sub-interfaces associated with the switch trunk showed an "up/up" status)
- show interface [physical interface trunked to the switch] (no errors were present, but the counters were not moving, either, not even after the interface counters were cleared; yet the "up/up" status persisted)
- show ip route (to verify an active path back to the core router; the same command was initiated from the core router to verify a path to the remote site)
- show log (on the remote site's router; no unusual activity had been recorded)
A field support team was dispatched to the remote location. Their findings:
- The network cable between the site's router and the network switch was tightly twisted as if it had been wrung (apparently this way since the time of installation; it had not been tested, and considering the obvious discrepancy, it should never have been used)
- The network cable for one of two WiMax Redline RDL-300-Edge radios connecting the downstream switch was not fully seated/locked into the switch port
- The network cable for the second WiMax Redline RDL-300-Edge radio connecting the downstream switch was fully seated, but the link was flapping (indicated by a flashing ambler light)
- Further investigation into the dual WiMax setup revealed a spanning-tree conflict between the primary and secondary RDL-300-Edge radios despite offset cost settings between the VLAN's
- After shutting off one WiMax connection, the other Redline continued to struggle with the spanning-tree protocol, but at least the downstream switch had become reachable
- During troubleshooting, the downstream switch's global spanning-tree command was changed from "spanning-tree mode rapid-pvst" (a federal STIG requirement) to "spanning-tree mode pvst." The same change was then made to the primary site's network switch. After approximately one minute, the active WiMax connection renegotiated/re-synced, and it was stable; however, the secondary WiMax connection would not successfully re-integrate after being re-enabled, so it was returned to a disabled state pending further troubleshooting.
The remote site's router and network switch and the downstream site's network switch are remotely reachable again and are passing data.
