ex-engineer,
I like to think I am a cut above when it comes to the technical side of my job, but I do agree with you.
Milan and others,
I managed to find the problem and solve it. It was many poorly designed things all conflicting with each other and causing the problem. First problem was revealed when one of the Vlans stopped passing traffic. The line and protocol showed up, but no traffic would traverse the vlan. After digging through the switches having the issue, I learned that said vlan would show nothing when I did a show spanning tree root. It had the correct interface and ID listed for the other vlans. This made it extremely difficult to track down the offending devices. When I looked at the priority of the root bridge on that switch, it showed a value of 1 with a bridge ID of 0001.0000.0000.0000. So, I looked at the ITN it was connected to and it also showed the same thing. I then kept tracing back until I found a switch with the proper information.
I began troubleshooting be enabling root guard on the ITN's outbound interface towards the offending switches. Once I did that, the ITN recieved the proper STP information. I finally traced it all the way down to a server that was setup on a trunk port with no native vlan specified. I compared the MAC to the Core's ARP table, figured out what IP it had and placed the server on a native trunk for that vlan. Once I did that, everything started working.
This got me thinking and I looked at the STP config on each core. The original designers had put 10 vlans with a priority of 8192 on Core 1 and 24 with a priority of 4096. Core 2's config was reversed. Once I took a good look at it, I noticed there were quite a few vlans missing from the equation (10 to be exact). They also setup each vlan with a standby configuration for HSRP. They mirrored each other but their priorities were backward compared to which switch was root (guess they thought lower number was active standby).
After correcting these mistakes, things began to run MUCH smoother. I set Core 1 with a priority of 4096 for all vlans and Core 2 with 8192. After looking through the rest of the switches and removing oddball spanning-tree vlan (range) priority 8192/4096 on edge switches, I was getting closer to having a completely functional STP network. I then found the bombshell. Someone had applied 'no spanning-tree vlan 1-1000' on a switch. I didn't expect this to cause problems, but once I fixed it, the whole network took a 3 second breath, and traffic began to flow. I was rather impressed with how fast it converged.
Now I am down to one last switch that isn't working properly, but isn't affecting the rest of the network. It shows a blank root for a vlan with no bridge ID or priority. This switch was showing an access-port as the root bridge (a Windows Server 2003 box is plugged into it) at one point. It was showing this with portfast, bpdufilter, and bpduguard enabled on the port. I really have no clue why it is doing that. I set the switch it is connected to with root guard on the outbound interface, but said switch still ignores all STP information from Core 1 on that vlan. It shows Core 1 as the root for all other vlans.
I even reduced the allowed vlans to a minimal configuration and still shows blank. When I apply 'debug spanning-tree all' to the switch and monitor it, not a single debug message pops up (even when I shut/no shut all interfaces). Why would this happen? A reboot doesn't resolve it either.
I may have to open a TAC case for this switch, I just hope it is on our Cisco contract...
In any event, I wanted to give everyone a very big thank you. This experience has been very educational.
Regards,
SrA Steven Denham
