STP causes four day outage...

[jimmysands73_2]

Posted on Cisco Learning Network.....I really enjoyed the reading and learning.  Some other may have heard about this case.

They actually had to an ER due to the outage.

[jimmysands73_2]

Yawn...its early this moring in Vegas

>>Some other may have heard about this case.  They actually had to an ER due to the outage.

meant to say

Some others may have heard about this case.  They actually had to close an ER due to the outage.

[siddhartham]

Thanks for sharing..

Siddhartha

[Peter Paluch]

Jimmy,

I have went over the documents briefly, and I find them... I do not know. I do not like some of the reasoning I see there. Clearly, the network at the hospital was not designed using today's best practices which is logical, considering its age. But the document seems to me to be blaming STP for all the trouble, and it focuses too much on the fact that this STP appeared to fail in a network whose diameter was 10 and not 7. I'm sorry... the magic '7' regarding STP is related to a very conservative computation of default STP timers. But there is no technical reason why STP couldn't work in larger topologies even with no tweaking of the timers (up to roughly 17 switches in diameter!). Switches process BPDUs much faster than in order of seconds. The RSTP would be, in my opinion, even more stable.

So... yeah, an outdated design coupled with inherent STP shortcomings was clearly a cause of the problems described in the PDFs (including some buggy firmware or hardware if I correctly recall seeing that mentioned in the materials), but the technical explanation doesn't ring to me. Perhaps it was written for popular public, not for networkers...

Best regards,

Peter

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Peter - I read the references too, and my "take away" was the CIO's dog eat his homework, oops I meant the CIO's STP eat his network.

What I read, appeared to be, to me, network "dry rot".  If "engineering" or "materials" were substandard, who's responsible?  Management.

However it's good to know even ". . . a star among stars." can learn and recommend things such as "Document all changes, including keeping up-to-date physical and logical network diagrams."

Of course, you also "learn" (?) "Spanning tree protocol is like a traffic cop. Data arrives at a switch and asks spanning tree for directions."  My understanding of STP differs a little from this description.  Perhaps it's just an over simplification for lay people reading this document, but perhaps that's the actual technical understanding of management or even the "engineers" supporting the network.  (One hopes not the latter, but if STP diameters of 10 were implemented without consideration of the possible implications, one can wonder.)

I don't mean to disparage management, it's a tough job (and someone needs to do it) but like other professions it can be done well, or not so well.  Maybe John Halamka was a fine ER physician, and the expert to consult on mushroom and wild plant poisonings, but maybe he might not be the best choice to fix a leaky pipe or be a CIO.

Anyway, Peter, I'm so glad to read your comments, as this is one I wouldn't have commented on without seeing yours.  With too many decades in the industry trenches, one can get just a little cynical.

[turnera]

A good read for sure.

My take away was somewhat in line with Joseph's. While it is being reported as an STP problem, it was more of a "hey lets bolt on this new network right here and it will work." After a while the bolting of networks was the undoing of an old switch based topology.

Spanning tree, while very powerful and capable of doing what it is intended to do, could not keep up with the network bloat that it found itself in. A switch powered off here or a faulty NIC there and the next thing you know loops for everyone and data for none!

It was a good read though. Thanks for the post.

[Peter Paluch]

Hello,

Actually, I believe the fundamental flaw of Ethernet technology is the flooding which is performed whenever the recipient is unknown. Hence, any redundancy coupled with flooding will necessarily result in switching loops. We use STP as means to remove the redundancy at the logical level but as STP is a mechanism in its own right and may fail, its failure causes the redunancy to be reinstated and the flooding will kill the network. STP is often blamed for things it wasn't able to prevent. Principially, STP is a distance-vector routing protocol towards the bridge. Like any other routing protocol, STP can fail. However, while a routing protocol failure will result in networks being unreachable without routing loops occuring, STP failure will inevitably lead to switching loops. So the blame is really on the flooding concept in Ethernet.

This is what makes TRILL and FabricPath networks so different - there is no flooding anymore. There is thus no concept of a switching loop we know from our traditional Ethernet networks.

Best regards,

Peter

EDIT: See my post below.

[Peter Paluch]

Hello everyone,

After a discussion with Joseph Doherty, I believe my former statement of flooding being the "fundamental flaw of Ethernet technology" is a poor choice of words. It is too strong a statement. The flooding concept may not be the ideal, considering today's knowledge and technological advancement, but in the Ethernet beginnings, it was an absolutely logical thing to do.

Joe - thank you for correcting me and pointing out the need of being more cautious with my words, and my judgements!

Best regards,

Peter

[Joseph W. Doherty]

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

Also to everyone, Peter, I believe, is being a little too critical on himself.

In sidebar discussion with Peter, I asked him why he considered Ethernet flooding flawed.

Peter's thinking was Ethernet's flooding could be a flaw if you're trying to design to today's networking needs, not that it was necessarily flawed at the time it was originally designed.

I'm a bit embarrassed that my side bar question prompted Peter to correct his prior post, and post his note above, but as anyone who routinely reads Peter's posts knows, he strives for clarity and 100% accuracy.