Ahh, I see.  Yes, you've changed my mind; I *am* interested in MTBF figures for power supplies now (not published for most gear, regrettably). OK, I'm beginning to glimpse how hard this is.  Minimally, I need to know a lot about those MTBF guesstimates are calculated ... that's hard information to dig up, as far as I can tell.  Plus, I would need more MTBF figures, minimally around power supplies. BTW:  The point of the exercise isn't actually improving uptime; it is helping mgmt understand what they win (and lose) as they traverse the cost chain, from Cat2960X to Cat35XX to Cat4500. They don't like the fact that a lot of our time goes to fixing Cat2960X problems; they want us spending our time elsewhere.  As low-tech folks, they have been, quite naturally, focused on acquisition cost; I want to help them make a more nuanced trade-off, between acquisition cost and on-going staff time, as we make decisions on what to buy for the next building. I think I'll end up doing something like this: * Stackable approach tends to save you $$ both on acquisition and SmartNet * But you spend more on operations, as you've bought way more parts.  For example, a chassis-based IDF contains (2) power supplies, while a Stackable based IDF (given our port densities) contains (8), given the Cat2960X (or 16, given a higher-value stackable).  More parts, more failures.  One can mitigate this by buying a Stackable model which sports dual-power supplies ... the power supply failure rate doubles, but the urgency of replacing a failed PS goes down. * Similarly, trouble-shooting tends to be more complex, consume staff time, and inflict more end-user downtime (we seem to hit these "a single component affects the whole Stack" issues more than our Cisco Account Team believes is typical).  Complex failures on chassis' tend to involve the Sup card -- and the fix tends to be fairly simple:  replace the Sup card.  [Although, the whole IDF is down during this process.]  Complex failures on Stackables are hard ... there are (8) Members each with their own brains, (8) Stacking Cables, and (8) Stacking Modules, and binary search on an intermittent problem takes a lot of both calendar & staff time to resolve (although typically at least some ports / users are unaffected during this process). * Perhaps reliability improves as one climbs the value chain in Stackables (Cat2960X ... Cat3650... Cat3850) ... but I don't know how to assess this, so I won't go there. Joseph -- thanx for helping me think through this.  If you have further input, do let me know; else, I'll ride off to tilt at other windmills. --sk ... View more

I appreciate your helping me think through this Joseph. - Seems to me that one way of thinking about MTBF is "how long can I run until the first hardware element fails".  So, for example, how long can I run until the first power supply fails (I'm picking power supply since, anecdotally, I would claim that fans / power supplies are the things which fail most frequently).  From that perspective, then I claim my ~29,000 hours calculation is on the right track:  the more power supplies / fans I own, the more likely that at least one of them will fail on any given day. - I propose that the way you are thinking about MTBF is:  "how long can I run until the entire Stack is down?"  And yes, I can see how adding Members to a Stack might *increase* MTBF -- as each Member fails, another Member takes over the Master role, keeping the Stack functioning. In fact, in some sense, the Stack is still up & running just fine even if (7) of its (8) Members are down.  [Although a user attached to a failed Member might struggle to agree with this perspective.] - My mgmt cares about service disruption to end-users; they are less comforted by my claim of "Well, most folks are still running fine; it just the (48) users attached to the failed Member who are out of action" and more focused on "How do we get those (48) users up & running?"  So I am focused on Service Disruption ... i.e. "MTTSD",  Mean Time To Service Disruption. - For the Cat2960X, failure of a power supply means service disruption, so again, that ~29,000 figure seems to me to be on the right track.  For most other platforms, equipped with redundant power supplies ... I would then say ... well, MTBF to the failure of the first power supply ... isn't particular interesting.  In fact, at that point, I would only care if *both* power supplies were down at the same time, causing service disruption. - Still, given my perspective of wanting to calculate MTTSD , I see several challenges that at this moment I do not know how to surmount: (a) Does that C4K 179,000 number count MTBF including power supplies?  or MTTSD (the chance of a failed power supply causing a Service Disruption being fairly small, given a dual-power supply chassis)? (b) How to include the effect of failing Stacking Cables & Modules?  I suspect that the Cat2960X 233,000 hour figure does *not* include this influence? (c) How to include my own experience ... in my experience, entire Stacks are severely degraded by the (intermittent) failure of a single element ... typically a failing Member ... I'm working a TAC case right now where we suspect intermittently failing RAM / TCAM in one of the Members ... sometimes a failing Stacking Cable or Module.  Is my experience just an outlier (I started managing network gear in 1991), in which case, we ignore this contribution.  Or is my experience fairly typical, in which case, I very much want to include the effect that the more Members one adds to the Stack, the more likely that the entire Stack will suffer consequences when a single element degrades. How would you push back on this thinking? --sk ... View more

OK, my first error was typing a decimal point instead of a comma ... that threw off the arithmetic by a factor of a thousand.  And I swapped the Cat2960X MTBF with the Cat4500 MTBF. Here is a revised calculation: Lamba 1 = 1,000,000 / 233,370 = 4.3 Lamba 2 = 1,000,000 / 233,370 = 4.3 Lamba 3 = 1,000,000 / 233,370 = 4.3 Lamba 4 = 1,000,000 / 233,370 = 4.3 Lamba 5 = 1,000,000 / 233,370 = 4.3 Lamba 6 = 1,000,000 / 233,370 = 4.3 Lamba 7 = 1,000,000 / 233,370 = 4.3 Lamba 8 = 1,000,000 / 233,370 = 4.3   Lambda Stack = 4.3 * 8 = 34.4 MTBF Stack = 1,000,000 / 44,512 = 29,070 hours And to your point, this approach assumes that the failure of any Member takes out the entire Stack ... I suspect that this is happening on one of my Stacks now ... that a failing Member starts a cascade of events which results in the entire Stack hanging ... but generally is not true:  generally, a single Member can fail without impacting the rest of the Stack. So another approach would be to simply take the MTBF of a single Member -- 233,370 hours -- and divide by (8) (eight Members in a Stack): 233,370 / 8 = 29,171 hours Perhaps by chance, this number is awfully similar to the previous approach. Both approaches ignore the influence of Stacking Cables & Modules. And then, for comparison, I propose that the failure rate of a fully-loaded Cat4500 (eight line cards) would be ~179,000 hours. So, comparing a (384) port Cat2960X IDF with a Cat4510 IDF: ~29,000 hours MTBF vs ~179,000 hours MTBF With the Cat2960X MTBF likely lower than that figure, on account of the missing Stacking hardware. This seems at least plausible.  But I realize this is a complex space, and I'm naive to it.  Have I stumbled somewhere? --sk ... View more

OK, I've read the Cat6500 white paper, which sketches out how hard this is. I also read up on calculating MTBF with multiple components: http://answers.google.com/answers/threadview?id=390140 And now I'm ready to try my hand at this. Recall: 233,370 hours:  MTBF for Catalyst 2960X-48FPD-L http://www.cisco.com/c/en/us/products/collateral/switches/catalyst-2960-x-series-switches/data_sheet_c78-728232.html •    Assume that this figure covers a stand-alone Cat2960X, for a total port count of (48). 179.714 hours:  MTBF for Catalyst WS-C4510R+E:   http://www.cisco.com/c/en/us/products/collateral/switches/catalyst-4500-series-switches/product_data_sheet0900aecd801792b1.html •    Assume that this figure covers a Cat4510R equipped with redundant power supplies and (8) line cards, for a total port count of (384). Lamba 1 = 1,000,000 / 179.714 = 5564 Lamba 2 = 1,000,000 / 179.714 = 5564 Lamba 3 = 1,000,000 / 179.714 = 5564 Lamba 4 = 1,000,000 / 179.714 = 5564 Lamba 5 = 1,000,000 / 179.714 = 5564 Lamba 6 = 1,000,000 / 179.714 = 5564 Lamba 7 = 1,000,000 / 179.714 = 5564 Lamba 8 = 1,000,000 / 179.714 = 5564 Lambda Stack = 5564 * 8 = 44,512 MTBF Stack = 1,000,000 / 44,512 = 22 hours Since this calculation ignores Stacking Cables and Stacking Modules, the actual figure would be much lower. Now, stepping back for a moment – I don’t believe this answer.  I don’t believe that an (8) Member Catalyst 2960X Stack will, typically, experience a hardware failure every 22 hours So, clearly, I'm fumbling something.  Has anyone else tackled this problem, can offer insights as to how I'm derailing? --sk ... View more

TAC tells me that they ignore a Fail on this test when it is run On-Line and investigate when it shows up on POST. I also note this defect:  https://bst.cloudapps.cisco.com/bugsearch/bug/CSCtl08303/?referring_site=bugquickviewredir, which to my read basically says "someone else has observed this also" ... View more

As an aside, I note that this Stack logs these sorts of messages, when it is crumping: 2017-01-30T15:01:55.019979-08:00 5n-esx-mgmt 990: 000980: Jan 30 15:01:54.543 ps t: %SUPQ-4-PORT_QUEUE_STUCK: Port queue Stuck for asic 1 port 14 queue 0 2017-01-30T15:01:55.021329-08:00 5n-esx-mgmt 991: 000981: Jan 30 15:01:54.543 ps t: Error disabling queue 0 for asic 1 port 14 2017-01-30T15:02:36.012774-08:00 5n-esx-mgmt 999: 000989: Jan 30 15:02:35.996 ps t: %XDR-3-XDROOS: Received an out of sequence IPC message. Expected 9373 but got  9369 from slot 5. 2017-01-30T15:02:42.566506-08:00 5n-esx-mgmt 1026: -Traceback= 555EECz 7139Cz 76 58Cz 8A768z 102113Cz 1021C0Cz 1025CF4z 1026A58z 1027280z 1027A14z 28252C4z 275BA4Cz 275E770z 2743F78z 273B524z 273B4CCz 2017-01-30T15:06:46.324424-08:00 5n-esx-mgmt 1827: 001406: 000035: Jan 30 15:06: 45.249 pst: %XDR-6-XDRLCDISABLEREQUEST: Client XDR Interrupt Priority Client req uested to be disabled. Due to XDR Keepalive Timeout (5n-esx-7) But again, I think the specifics are not so important; rather, I'm interested in methodology for isolating bad hardware. --sk ... View more

OK Julio, so you are pointing out that 'show switch' output can reveal some level of problems: 5n-esx#sh switch detail      Load for five secs: 39%/0%; one minute: 40%; five minutes: 42% Time source is NTP, 06:40:35.045 pst Tue Jan 31 2017 Switch/Stack Mac Address : 00da.553e.f080                                            H/W   Current Switch#  Role   Mac Address     Priority Version  State ---------------------------------------------------------- *1       Master 00da.553e.f080     15     4       Ready                 2       Member 38ed.1813.2780     14     4       Ready                 3       Member 38ed.1812.e500     13     4       Ready                 4       Member 38ed.1813.7980     12     4       Ready                 5       Member 38ed.1813.7900     11     4       Ready                 6       Member 1cde.a7a9.3f80     10     4       Ready                 7       Member 00da.5513.4b00     9      4       Ready                 8       Member 0038.df04.b600     8      4       Ready                         Stack Port Status             Neighbors      Switch#  Port 1     Port 2           Port 1   Port 2 --------------------------------------------------------   1        Ok         Ok                2        8   2        Ok         Ok                3        1   3        Ok         Ok                4        2   4        Ok         Ok                5        3   5        Ok         Ok                6        4   6        Ok         Ok                7        5   7        Ok         Ok                8        6   8        Ok         Ok                1        7 5n-esx#sh switch stack-ports Load for five secs: 39%/0%; one minute: 40%; five minutes: 42% Time source is NTP, 06:40:38.869 pst Tue Jan 31 2017   Switch #    Port 1       Port 2   --------    ------       ------     1           Ok           Ok        2           Ok           Ok        3           Ok           Ok        4           Ok           Ok        5           Ok           Ok        6           Ok           Ok        7           Ok           Ok        8           Ok           Ok    5n-esx# And the URL you reference illustrates using 'show platform stack-manager all' output to point toward other issues (version mismatches, for example): 5n-esx#              show platform stack manager all Load for five secs: 40%/0%; one minute: 43%; five minutes: 43% Time source is NTP, 06:36:08.033 pst Tue Jan 31 2017 Switch/Stack Mac Address : 00da.553e.f080                                            H/W   Current Switch#  Role   Mac Address     Priority Version  State ---------------------------------------------------------- *1       Master 00da.553e.f080     15     4       Ready                 2       Member 38ed.1813.2780     14     4       Ready                 3       Member 38ed.1812.e500     13     4       Ready                 4       Member 38ed.1813.7980     12     4       Ready                 5       Member 38ed.1813.7900     11     4       Ready                 6       Member 1cde.a7a9.3f80     10     4       Ready                 7       Member 00da.5513.4b00     9      4       Ready                 8       Member 0038.df04.b600     8      4       Ready                         Stack Port Status             Neighbors      Switch#  Port 1     Port 2           Port 1   Port 2 --------------------------------------------------------   1        Ok         Ok                2        8   2        Ok         Ok                3        1   3        Ok         Ok                4        2   4        Ok         Ok                5        3   5        Ok         Ok                6        4   6        Ok         Ok                7        5   7        Ok         Ok                8        6   8        Ok         Ok                1        7                Stack Discovery Protocol View ============================================================== Switch   Active   Role    Current   Sequence   Dirty Number                    State     Number     Bit -------------------------------------------------------------------- 1        TRUE    Master   Ready       250       FALSE 2        TRUE    Member   Ready       130       FALSE 3        TRUE    Member   Ready       194       FALSE 4        TRUE    Member   Ready       195       FALSE 5        TRUE    Member   Ready       228       FALSE 6        TRUE    Member   Ready       206       FALSE 7        TRUE    Member   Ready       131       FALSE 8        TRUE    Member   Ready       083       FALSE                             Stack State Machine View ============================================================== Switch   Master/   Mac Address          Version    Current Number   Member                          (maj.min)  State    ----------------------------------------------------------- 1        Master    00da.553e.f080          1.56        Ready      2        Member    38ed.1813.2780          1.56        Ready      3        Member    38ed.1812.e500          1.56        Ready      4        Member    38ed.1813.7980          1.56        Ready      5        Member    38ed.1813.7900          1.56        Ready      6        Member    1cde.a7a9.3f80          1.56        Ready      7        Member    00da.5513.4b00          1.56        Ready      8        Member    0038.df04.b600          1.56        Ready      Last Conflict Parameters Switch Master/ Cfgd Default Image H/W  # of    Mac Address    Number Member  Prio Config   Type Prio Members                     -----------------------------------------------------------------------  1     Master   15   No       4    5   6  00da.553e.f080  3     Member   13   No       4    5   0  38ed.1812.e500  4     Member   12   No       4    5   0  38ed.1813.7980  5     Member   11   No       4    5   0  38ed.1813.7900  6     Member   10   No       4    5   0  1cde.a7a9.3f80  7     Member   9    No       4    5   0  00da.5513.4b00  8     Member   8    No       4    5   0  0038.df04.b600 And the URL mentions other possible clues -- e.g. Stack Master flapping, Stack Partitioning -- neither of which affect. I think I'm working a different space, though: - That URL does not mention the pathology I'm seeing - And in any case, as far as 'show switch' and 'show platform stack-manager all' go, this Stack is fine. - Except that this Stack most definitely is not fine. I'm wonder if I'm bumping into one of the challenges in trouble-shooting a distributed system (i.e. numerous brains ... (8) in this case ... collaborating) -- that this is just hard ... perhaps binary search is the only effective solution. Anyway, other insights into how to trouble-shoot, generically, a Stack issue? --sk ... View more