MS350 Stacking Issues

[brian holmes]

I have 3 MS350 stacks, 6 switches in each stack, running on 8.10 software.

All stacks have been up and running without issue for the past month or so. No changes have been made on the Meraki switches or the Cisco Nexus 9k's that are upstream.

This morning within an hour all 3 stacks had switches that removed themselves from the stack and stopped forwarding traffic.

Stack 1 - 1 switch disappeared from the stack and stopped forwarding traffic - Did not come online until after a reboot. Uplink port was not on this switch.

Stack 2 - 1 switch disappeared from the stack and stopped forwarding traffic - Did not come online until after a reboot. Uplink port was not on this switch.

Stack 3 - 5 of the switches disappeared from the stack - 1 of the uplinks was on one of the switches that disappeared from the stack. After rebooting the switch with the uplink the 5 switches rejoined the stack. 1 switch in this stack stayed online, that switch has an uplink to the core.

It seems really strange that 3 stacks had the same issue within an hour, Meraki support suggested we hit a bug (8.10) and have requested I upgrade to 9.19.

Has anyone else been having stacking issues?

I typically put in Cisco 4500's or 3850's and these are my first 3 MS350 stack closets. This obviously isn't giving me a very warm and fuzzy feeling to have a failure like this after a month or so.

Curious what other users experiences have been when using MS350 stacks ???

[BHC_RESORTS]

Firmware updates are the only thing that somewhat scares me on Meraki products. Not having your trusty serial connection to xmodem over an update in case it didn't work...99% of the time, they go smoothly, but when they don't...

[MRCUR]

Stacks dropping their uplink (and then doing absurd things afterwards) is a well known issue on the MS team from what I've gathered. The uplink being dropped is specifically called out as a bug fix beginning with firmware version 9.27 which was just released.

[MRCUR]

As a followup from my original post mentioning 9.27 - I now have three buildings running 9.27 and it's been fantastic. Easily the best MS firmware I've used so far. Stacks are actually behaving correctly for the first time ever, including stacks with uplinks spread across switches.

[dayoder]

Excellent. That's good to hear. Thanks for the update.

[Mr_IT_Guy]

Thanks for the update!! Might have to talk with the boss on this one.

[brian holmes]

I have been on 9.26 for a few weeks now, I have had one issue.

The building had a power outage taking my 3 MS350 stacks down. 2 of the stacks came up without issue, 1 of the stacks had 2 switches that didn't join the stack. That stack had to be rebooted manually and then all switches came up in the stack.

I will be upgrading to 9.27 in the near future.

[MRCUR]

@brian holmes 9.27 seems to be much better at recovering from this issue. While in the past I had to reboot stacks where they were reporting incorrect membership after power outages, on 9.27 this alert has been resolving itself or not happening at all.

[brian holmes]

Unfortunately 9.27 has not resolved the stacking issue I am having.

OSPF goes up then down and switches in the stack without their own uplink can fall out of stack.

I have a maintenance window scheduled for this weekend and will engage support so hopefully they can see this issue happen in real time.

[Mr_IT_Guy]

Soo I took my network to 9.27 per the suggestion of a tech for another unrelated issue. When we pushed it out to one of our networks, it broke my routing. Ended up rolling back to 8.10.

[MRCUR]

@Mr_IT_Guy I think routing not working is listed as a known issue for 9.27 now. It's been fine for me on MS350's though for both local interfaces and OSPF.

[brian holmes · Accepted solution]

@Mr_IT_Guy- Upgraded my MS350 stacks to 10.14 a few weeks back and so far they have been solid. The original issue was nmap scan's would cause some switches in a MS350 stack to become unreachable, they would also no longer forward traffic. The only way to bring the switch back online was to reboot it.

Here is the reported fix from the Meraki engineering team.

"The fix was to implement a rate-limiter in hardware to avoid the CPU from being hampered by the scans."

[steven.lane]

8.10 with physical stack cables upgrade ASAP. I do find it strange that 8.10 is so called stable release when it has so many bugs vs the so called beta versions that address most the issues.

[MerakiGnome]

This has certainly got me worried now. Recently deployed five stacks of approx 6-7 switches per stack and we've already seen some of these funnies occur. Will have to look at upgrading the stacks but the upgrade process withou a trusty blue console cable worries me!

[pbrischetto]

I'm also having issues with stacked MS350 switches however it's a different (yet equally concerning issue).

I have 2 stacks; Stack 1 and Stack 2. Stack 1 has 2 switches and stack 2 has 3 switches. Between the 2 stacks I have Link Aggregation (LACP) setup using 2 x 10Gb fibre. These connections are split between the 2 switches in stack 1 and are plugged into switches 1 & 3 in stack 2.

So connected as below:

Stack 1 / Switch 1 --- Stack 2 / Switch 1

Stack 1 / Switch 2 --- Stack 2 / Switch 3

Stack 1 / Switch 1 then connects to my MX.

I then take my laptop and plug it into Stack 2 / Switch 2 and run a constant ping to both the MX and Google.

If I then pull power to Stack 2 / Switch 1 I lose pings to the MX & Google for 3-4 pings. Being a LACP Link Aggregation I would expect to lose 1 ping at the most. The other strange thing is that I also lose 3-4 pings when Stack 1 / Switch 1 comes back online.

Same thing happens also if I kill power to Stack 2 / Switch 3 or Stack 1 / Switch 2.

Logged the issue with Meraki support who have run me through numerous firmware updates but none of these have resolved the issue. They have now escalated it to their engineers.

Thankfully the switches are still in my lab and are not due to be installed until later this year but I don't think I'll be installing them until this is fixed.

[brian holmes]

What version of firmware are you on?

I'm taking my entire environment to 9.27 this weekend.