Solved: Dashboard Issues?

Twitch0331 · ‎03-15-2021

Anyone else seeing weird issues with the Dashboard today? I was able to access it just fine earlier today, but then access to the it just went belly-up. Not long after that, I began receiving all of these notifications that devices are going down at all sites, but when I look at them in the Dashboard (which became reachable again) they appear to be just fine. The fact that my email and phone are not blowing-up with people in a panic seems to confirm this.

Are there any known-issues at the moment??

Thanks!

Twitch

RodrigoCW · ‎03-16-2021

Hey Everyone!

Checking in to provide a quick update. We are still working on putting together a full public RCA to share with you all. Due to the nature of the issue, this might take several more days.

That said, I wanted to share what we know for certain at this time.

There was an outage on March 15th that affected customers hosted in the Americas region. The outage started 10:55 PT and lasted 8 minutes. During this time, some customers were not able to log into or access Dashboard. All services were restored at 11:03 PT.

The outage was due to a core network device managed by one of our data center service providers that needed to be rebooted and reconfigured. We are working with the provider to ensure this type of issue does not happen in the future.

Dashboard (dashboard.meraki.com) was unavailable during this time, however, there was no disruption to network connectivity and services at customer locations. Meraki devices were online throughout, though they will report as being offline during that time.

I hope this proves helpful and as always, please don't hesitate to reach out if there is anything else I can clarify for you.

Happy Tuesday to you all! 😃

View solution in original post

Tim Bisel · ‎03-15-2021

I have gotten >70 alerts in 10 minutes for three organizations across three states and close to 40 networks. All other monitoring I have in place show nothing but alerts from Meraki are going nuts.

felix.moreno · ‎03-15-2021

Same issue here. Waiting to get TAC on the phone so they can confirm. Last time we experienced this, the root cause was maintenance being done on the Meraki Network my company resides in.

vassallon · ‎03-15-2021

@Twitch0331

Yeah it appears that the Dashboard was not accessible for 10ish minutes. In that time frame devices were not able to check in which is triggering the email messages about devices being offline when in fact they have been working fine the whole time.

Found this helpful? Give me some Kudos! (click on the little up-arrow below)

Dylan_YYC · ‎03-15-2021

Yep, same thing here!

rlwilson · ‎03-15-2021

I got kicked out of the dashboard and then couldnt get logged back in. Now I am getting a ton of email alerts about my VPN tunnels being down.

ncfb-awarsame · ‎03-15-2021

We had the same experience. All our tunnels went down, but why ? Meraki always say that the network would still function without the dashboard, you just can't manage it. Clearly that was not the case today.

DHAnderson · ‎03-15-2021

When you say the tunnels went down, did they actually go down, and no data could be transferred, or that the dashboard reported them down?

Dave Anderson

RodrigoCW · ‎03-15-2021

Hey @ncfb-awarsame,

Dashboard's only role in the formation of an Auto-VPN tunnel is sharing Port and IP info with Peer MXs. A Dashboard issue like this would make previously established tunnels show offline on the portal, but remain up in actuality. Were you unable to access resources on the other side of your tunnels?

ShaneBright0602 · ‎03-15-2021

@RodrigoCW What would be the recommendation on determining the real tunnel status? Or do we have to test each tunnel comm individually on our own, which is impractical. Are we just blind until this is resolved?

RodrigoCW · ‎03-15-2021

Hey @ShaneBright0602,

The best way to confirm if your tunnels are working would be to send some traffic across some of them. For some people, testing all tunnels might not be viable.

My advice for anyone in this situation would be to test your most critical tunnels, confirm that traffic is flowing, and then send an FYI to people using those tunnels to let you know if they encounter any issues.

nb12345 · ‎03-16-2021

Hello,

Did you confirm the RCA? what was the reason for outage please?

Thanks,

El-Guero-HTX · ‎03-15-2021

Not the best advice during this potential outage, but I learned my lesson some years back. We stood up PRTG to monitor our network in additional to the Merkai alerts.

I still almost wet my fresh pants when seeing all of the Meraki alerts.

Twitch0331 · ‎03-15-2021

@El-Guero-HTXI know that feeling! We all need some Meraki-branded Depends if this is going to keep happening.

luke11 · ‎03-16-2021

"Meraki-branded Depends"

...You know, they are big on swag... 😂