Hosted Services Issues

[giacomos]

Good day everyone,

We are currently experiencing an issue with multiple hosted services and configuration fetches from Dashboard.

The impacted services would include:

- Device configuration fetch (all devices)

- Client VPN with Meraki Authentication

- Systems Manager enrollment (if gated behind Meraki Authentication)

- SSID scheduling

- Any other service using Meraki Authentication

- Site to Site VPNs in a few shards

Our teams are aware and are working to resolve this as promptly as possible.

The waiting times calling in are currently extended.

We will provide further updates here as we receive them.

Many thanks.

Giac

Cisco Meraki Support

[giacomos]

Hi all,

We'll start marking updates as "Solutions" so we can have them at the top of the thread.

Please be on the lookout for them.

Many thanks.

Giac

Meraki Support

[GrahamG]

This is less than reassuring. The first image shows the current dashboard alert message, the second shows the alert message from a couple of hours ago. What happened to "Meraki devices and networks will continue functioning normally "?

Current alert messageAlert message from 8 am PST

[krolunger]

My email and my phone won't stop ringing with questions and anger. Please resolve this promptly.

[mrouton72]

All Meraki customers' Overview map should be showing applicable outage indications (Unavailable SSIDs, Failed Policies, etc.)... Our Overview map shows ALL GREEN which is NOT correct! Meraki needs to update the Overview map to reflect correct system statuses for these types of issues.

[JulienV1]

VPN looks back for my clients

[GrahamG]

Same here. VPN just connected

[giacomos]

Good day everyone,

Our development teams have successfully identified the root cause of the issue and are continuing to work on implementing a fix.

We understand your frustration and appreciate your continued patience and support during this time.

[gtatechinc]

I was able to connect finally and can see now that life may return back to normal. I hope this never happens again!

[RajuD]

We are able to connect as well.

We have so many questions:

1. Why was there no status page ?

2. Why was no one from Meraki responding on social media about the outage?

3. It took 1 hr 40 mins to get Meraki support on phone just to confirm the issue. It could have been proactively.

4. Why was the change pushed during business hours (if this was caused by change), ? I dont seem to recall any notice about scheduled upgrades.

[srttallhat]

Though I appreciate the PR level of vague updating, We will need an adequate evaluation on what broke, why it broke, and how it can be prevented in the future. It may be steep, but this seemed to have been a meraki wide outage, and due to the nature of more and more companies relying on remoting services, this 5 hour outage has costed millions in disturbances.

[JRMcGee1]

@giacomos

As long time Meraki customer, why did it take 5+ hours to fix this issue. What was the root cause? How is Meraki going to prevent this from happening again? Is there a plan in place to remedy issues like this? This is very concerning that this has happened.

[Maurice_Fellay]

Site to Site - Non Meraki VPNs are working again.

[Marcel Kamenz]

I'm really surprised, almost every cloud product from cisco now has a status page where even an incident can be tracked transparently for everyone.

Examples?:
https://status.webex.com/service/status?lang=en_US
https://status.umbrella.com/#/

Why don't you have something like that???

[giacomos]

@Marcel Kamenz we hear you and we are passing the feedback on internally.

Thank you for sharing it!

Giac

[giacomos]

Community,

We'd like to thank you all for being so understanding and supportive. We completely understand how difficult the situation has been for many of you. We want you to have a great experience when you use our platform and our products and we are committed to ensure that you do.

Please be on the lookout for comms via the cases and engage with our Support teams when necessary.

Keep the feedback coming, it never falls on deaf ears.

Thank you again for all your patience and engagement!

Giac