Hosted Services Issues

[giacomos]

Good day everyone,

We are currently experiencing an issue with multiple hosted services and configuration fetches from Dashboard.

The impacted services would include:

- Device configuration fetch (all devices)

- Client VPN with Meraki Authentication

- Systems Manager enrollment (if gated behind Meraki Authentication)

- SSID scheduling

- Any other service using Meraki Authentication

- Site to Site VPNs in a few shards

Our teams are aware and are working to resolve this as promptly as possible.

The waiting times calling in are currently extended.

We will provide further updates here as we receive them.

Many thanks.

Giac

Cisco Meraki Support

[bschlegelno]

One other suggestion.

This isn't the first time there has been a backend cloud issue and when you call in and wait on hold forever it doesn't help anyone's patience. It might go a long way to announce an issue like this before putting the call into the support que. Being able to call Meraki support easily is one of the best features, but being on hold for 45 minutes waiting to talk to someone while back end issues are being worked on by engineers is extremely frustrating.

[SteveB2021]

I second that.

Had the same experience too.

The hold music should be changed to an on hold message stating theirs and outage.

[giacomos]

@SteveB2021 @bschlegelno thank you for sharing!

You are spot on there and I'm happy to share that this is something that we have included in our process. Unfortunately due to some technical difficulties, the announcement was injected after a few hours from the start of the outage and was not immediately played for those who were already in the queue waiting to speak with us.

Many thanks!

Giac

[r393292492]

Thank you Meraki, my life is now ruined. My wife is walking out the door with the kids, my dog ran away, and my family has disowned me. All I can hear is the constant ringing of my phone and banging on the door of the server closet. I'm scared for my life. Will anyone save me? What will you do to fix this? Will it ever end?

[RazzleDazzle]

Don't forget the angry mob with torches and pitchforks outside your door that is setting fire your home and has brought a vat of boiling Meraki™ oil, just for you.

[Maurice_Fellay]

Life really sucks when your dog ran away, all other issues you mention are just the most common situation for an IT guy.

But hey, you can always adopt a new dog.

[Maurice_Fellay]

Client VPNs are starting to authenticate, Still changes to dashboard are not being sent to appliances. but at least we are starting to get into normal operation.

Will Meraki send information on what was the issue, why did it happen and what will be the remediation in place so it will not happen again?

[giacomos]

@Maurice_Fellay @srttallhat , please make sure you raise a Dashboard case to request a reason for outage.

Many thanks.

Giac
Meraki Support

[srttallhat]

I have already done so. But transparency in communication to all would help. If I receive an official and adequate response, Ill make sure to post it here.

[giacomos]

Hey everyone,

We’re seeing service recovery for the issue impacting certain cloud-based services and working towards full restoration as soon as possible.

We thank our customers and partners for their patience.

[mrouton72]

Please post cause of outage and resolution details.

[micronet_info]

@mrouton72 my issue is not even fixed and you asking for cause and resolution details? lol

Config still not pushing to devices and showing out of date.

[srttallhat]

@micronet_info One of many issues stemming from the bigger outage. all in all, an overall official documentation once all is restored is in order, agree?

[Stevesdown]

it's not working for Windows 7. I might add we have a project removing the last ones from our environment. We're using a connection with the extension *.PBK. This method is still not working.

[SanketKosrabe]

Hello Everyone,

Just to keep you all informed on the config fetch outage/Client VPN, a fix has been deployed to all shards, nodes will now be able to fetch their configurations if a Dashboard change is initiated or so out of date nodes should self-correct as they fetch configs according to their backoff timers.

We thank you for your patience and support on this.