Hosted Services Issues

[giacomos]

Good day everyone,

We are currently experiencing an issue with multiple hosted services and configuration fetches from Dashboard.

The impacted services would include:

- Device configuration fetch (all devices)

- Client VPN with Meraki Authentication

- Systems Manager enrollment (if gated behind Meraki Authentication)

- SSID scheduling

- Any other service using Meraki Authentication

- Site to Site VPNs in a few shards

Our teams are aware and are working to resolve this as promptly as possible.

The waiting times calling in are currently extended.

We will provide further updates here as we receive them.

Many thanks.

Giac

Cisco Meraki Support

[DavidHenley]

First, thank you to all for getting this resolved.

But most of us have to answer to someone on what happened today. When will you release a statement on the issues and fixes happening at Meraki to ensure this doesn't happen again?

[alejandro.macaya]

I continue with the problem, we have not been able to connect through VPN Client since yesterday, I have case 06834937 open without response, if you can review it again, a capture is attached.

[SanketKosrabe]

@alejandro.macaya

I see that one of the VPN clients was connected on Aug 5 at 13:22:45. If you are still having an issue, I would recommend troubleshooting the issue with support. I see that case is supported pending, NSE will reach back to you soon.

Meanwhile, you can provide a packet capture in Wireshark taken on the MX when the device is trying to connect along with the username from which the connection was being attempted. Also, provide the error core you are seeing on the end device.

I would also recommend testing a VPN connection from another machine such as Mobile/MacOS.

Make sure, the configurations on the machine is not changed and it is the same as per the document: https://documentation.meraki.com/MX/Client_VPN/Client_VPN_OS_Configuration

- Sanket

[wrespawn]

Wifi back up, thanks @giacomos for the constant updates. Must be a hellova day for you too.

[HourITAdmin]

Giacomo,

I appreciate the frequent updates in the thread. While I have opened a case to request a reason for the outage, I feel that an issue of this magnitude deserves a more official and transparent response. Root cause and immediate fix information both matter but I believe that future prevention/mitigation is what most of us admins will want to see. This is what will let us assure our teams, supervisors, and c-levels that this tech stack is still the way to go in our orgs.

-J-

[SanketKosrabe]

@HourITAdmin @DavidHenley @JRMcGee1

I totally understand the Major issue we had before. We are still working on RCA for the issue and waiting for update from the Engineering team.

Once that verbiage is created, we’ll update all of the cases attached to the issue so our customers get an update regarding RCA. Please look for an email on the cases you have created.

Once again I sincerely thank you for patiently waiting for the issue to be fixed.

Regards,

Sanket Kosrabe

Cisco Meraki Support

[BigMtnIT]

Client VPN with Meraki authentication is working again for most of my clients! Thank you!

[kkwok]

I was talking to support engineer and he has no clue what going on and he wants to start troubleshooting with me. I told him the problem is so i just hang up.

Unbelievable!!

[srttallhat]

@kkwok In their defense, When something this serious hits so quickly, it requires swift resolution. So understandably the messaage may not have gotten to all the first level techs. But depending on when you spoke with them, if it was hours after, that is rather unacceptable.

[jbandlow]

Configs seem to be working for me again. The symptoms I saw were port config changes (i.e. VLAN changes) showing the correct new settings but port status showing the old ones, and the ports actually behaving somewhere in between. This caused blocked traffic on port changes that were in progress when this issue hit. I started to notice some weirdness yesterday afternoon.

[ervincen]

While our team continues to work on finalizing the RCA, please feel free to share this summarized report of today's issue:

Today, at 12:00 PM UTC, Meraki dashboard encountered a back end issue that impacted certain cloud-based services. Our Engineering teams have isolated the problem and a fix has been deployed. Teams are actively monitoring the situation. A detailed RCA will follow soon and all impacted customers will be notified once complete. We thank you for your patience.

[csalese]

Captive portal authentication is still failing. Any updates on the other issues listed in the original ticket?

[Ryan_Miles]

captive portal should be working. what is your config for the splash page? click through, sponsored, sign-on with (insert your auth provider here), SMS, ISE, SM?

have you opened a Support case?

[csalese]

We haven't opened a support case yet. Our configuration is click-through.

[giacomos · Accepted solution]

Dear Cisco Meraki Customers and Partners,

On Thursday, August 5, 2021, at 12:00 pm UTC, Cisco Meraki encountered a problem with a certificate expiration that impacted certain cloud-based services, including device configuration, SSID availability, Meraki Authentication, Systems Manager, MV cloud archive, and MT sensor data.

Our engineering teams isolated the problem and renewed the certificate. Services were fully restored at 4:40 pm UTC. Our team is actively working to add additional safeguards to avoid these occurrences in the future

We apologize for any inconvenience this may have caused and thank you for your patience. Should you have any questions or concerns about this update, please contact Meraki support.

Sincerely,
The Cisco Meraki Team