Meraki API Performance degraded?

[mrowan]

Hi,
Is there an ongoing issue with the performance of the Meraki Dashboard API?

We've developed software for a number of Meraki SD-WAN customers and in recent days have been experiencing issues with a significant increase in the API reliability and response times.

Whilst requesting data from various endpoints we're seeing issues, which can also be reproduced in Postman

• Some requests are made and never responded to
• Some requests respond with a 200 header, but then don't appear to respond with any payload data.
• Requests that do succeed are taking a lot longer than expected, sadly I don't have precise figures to be able to quantify this, as an example query for the 1st page (300) VPN statuses {{baseUrl}}/organizations/:organizationId/appliance/vpn/statuses takes between 4 and 25 seconds (at 7-7:30pm UK time today). Earlier today I experienced times of > 1min.

This problem was seen with multiple unrelated customer organisations. For example one his hosted on n376.meraki.com and another on n271.meraki.com

Looking at some historical data this issue appears to have started around 11th Jan and has been getting progressively worse. It also appears to be worse from 8am until 8-9pm.

I'm not aware of anywhere to see service status or performance if there is such a thing. But it looks very much like a load issue

[PKoelemij]

We've seen a dramatic increase in API response times too, I've logged a case about it on the 19th, but I couldn't seem more information there.

The job that gets all device statuses for a 150-ish organizations went from ~42 sec to ~6 minutes. Calls on specific API shards became significantly slower, the graph would be way higher if the timeout for the API client wasn't 60 seconds, because some calls just took 2-4 minutes to respond.

99th percentile going up:

Averages going up:

[sungod]

For us the issue was resolved on 21st around midday UTC.

If you still see a problem then I suggest post a comment on the case you opened asking for an update.

[PKoelemij]

Yeah our issues have been resolved.

[mrowan]

Can you confirm which API Endpoints you're using?
We have an issue where some times API requests don't respond before an internal timeout (60 seconds), I sometimes see the same issue using just PostMan, as a result, we changed our implementation to have an additional retry policy, such that if we don't start to get a reply within a second, we drop that request and make it again (after a short delay). This has improved the overall reliability significantly. Not sure if you're hitting the same issue.

[PKoelemij]

@mrowan I was having slow responses on:

GET /v1/networks/{networkId}/firmwareUpgrades

GET /v1/organizations/{organizationId}/devices

GET /v1/organizations/{organizationId}/devices/statuses

I'm temporarily applying the same strategy as well: implemented a lower timeout and retries but that's just putting on a band aid unfortunately.

[AutomationDude]

I'm running a script that loops through many orgs and it's quite clear that certain orgs on specific shards are experiencing a ton of latency. Most of the orgs however are doing just fine. It also seems to be specific API calls on specific orgs that are timing out.

[Pettho]

I've also experienced time out on several of my scripts runneing API calls. I have narrowed it down to n146.meraki.com timing out. Contaced Meraki Support, but they have no solutions.

[AutomationDude]

Yep, it's the same shard for me thats causing issues. Doubt Meraki support can do anything about this, the problem is likely to be at a much deeper level.

[sungod]

You need to persuade them to open an internal case with the Dashboard-API support team.

[Pettho]

It's already done! But they can't say anything......