Error: 429 Client Error: Too Many Requests

[Adrian41]

Hello,

Recently I have been running into this unexpectantly. If I wait 5 or 10 mins it goes away but it kept happening so I ran the API requests summary endpoint each time it happened and I saw thousands of requests with a 429 response.

Since 429 is the too many requests error, I assumed a first call 429 and then it re-try's a thousands times immediately after until it gives up which is what gives me all those 429?


I tried to dig deeper into where they were coming from....

all the 429 requests were apparently getNetworkWirelessClientCountHistory ?!? - that doesn't make any sense as i only ran that once and it was successful.

also - in the thousands, they seem to alternate from ssid 4 to ssid14 over and over - not sure what that is about as I am not jumping around SSIDs and the one im connected to isnt 4 or 14.


can anyone shed some light?

[sungod]

The Meraki Python library call does the wait on rate limit itself.

You specify maximum retries 10, so as long as the library call needs fewer retries you would not get a 429 exception.

I.e. ideally you should not need to do the extra wait/retry in your exception handler.

At a high enough rate, I think the 429 mechanism breaks down, so if I know I'll be exceeding the rate limit I tend to add code to limit concurrent calls and/or add some additional back-off/retry that increases retry time exponentially.

[obrigg]

Your script may be using 10 calls per second, but there might be other integrations/script consuming more calls at the same time - resulting in the organization exceeding the rate limit.
You can use getOrganizationApiRequests to examine who/what else is making API calls to the organization.

[rares-pauna]

Thanks for the input, that's how i'm already checking the numer of calls by status code. All are provided by my script thst generates an output of aprox. 500 calls(200+429) for an organization with 120 networks, so the number of calls should be 120. My guess it's that meraki's library parameter wait_on_rate_limit and maximum_retries are causing many duplicates that are unnecessary. May the problem be that i initialize the meraki.aio for every call and it is unaware that it has other instances that are already doing the same thing

[sungod]

Initialising aio per-call sounds likely to cause problems.

This sort of thing is how I generally do concurrent calls with aio, here on a list of networks, each instance of the called function then each does a request on one network.

# process eligible networks concurrently
networkTasks = [processNetworkAppliances(aiomeraki, net) for net in applist]
for task in asyncio.as_completed(networkTasks):
    await task

This and all other aio stuff happens inside a single async block...

async def main():

    # any set-up stuff here

    async with meraki.aio.AsyncDashboardAPI(
        api_key=API_KEY,
        base_url='https://api.meraki.com/api/v1/',
        print_console=False,
        output_log=False,
        suppress_logging=True,
        wait_on_rate_limit=True,
        maximum_retries=100

    ) as aiomeraki:

        # everything using aio goes in here

    # aio stuff is finished, do whatever happens next, if anythng

[rares-pauna]

I don't see any kind of semaphore in your script. Using asyncio.as_completed should start all of them at once, so if your organization has 30 networks you start 30 workers and hit the rate limit. Am I wrong with this logic?

[sungod]

Never bothered to use semaphores. The wait/retry is generally enough.

But as I said in another post, if I know there'll be large numbers of concurrent calls, I just break the list into smaller chunks to avoid excessive load on the service.

It's a simple method and works at scale.

[Philip D'Ath]

That is not how it works. The Meraki asyncio internally uses a rate limiter of 10. It will submit no more than 10 API calls at a time.

[RC0008]

I really wouldn't suggest changing the code to anyone. Requests Retry-After will give you the identical results. I've built all my python codes using Meraki library and when I tried converting it to Requests with the IF statement I still get the same error 429 with a Retry-After in 59 or 60 seconds. Nothing greater or less and I had a max retry of 10. I thought the Retry-After would give off random seconds, but it was consistently 59 or 60.

Because of this result, I'm adding time.sleep() with a random integer in between API calls and use this setting for my dashboard.

dashboard = meraki.DashboardAPI(API_KEY, suppress_logging=True,wait_on_rate_limit=True,maximum_retries=10)

Supposedly Meraki is looking into increasing the API calls to 50, but this was in early Alpha stages when I asked my rep.

[obrigg]

Question to the group. When you hit 429, how do you investigate/troubleshoot/workaround it?

[Philip D'Ath]

I'm going to start a new reply. The asyncio uses a ratelimiter by default. They way this works is it submits all 10 API calls immediately, and then as one request finishes it dispatches another.

When I have high levels of concurrency I have had more luck using a Throttler. If you have 10 API requests pending, it spaces them out in 100ms intervals. I use code like:

import asyncio,meraki.aio,throttler
...
async with meraki.aio.AsyncDashboardAPI(
	output_log=False,
	print_console=False,
	maximum_retries=100,
	wait_on_rate_limit=True
) as dashboard:
dashboard._session._concurrent_requests_semaphore = throttler.Throttler(rate_limit=4, period=1.0)
...

[rares-pauna]

I had absolutely no idea about ._session._concurrent_requests_semaphore. So this by default submits 10 calls?

Where you found it?

Thanks

[Philip D'Ath]

How did I find it?

Something I would have been doing would have been broken. I go look at the source code for the Meraki SDK to understand how it works. I then consult Google. I then try 90 different approaches and fail. I then get it working and spend the next 10 iterations tidying it up and working nicely.

That's how I found it.

[Philip D'Ath]

Search for semaphore.

https://github.com/meraki/dashboard-api-python/blob/main/meraki/aio/rest_session.py