Error: 429 Client Error: Too Many Requests

[Adrian41]

Hello,

Recently I have been running into this unexpectantly. If I wait 5 or 10 mins it goes away but it kept happening so I ran the API requests summary endpoint each time it happened and I saw thousands of requests with a 429 response.

Since 429 is the too many requests error, I assumed a first call 429 and then it re-try's a thousands times immediately after until it gives up which is what gives me all those 429?


I tried to dig deeper into where they were coming from....

all the 429 requests were apparently getNetworkWirelessClientCountHistory ?!? - that doesn't make any sense as i only ran that once and it was successful.

also - in the thousands, they seem to alternate from ssid 4 to ssid14 over and over - not sure what that is about as I am not jumping around SSIDs and the one im connected to isnt 4 or 14.


can anyone shed some light?

[sungod]

Assume you are using the Meraki Python library.

Are you telling the library to wait on retry? This is important to avoid retries being made too fast.

Combined with setting a generous retry limit should ensure calls eventually complete ok.

        wait_on_rate_limit=True,
        maximum_retries=100

For instance...

    async with meraki.aio.AsyncDashboardAPI(
        api_key=API_KEY,
        base_url='https://api.meraki.com/api/v1/',
        print_console=False,
        output_log=False,
        suppress_logging=True,
        wait_on_rate_limit=True,
        maximum_retries=100
    ) as aiomeraki:

If you are using some different code to handle 429s, make sure it is obeying the Retry-After header in the 429 response, see https://developer.cisco.com/meraki/api-v1/rate-limit/

[Adrian41]

thank you! to be honest I had never come across 429 errors until a day or two ago 😛

[Philip D'Ath]

I haven't checked - but are you saying wait_on_rate_limit is not True by default? I'm gobsmacked.

[sungod]

Tbh I think it defaults to on, but decades of coding paranoia leads me to always set it explicitly in my scripts 😀

[Philip D'Ath]

I checked, and wait_on_rate_limit is on by default.

https://github.com/meraki/dashboard-api-python/blob/main/meraki/config.py#L19

and maximum_retries defaults to 2.

https://github.com/meraki/dashboard-api-python/blob/main/meraki/config.py#L37

And for anyone finding this using Google, here are all the Meraki Python library defaults:

https://github.com/meraki/dashboard-api-python/blob/main/meraki/config.py

[Adrian41]

the problem has returned 😞

every so often getting 429 errors and when I check the summary I see this "429": 1307,

So two things - one, when it gets a 429, it is clearly re-trying over thousand times in tiny time frame -
not sure I am using these variable properly

wait_on_rate_limit=True,
maximum_retries=100

Am I supposed to pass them as part of the API headers or something?

Secondly, I must still be doing something to trigger the 429 rate limit in the first place. The limit is 5 requests per second I think? After each and every request call I have put in a 0.3 second delay which should mean I cant be sending more than 4 a second so how is this happening?

To produce the delay I am using the time module and setting a global variable of

DELAY = 0.3

then after every request putting the line

time.sleep(DELAY)

networks = requests.get(networks_url, headers=headers, verify=f"{root_path}/Cisco Umbrella Root CA.crt")
time.sleep(DELAY)

[sungod]

Are you using the Meraki Python library?

[Adrian41]

Im using "requests" HTTP library

[sungod]

Then you are not using the Meraki Python library.

These set behaviour of the library, if you are not using it then they will not help you.

wait_on_rate_limit=True,
maximum_retries=100

I recommend using the library.

If don't want to use it, you will need to do as suggested above and make sure you code to obey the Retry-After header in the 429 response, see https://developer.cisco.com/meraki/api-v1/rate-limit/

For instance based on the example on the linked page...

response = requests.request("GET", url, headers=headers)
​
if response.status_code == 200:
    # Success logic
elif response.status_code == 429:
    time.sleep(int(response.headers["Retry-After"]))
else:
    # Handle other response codes

[Adrian41]

Hi,

Thanks for the reply! makes sense.

However, the 429 error means iv already hit the rate limit, so adding a sleep timer on that code will help reduce the amount of time i have to wait before i can send again (by preventing the re-try spam) but I still have the issue of hitting the limit in the first place 😞

[sungod]

The rate limit is just a fact of life, but if you use the back-off and retry mechanism your calls will eventually succeed.

For example, I have scripts using the Meraki Python library aio functions that could potentially make thousands*** of calls 'at once', but the rate limit and retry handling kicks in, things back off and retry, resulting in throughput at the rate limit, the library handles it all for me.

The alternative is to set timers and not issue calls any faster than the limit, but that is unreliable as someone else may make calls on the org at the same time as you and then rate limiting can still occur.

For some purposes you can use action batches, giving higher throughput...

https://developer.cisco.com/meraki/api-v1/action-batches-overview/

***if I am doing something that will generate high call volumes, I generally split it into smaller chunks to avoid just hammering on the org.

[rares-pauna]

I am using the meraki library, with an semaphore of 10 conccurent workers on top. Even so, when i check the number of calls, the 429 are hundreds. I don't know how to stop it. Once it start getting them, it seems it's never stopping

[RC0008]

Put your code in chatgpt or any other AI and ask it to help continue the script with 429 errors. It can adjust the script to add some back off timers in between calls. Keep trying until you have a script running without any interruptions. It will error out and it will give you 429, but the script will continue to run until completed.

I’ve done this with the meraki and other APIs

[rares-pauna]

async def getNetworkAppliancevlansMeraki(self,id😞

async with self.semaphore:

try:

async with meraki.aio.AsyncDashboardAPI(self.api_key,self.base_url,output_log=False,print_console=True,suppress_logging=True,wait_on_rate_limit=True,maximum_retries=10) as aiomeraki:

vlans = await aiomeraki.appliance.getNetworkApplianceVlans(id)

return vlans

except meraki.AsyncAPIError as e:

if e.status == 429:

retry_after = int(e.response_headers.get('Retry-After', 1))

log.info(f"Rate limit exceeded. Retrying after {retry_after} seconds.")

await asyncio.sleep(retry_after)

else:

vlans= []

log.info(f"Couldn't get Vlans , error: {e}")

return vlans
Even if i see hundreds of 429 calls, nothing raises my exception