Buy or Renew
Buy or Renew
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
15028
Views
6
Helpful
15
Replies

Python Get API randomly hits limit

Go to solution
johanoosterwaal
Level 2
Level 2

‎06-23-2020 10:31 PM

Hi,

working on a script for Meraki to count all wireless clients in a organization and there capabilities. But the code run randomly in to the API limit which is 5 per second. As a test is added allot of sleep timer to slow down the code but even that doesn’t work and i cannot figure out why.

import requests
import time
import os.path


headers = {'X-Cisco-Meraki-API-Key': xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'}
se = requests.session()

orgid = {'Test': '111111'}

cnt_80211b = 0
cnt_80211n = 0
cnt_80211ac = 0
cnt_80211ax = 0


try:
    timer = 0
    client_info = []
    for org in orgid:
        networks = 'https://api.meraki.com/api/v0/organizations/{}/networks'.format(str(orgid[org]))
        time.sleep(1)
        networks_data = se.get(networks, headers=headers).json()
        time.sleep(1)
        response = requests.request("GET", networks, headers=headers)
        time.sleep(1)
        print ("networks_data")        
        if response.status_code == 200:
            time.sleep(1)
            if networks_data == 'errors':
                time.sleep(5)
            else:
                for each in networks_data:
                    print(".....................................................")
                    print (each['name'])
                    print(".....................................................")
                    time.sleep(1)
                    clients = 'https://api.meraki.com/api/v0/networks/{}/clients'.format(str(each['id']))
                    clients_data = se.get(clients, headers=headers).json()
                    time.sleep(1)
                    response = requests.request("GET", clients, headers=headers)
                    time.sleep(1)                    
                    if response.status_code == 200:
                        time.sleep(1)
                        for each_cli in clients_data:
                            if clients_data == 'errors':
                                time.sleep(5)                    
                            else:
                                if each_cli['ssid'] is not None:
                                        time.sleep(1)
                                        clientsurl = 'https://api.meraki.com/api/v0/networks/{}/clients/{}'.format(str(each['id']), str(each_cli['id']))
                                        clientsdata = se.get(clientsurl, headers=headers).json()
                                        time.sleep(1)
                                        response = requests.request("GET", clientsurl, headers=headers)
                                        time.sleep(1)
                                        if response.status_code == 200:
                                            if clientsdata == 'error':
                                                print ("Error")
                                                print (clientsdata)
                                                time.sleep(5)                    
                                            else:
                                                time.sleep(0.3)
                                                if "802.11b" in clientsdata['wirelessCapabilities']:
                                                    print ("802.11b")
                                                    cnt_80211b = cnt_80211b + 1
                                                elif "802.11n" in clientsdata['wirelessCapabilities']:
                                                    print ("802.11n")
                                                    cnt_80211n = cnt_80211n + 1
                                                elif "802.11ac" in clientsdata['wirelessCapabilities']:
                                                    print ("802.11ac")
                                                    cnt_80211ac = cnt_80211ac + 1
                                                elif "802.11ax" in clientsdata['wirelessCapabilities']:
                                                    print ("802.11ax")
                                                    cnt_80211ax = cnt_80211ax + 1 
                                        elif response.status_code == 429:
                                            time.sleep((int(response.headers["Retry-After"]))+ 1)                               
                    elif response.status_code == 429:
                        time.sleep((int(response.headers["Retry-After"]))+ 1)
                        
        elif response.status_code == 429:
            time.sleep((int(response.headers["Retry-After"]))+ 1)
    print ("...............................")
    print ("802.11b")
    print (cnt_80211b)
    print ("...............................")

    print ("...............................")
    print ("802.11n")
    print (cnt_80211n)
    print ("...............................")

    print ("...............................")
    print ("802.11ac")
    print (cnt_80211ac)
    print ("...............................")

    print ("...............................")
    print ("802.11ax")
    print (cnt_80211ax)
    print ("...............................")

except Exception as ex:
    print ("ooopsss")
    print(str(ex))
0 Helpful
15 Replies 15

Go to solution
johanoosterwaal
Level 2
Level 2
In response to rares-pauna

‎03-22-2023 03:30 AM

Hello RaresPauna,

As mentioned in the chat here is a Python class i created to help with the Meraki API ratelimit.

import sys, requests, os, json
import pandas as pd
from requests_futures.sessions import FuturesSession
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry

class MyHTTPSession:
    def __init__(self, ARG_APIKEY=None, URL_LIST=[]):
        assert ARG_APIKEY, f'{self.__class__.__name__} ARG_APIKEY cannot be empty'
        self.ARG_APIKEY = ARG_APIKEY
        self.BASE_URL   = "https://api.meraki.com/api/v1"
        # Create list of urls for further processing
        self.urls       = list(self.divide_chunks(URL_LIST))
        retry_strategy = Retry(
            total=4,
            backoff_factor=0.5,
            status_forcelist=[429, 500, 502, 503, 504],
            method_whitelist=["HEAD", "GET", "OPTIONS"]
        )
        adapter = HTTPAdapter(max_retries=retry_strategy)
        self.http = requests.Session()
        self.http.mount("https://", adapter)
        self.http.mount("http://", adapter)
        self.headers={
                        'X-Cisco-Meraki-API-Key': self.__dict__['ARG_APIKEY'],
                        'Content-Type': 'application/json',
                        'Accept-Encoding': 'gzip'
                    }

    def divide_chunks(self, l):
        n = 5
        # looping till length l
        for i  in range(0, len(l), n):
            yield l[i:i + self.n]

    def count_len(self, l):
        count = 0
        for i  in range(0, len(l), self.n):
            count+=1
        self.netlist_len = count
 
    def clear_console(self):
        platforms = {
            'linux1' : 'Linux',
            'linux2' : 'Linux',
            'win32' : 'Windows'
        }
        if sys.platform not in platforms:
            raise f" Platform:{sys.platform} is not in the list clear_console"
        else:
            if platforms[sys.platform] == 'linux':
                os.system('clear')
            elif platforms[sys.platform] == 'Windows':
                os.system('cls')

    def process_bar(self):
        try:
            precent = 100 * (self.cnt / self.netlist_len)
        except ZeroDivisionError:
            precent = 0 
        bar = '#' * int(precent) + '-' * (100 - int(precent))
        print(f"\r|{bar}| {precent:.2f}%", end="\r")

    def get_data(self):
        self.clear_console()
        df = pd.DataFrame()
        end_df = pd.DataFrame()

        self.__dict__.setdefault('json_dept', 1)
        self.__dict__.setdefault('data', '')
        
        self.cnt = 0
        self.process_bar()
        self.cnt += 1
        for urls in self.urls:
            futures = [(url[1],self.http.get(self.BASE_URL + url[0], data=json.dumps(self.data), headers=self.headers)) for url in urls]
            self.process_bar()
            self.cnt +=1
            for future in futures:
                try:
                    r = future[1]
                    if r.status_code == 200:
                        if len(r.json()) != 0:
                            df = pd.json_normalize(r.json(), max_level=self.json_dept, errors='ignore')
                            for key, value in future[0].items():
                                df[key] = value
                            df = df.fillna(value=np.nan)
                            df = df.astype(str)
                            while 'next' in r.links:
                                r = self.http.get(r.links['next']['url'], headers=self.headers)
                                if len(r.json()) != 0:
                                    dff = pd.json_normalize(r.json(), max_level=self.json_dept, errors='ignore')
                                    for key, value in future[0].items():
                                        dff[key] = value
                                    dff = dff.fillna(value=np.nan)
                                    dff = dff.astype(str)
                                    df = pd.concat([df, dff])
                    elif r.status_code == 404:
                        print("Page not found")
                except requests.exceptions.RequestException as err:
                    print(f"Error occurred during HTTP request: {err}")
                    continue
                end_df = pd.concat([end_df, df])
        self.response = end_df


This class takes a list of lists containing urls to check.
As an example, I'm using the url endpoint '/organizations/{Organization IDs}/networks' to collect all networks inside an organization.

lst = []
records = <Source from where to select>
for row in records:
    URL = f"/organizations/{<Organization IDs>}/networks"
    lst.append([URL, {}])

my_meraki_session = MyHTTPSession(ARG_APIKEY='123456789', URL_LIST=lst)
df_result = my_meraki_session.get_data()

Records is a list of the organization IDs that the API key can access.
There's an extra dictory [] after the url in the lst. This dictionary can be used to add more information to the Pandas dataframe later on. Things like organization names
Example:

records = Mysql_connection(dictionary=None).Select('ORGIDS', 'id', 'name' ,DISTINCT='DISTINCT')
for row in records:
    URL = f"/organizations/{row[0]}/devices"
    lst.append([URL, {'organizationId': row[0], 'organizationName':row[1]}])


MyHTTPSession is a custom HTTP session class that extends the requests.Session class. The class is initialized with two parameters: ARG_APIKEY and URL_LIST. The ARG_APIKEY is a mandatory parameter that represents the API key for the Meraki dashboard API. URL_LIST is an parameter that is a list of URLs that will be processed.

The class has several methods.

  1. Init(self, ARG_APIKEY=None, URL_LIST=[]): This is the class constructor that initializes the class properties ARG_APIKEY, BASE_URL, urls, http, and headers. It also initializes the Retry and HTTPAdapter objects that will be used for handling retries and HTTP requests. It divides the URL_LIST into chunks and sets the chunk size to 5. It also sets default values for json_dept, data, and netlist_len.

  2. Divide_chunks(self, l): This method is used to divide the URL_LIST into chunks of size 5. It takes in a list l as a parameter and uses a yield statement to return chunks of size 5.

  3. Count_len(self, l): This method is used to count the number of chunks in the URL_LIST. It takes in a list l as a parameter and divides the list into chunks of size 5. It then calculates the number of chunks and sets the result to the netlist_len property.

  4. Clear_console(self): This method is used to clear the console. It uses the os and sys modules to determine the platform and clear the console accordingly.

  5. Process_bar(self): This method is used to display a progress bar. It calculates the percentage of completion and displays the progress bar in the console.

  6. Get_data(self): This is the main method that retrieves data from the Meraki dashboard API. It initializes two data frames, end_df and end_df. It sets the JSON_dept and data properties if they are not already set. It sets the cnt property to 0 and displays the progress bar. It then iterates over the urls and creates a list of futures using the http.get method. It then iterates over the futures and processes the responses. If the status code is 200, it normalizes the JSON response using the PD.json_normalize method and adds any additional columns from the future. If there are more pages in the response, it continues to retrieve data until all pages are retrieved. It then concatenates the data frame df with end_df. If the status code is 404, it prints an error message. If there is an error during the HTTP request, it prints an error message and continues. Finally, it sets the response property to end_df.

Customers Also Viewed These Support Documents