DNA Center API 500 record limit

[bunjiega]

I am fairly new at API's (coming from route/switch background), but I have DNA Center API calls working, but I only get back a max of 500 records, which is expected from what I see in the documentation. How would we get more records than 500 since I have far more than 500 in my DNAC inventory.

I tried this --> https://community.cisco.com/t5/automation-and-analytics/method-to-return-more-than-500-devices-in-api-get-quot-network/td-p/3437814  - which does work, but only when NOT using a filter at the end of the URL.

An example is trying to get all of the access-points from DNAC with this URL: 

https://dnac.xxx.org/api/v1/network-device?series=.*Unified.*

This works for only the first 500 - how would I get the next 500?

Thank you!!!

[bunjiega · Accepted solution]

Did finally find using the filter 'offset=500' gets the next batch!

[OaklandCounty_Mike]

Hello!

i am having a issue here too.  if i place offset=500 as a filter at the end of my query and i actually get less than 500.  i have over 700 devices....how would i modify the rest api call to get all my inventory items?  

Thanks in Advance,

def network_device_list(dnac

        token = dnac_login(host, username, password)

        url = f"https://{host}/api/v1/network-device?offset=500"

        headers = {'x-auth-token': token}

        response = requests.get(url, headers=headers, verify=False)

        data = response.json()

        return data

Mike

[bunjiega]

My understanding is that you would need to make 2 calls. 1 without the offset filter and the next one with it. The first call would get the first 500 devices, the second call would start at device 501 (since you are using the offset) and it would get the rest.

Here is what I made and it works for me...

import requests 
from requests.auth import HTTPBasicAuth 
from requests.packages.urllib3.exceptions import InsecureRequestWarning #suppresses insecure warning message 
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)   #suppresses insecure warning message 
url = "https://XXXXXXXXXXXXXXXXX/api/system/v1/auth/token" 
headers = {"content-type": "application/json"}
response = requests.post(url, auth=HTTPBasicAuth(username="XXXXXXXXXXXXX", password="XXXXXXXXXXXXX"), headers=headers, verify=False) 
token = response.json()["Token"] 
#print (token) 
headers['x-auth-token'] = token 


def call_with_pagination(): 
    url = "https://XXXXXXXXXXXXXXXX/api/v1/network-device?" 
    my_json_list = [] # this is used to store all the different calls into one list (this is a list of dictionaries) 
    paginate_number = 0 
    network_devices = requests.get(url, headers=headers, verify=False) # make 1st call
    print (f"""Call 1 has { len(network_devices.json()['response']) } records""")
    for ech in network_devices.json()['response']:  # Add received json to my_json_list 
        my_json_list.append(ech)
    #    
    while len(network_devices.json()['response']) == 500:  
        url = "https://XXXXXXXXXXXXXXXXX/api/v1/network-device"   # need to reset the url each time   
        paginate_number += len(network_devices.json()['response']) # Adds to the pagination number so we can get the rest of the devices
        url = url + f"/{paginate_number}/500"  # Adding the pagination filter to the url
        #     
        network_devices = requests.get(url, headers=headers, verify=False) # make another call 
        print (f"""This Call has { len(network_devices.json()['response']) } records""") 
        for ech in network_devices.json()['response']:  # Add received json to my_json_list 
            my_json_list.append(ech) 
    return my_json_list        
   
my_list = call_with_pagination() 
print ("Total number of recieved records: " + str(len(my_list)))
#print(my_list) # uncomment this line to actually print ALL the devices

[OaklandCounty_Mike]

thanks for the reply Jeremy.  so i also had a heck of a time formatting the output the way i wanted it because i realized the output was a listed of dictionaries.  does your logic create more of this since you are storing the results into another list?

-Mike

[giomanjee]

If the ORG grows  to more than 1000++ device. this will not work. 

Edited the code to accommodate pretty much anything more than 500 devices. 

def endpoint_caller_intent_api(my_endpoint, method='get', data=None

    paginate_number = 0

    my_json_list = []

    url = f"{<you_DNAC_site>}/dna/intent/api/v1/{my_endpoint}"

    headers = {

      "Content-Type": "application/json",

      "Accept": "application/json",

      "X-auth-Token": get_token() ### get token from another function.. or siply plug value here. 

    }

    response = requests.request(method, url, headers=headers, data=data, verify=False)

    if response.ok:

        print(response.cookies, response.elapsed)

        print (f"Call 1 has { len(response.json()['response']) } records")

        for ech in response.json()['response']:  # Add received json to my_json_list

            my_json_list.append(ech)

        get_all_device_flag = True  ### created FLAG that breaks while TRUE loop if the return len of subsequent call is less than 500

        while get_all_device_flag:

            paginate_number += 500  #  set initially value with max increment limit of api. 

            new_url = f"{url}/{paginate_number}/{int(500)}" # create new URL

            network_devices = requests.get(new_url, headers=headers, verify=False) # make the call

            if len(network_devices.json()['response']) < 500: ## if less than value is less than 500, kill the while loop, this time you ALL

                print(f"{new_url}  --- LEN ",len(network_devices.json()['response']))      ## just printing output          

                get_all_device_flag = False # kill switch for the infinite loop

                for ech in network_devices.json()['response']:  # Add received json to my_json_list

                    my_json_list.append(ech)

            elif len(network_devices.json()['response']) == 500: #### continue to gather device.. 

                print(f"{new_url}  --- LEN ",len(network_devices.json()['response']))

                for ech in network_devices.json()['response']:  # Add received json to my_json_list

                    my_json_list.append(ech)

        print('jsonlist', len(my_json_list))

        return my_json_list

    else:

        print('Error', response.status_code)

get_devices = endpoint_caller_intent_api('network-device') ### call endpoint

[o-e-niemi]

There still is an issue with this method. When you do API calls with the offset, the order of the devices changes. Thus I always get a list of 2151 devices, but there are around 100-200 duplicates each time because the records are in different order each time. This is even when doing these calls back-to-back, with exact same start and end times defined as well. Anyone have an idea how to fix this issue?

[bigevilbeard]

@o-e-niemi  isnt an uncommon problem when dealing with APIs that don't guarantee a stable ordering of results. My suggestion here to fix this issue, you can use a set to keep track of unique devices and avoid duplicates. I think you can do this by using a set to store unique devices, you then ensure that each device is only added once, even if the API returns them in a different order.

Hope this helps.