404 error in Python script to DELETE objects from FMC using a CSV file

[Maneck_UK]

I have a fully working python script that can POST objects to the FMC (6.5.0.4) API using a .csv file for the data.

I want to use this same approach/method to DELETE objects, but after many days I have been unable to get the script to work.

Initially I was adding the object to the script as data rather than in the URL, I fixed this using a single object, if I add the object into the script using:

del_data = ['005056A6-6972-0ed3-0000-090194313992']

the URL works and the objects deletes.

But this defeats the purpose of the script as I can delete a single object in simpler ways.

I believe I now have a script that adds the host object ID to the url but it continues to fail with a url 404 error:

Connection Error --> 404 Client Error: Not Found for url: https://<FMC-NAME>/api/fmc_config/v1/domain/e276abec-e0f2-11e3-8169-6d9ed49b625f/object/hosts/005056A6-6972-0ed3-0000-090194314504

I'm new to this Python scripting business and I would be grateful if anyone has any pointers.

The relevant parts to my script are below:

Thanks in advance

URL build script component

f = open("delid.csv")
objectsfile = csv.DictReader(f)

for object in objectsfile:
    del_data = {
        object["objectid"]
    }

for object in del_data:
        del_url = server + api_path + object

try:
    r = requests.delete(del_url, headers=headers, verify=False)
    status_code = r.status_code
    resp = r.text
    log = open('delete_objects.log', 'a')
    print(" Status Code: "+str(status_code))
    json_resp = json.loads(resp)
    log.write(json.dumps(json_resp,sort_keys=True,indent=4, separators=(',', ': ')))

    if status_code == 201 or status_code == 202:
        print(" SUCCESS ")
    elif status_code == 400:
        print((" Message: ")+ resp + ('\n'))
    else:
        r.raise_for_status()
        print((" Message: ")+ resp + ('\n'))

except requests.exceptions.HTTPError as err:
    print("Connection Error --> "+str(err))
finally:
    if r: r.close()

[Maneck_UK]

URL build script component

f = open("delid.csv")
objectsfile = csv.DictReader(f)

for object in objectsfile:
    del_data = {
        object["objectid"]
    }

for object in del_data:
        del_url = server + api_path + object

try:
    r = requests.delete(del_url, headers=headers, verify=False)
    status_code = r.status_code
    resp = r.text
    log = open('delete_objects.log', 'a')
    print(" Status Code: "+str(status_code))
    json_resp = json.loads(resp)
    log.write(json.dumps(json_resp,sort_keys=True,indent=4, separators=(',', ': ')))

    if status_code == 201 or status_code == 202:
        print(" SUCCESS ")
    elif status_code == 400:
        print((" Message: ")+ resp + ('\n'))
    else:
        r.raise_for_status()
        print((" Message: ")+ resp + ('\n'))

except requests.exceptions.HTTPError as err:
    print("Connection Error --> "+str(err))
finally:
    if r: r.close()

[JonasKs]

In your code you do this:

for object in del_data:
    del_url = server + api_path + object

try:
    ... 

When you need to indent your `try` to actually call the `del_url` every time:

for object in del_data:
    del_url = server + api_path + object
    try:
        ... put code here. 

Also never use `object` as a variable name, it shadows the Python builtin class `object`. Name it `element` or something instead.

For your question, I'd recommend printing out your URL before you call `r.delete(..)` in order to see if it's formatted correctly. The 404 indicates that what ever resource you're trying to reach, it does not exist. You have most likely already deleted it in a previous run, and since your current version don't indent correctly, you've only called the API towards the last element in your CSV.

[Maneck_UK]

@JonasKs Thanks for the help.

At the first time of reading the information, I needed an emoji for "my head just exploded"

I've read through it a few more times and I think I understand; I will go and try it.....

[JonasKs]

Hehe, I understand. 

It really boils down to these steps:

1. Print the URL that you post to. This will make it clear that this is happening to your code. I'll try to write it out for you, though:

for object in objectsfile:
    del_data = {
        object["objectid"]
    }

In this snippet, you create a set of elements. del_data will now look like this:

del_data = {'first element', 'second element', 'third element'}

You then loop over that set:

for object in del_data:
    del_url = server + api_path + object

And then you exit the loop. So what's happening is this:

# first loop
del_url = 'first_element'
# second loop
del_url = 'second_element'
# third loop
del_url = 'third_element'

Now, the next thing you do is to execute your code:

try:
    r = requests.delete(del_url, headers=headers, verify=False)

But as you can see, you're not doing this inside your for-loop. So this only happens once. If we take a closer look at this request, the del_url is third_element, and I assume that you really wanted this to happen for every element in your set:

try:
    r = requests.delete('third_element', headers=headers, verify=False)

So, what I wrote above is that you should make this:

for object in del_data:
   del_url = server + api_path + object

try:
    r = requests.delete(del_url, headers=headers, verify=False)

look like this:

for object in del_data:
    del_url = server + api_path + object

    try:
        print(del_url)
        r = requests.delete(del_url, headers=headers, verify=False)

That will ensure that for every element in your del_data set, you will execute that command. 

2. Don't call your variables object , it's an illegal name to use in Python, due to the fact that it's a built in class. 

[Maneck_UK]

@JonasKs Thanks, I wrote my update post without seeing this update (my post took ages to write). This is great help. I will take another look.

[Maneck_UK]

I've looked over the extra help and I have cracked the deletion part. I just needed to indent the:
for element in del_data:
del_url = server + api_path + element
try:
so that it ran under:
for element in elementsfile:
del_data = {
element["objectid"],
}


f = open("del-hostid.csv")
elementsfile = csv.DictReader(f)

for element in elementsfile:
del_data = {
element["objectid"],
}

for element in del_data:
del_url = server + api_path + element
try:
print(del_url)
r = requests.delete(del_url, headers=headers, verify=False)
status_code = r.status_code
resp = r.text
log = open('delete_objects.log', 'a')
print(" Status code: "+str(status_code))
json_resp = json.loads(resp)
log.write('\n=====\n')
log.write(json.dumps(json_resp,sort_keys=True,indent=4, separators=(',', ': ')))

if status_code == 201 or status_code == 202:
print(" SUCCESS ")
elif status_code == 400:
print((" Message: ")+ resp + ('\n'))
else:
r.raise_for_status()
print((" Message: ")+ resp + ('\n'))



except requests.exceptions.HTTPError as err:
print("Connection error -> "+str(err))
finally:
if r: r.close()

[Maneck_UK]

I have updated my code to reflect the suggested changes and I can now delete one object from the csv file. Which is a big  improvement, thanks. But only the last object in my csv deletes. 

Also, if there is an object that is already deleted the, script ends and reports the 404 error. My attempts to update the error msg operation from 400 to 404 (elif status_code == 404:) to help get past the 404 results in an error event:

Error:

Status code: 404
Message: {"error":{"category":"FRAMEWORK","messages":[{"description":"No resource found"}],"severity":"ERROR"}}

But when my script ends because of the object already being deleted I do get a 404 error:

Error:

Status code: 404
Connection error -> 404 Client Error: Not Found for url: https://v-uxfmc01.brunel.ac.uk/api/fmc_config/v1/domain/e276abec-e0f2-11e3-8169-6d9ed49b625f/object/hosts/005056A6-6972-0ed3-0000-090194314100

When using this operation in my working POST object from csv script, the script would continue past an object that already existed.

So now I'm looking for help again please....

Firstly in understanding why only the last object in my csv is used and how to get it to delete all the objects in the file?

and

How to continue past an object that no longer exists?

Many thanks for everyone's help so far, I would not have got this far without you.

Updated script

f = open("del-hostid.csv")
elementsfile = csv.DictReader(f)

for element in elementsfile:
    del_data = {
        element["objectid"],
    }

for element in del_data:
    del_url = server + api_path + element
    try:
        r = requests.delete(del_url, headers=headers, verify=False)
        status_code = r.status_code
        resp = r.text
        log = open('delete_objects.log', 'a')
        print(" Status code: "+str(status_code))
        json_resp = json.loads(resp)
        log.write('\n=====\n')
        log.write(json.dumps(json_resp,sort_keys=True,indent=4, separators=(',', ': ')))

        if status_code == 201 or status_code == 202:
            print(" SUCCESS ")
        elif status_code == 400:
            print((" Message: ")+ resp + ('\n'))
        else:
            r.raise_for_status()
            print((" Message: ")+ resp + ('\n'))

    except requests.exceptions.HTTPError as err:
        print("Connection error -> "+str(err))
    finally:
        if r: r.close()

print('\nLog file "delete_objects.log" appended\n')
input("Press <Enter> to return to CMD prompt")

[JonasKs · Accepted solution]

In order to not break the script if a resource is not found, you can simply add a if status_code == 404 to your logic, like this:

        <... your code>
        elif status_code == 400:
            print(" Message: " + resp + '\n')
        elif status_code == 404:
            print(" Message: " + resp + '\n')
        else:
            r.raise_for_status()
            print(" Message: "+ resp + '\n')
        <...your code>

It's the 

r.raise_for_status()

that actually raises an exception, making the program crash. (Unless you catch that exception)

[Maneck_UK]

@JonasKs Top notch help, thanks

I had tried this but it had not worked, it turns out the msg handling failure was due to the incorrect indentation which had stopped my script looping through the csv file.

[JonasKs]

Aha, yes, 4 spaces per indention is the way to go. I really recommend using an editor that help you see those things, such a PyCharm or Visual studio code with the Python plugins. Glad you solved it. 

[Maneck_UK]

I have one issue remaining, if anyone could help with it; this would be amazing:

If there is an object that is already deleted the, script ends and reports a 404 error. My attempts to update the error msg operation from 400 to 404 (elif status_code == 404:) to help get past the 404 results in an error event:

Error:

Status code: 404
Message: {"error":{"category":"FRAMEWORK","messages":[{"description":"No resource found"}],"severity":"ERROR"}}

But when my script ends because of the object already being deleted I do get a 404 error:

Error:

Status code: 404
Connection error -> 404 Client Error: Not Found for url: https://v-uxfmc01.brunel.ac.uk/api/fmc_config/v1/domain/e276abec-e0f2-11e3-8169-6d9ed49b625f/object/hosts/005056A6-6972-0ed3-0000-090194314100

When using this operation in my working POST object from csv script, the script would continue past an object that already existed.

Anyone know how to fix my issue in getting past an object that no longer exists?

Thanks in advance 

[JonasKs]

By the way, if you click reply and then the insert code button it's easier for us to read your code.

[Maneck_UK]

I didn't know that. Thanks

[Maneck_UK · Accepted solution]

f = open("del-hostid.csv")
elementsfile = csv.DictReader(f)

for element in elementsfile:
    del_data = {
        element["objectid"],
    }

    for element in del_data:
        del_url = server + api_path + element
        try:
            print(del_url)
            r = requests.delete(del_url, headers=headers, verify=False)
            status_code = r.status_code
            resp = r.text
            log = open('delete_objects.log', 'a')
            print(" Status code: "+str(status_code))
            json_resp = json.loads(resp)
            log.write('\n=====\n')
            log.write(json.dumps(json_resp,sort_keys=True,indent=4, separators=(',', ': ')))
        
            if status_code == 201 or status_code == 202:
                print(" SUCCESS ")
            elif status_code == 400:
                print((" Message: ")+ resp + ('\n'))
            else:
                r.raise_for_status()
                print((" Message: ")+ resp + ('\n'))
    
        except requests.exceptions.HTTPError as err:
            print("Connection error -> "+str(err))
        finally:
            if r: r.close()

[JonasKs]

f = open("del-hostid.csv")
elementsfile = csv.DictReader(f)

for element in elementsfile:
    del_data = {
        element["objectid"],
    }

for element in del_data:
    del_url = server + api_path + element
    try: 
        print(del_url)
        r = requests.delete(del_url, headers=headers, verify=False)
        status_code = r.status_code
        resp = r.text
        log = open('delete_objects.log', 'a')
        print(" Status code: "+str(status_code))
        json_resp = r.json()
        log.write('\n=====\n')
        log.write(json.dumps(json_resp,sort_keys=True,indent=4, separators=(',', ': ')))
        if status_code == 201 or status_code == 202:
            print(" SUCCESS ")
        elif status_code == 400 or status_code == 404:
            print(" Message: " + resp + '\n')
        else:
            r.raise_for_status()
            print(" Message: " + resp + '\n')

    except requests.exceptions.HTTPError as err:
        # This will only happen if the `r.raise_for_status()` line is hit
        print("Connection error -> "+str(err))

Try this.

I've changed 

elif status_code == 400:

to

elif status_code == 400 or status_code == 404:

and

json_resp = json.loads(resp)

to

json_resp = r.json()

as well as removed your double for-loop.