topic Re: 404 error in Python script to DELETE objects from FMC using a CSV file in Network Security

404 error in Python script to DELETE objects from FMC using a CSV file

Maneck_UK — Sun, 07 Jun 2020 11:00:43 GMT

I have a fully working python script that can POST objects to the FMC (6.5.0.4) API using a .csv file for the data.

I want to use this same approach/method to DELETE objects, but after many days I have been unable to get the script to work.

Initially I was adding the object to the script as data rather than in the URL, I fixed this using a single object, if I add the object into the script using:

del_data = ['005056A6-6972-0ed3-0000-090194313992']

the URL works and the objects deletes.

But this defeats the purpose of the script as I can delete a single object in simpler ways.

I believe I now have a script that adds the host object ID to the url but it continues to fail with a url 404 error:

Connection Error --> 404 Client Error: Not Found for url: https://<FMC-NAME>/api/fmc_config/v1/domain/e276abec-e0f2-11e3-8169-6d9ed49b625f/object/hosts/005056A6-6972-0ed3-0000-090194314504

I'm new to this Python scripting business and I would be grateful if anyone has any pointers.

The relevant parts to my script are below:

Thanks in advance

URL build script component

f = open("delid.csv")
objectsfile = csv.DictReader(f)

for object in objectsfile:
del_data = {
object["objectid"]
}

for object in del_data:
del_url = server + api_path + object

try:
r = requests.delete(del_url, headers=headers, verify=False)
status_code = r.status_code
resp = r.text
log = open('delete_objects.log', 'a')
print(" Status Code: "+str(status_code))
json_resp = json.loads(resp)
log.write(json.dumps(json_resp,sort_keys=True,indent=4, separators=(',', ': ')))

if status_code == 201 or status_code == 202:
print(" SUCCESS ")
elif status_code == 400:
print((" Message: ")+ resp + ('\n'))
else:
r.raise_for_status()
print((" Message: ")+ resp + ('\n'))

except requests.exceptions.HTTPError as err:
print("Connection Error --> "+str(err))
finally:
if r: r.close()

Re: 404 error in Python script to DELETE objects from FMC using a CSV file

JonasKs — Sun, 07 Jun 2020 11:54:59 GMT

In your code you do this:

for object in del_data:
    del_url = server + api_path + object

try:
    ...

When you need to indent your `try` to actually call the `del_url` every time:

for object in del_data:
    del_url = server + api_path + object
    try:
        ... put code here.

Also never use `object` as a variable name, it shadows the Python builtin class `object`. Name it `element` or something instead.

For your question, I'd recommend printing out your URL before you call `r.delete(..)` in order to see if it's formatted correctly. The 404 indicates that what ever resource you're trying to reach, it does not exist. You have most likely already deleted it in a previous run, and since your current version don't indent correctly, you've only called the API towards the last element in your CSV.

Re: 404 error in Python script to DELETE objects from FMC using a CSV file

Maneck_UK — Sun, 07 Jun 2020 12:44:27 GMT

@JonasKs Thanks for the help.

At the first time of reading the information, I needed an emoji for "my head just exploded"

I've read through it a few more times and I think I understand; I will go and try it.....

Re: 404 error in Python script to DELETE objects from FMC using a CSV file

JonasKs — Sun, 07 Jun 2020 13:43:15 GMT

Hehe, I understand.

It really boils down to these steps:

1. Print the URL that you post to. This will make it clear that this is happening to your code. I'll try to write it out for you, though:

for object in objectsfile:
    del_data = {
        object["objectid"]
    }

In this snippet, you create a set of elements. del_data will now look like this:

del_data = {'first element', 'second element', 'third element'}

You then loop over that set:

for object in del_data:
    del_url = server + api_path + object

And then you exit the loop. So what's happening is this:

# first loop
del_url = 'first_element'
# second loop
del_url = 'second_element'
# third loop
del_url = 'third_element'

Now, the next thing you do is to execute your code:

try:
    r = requests.delete(del_url, headers=headers, verify=False)

But as you can see, you're not doing this inside your for-loop. So this only happens once. If we take a closer look at this request, the del_url is third_element, and I assume that you really wanted this to happen for every element in your set:

try:
    r = requests.delete('third_element', headers=headers, verify=False)

So, what I wrote above is that you should make this:

for object in del_data:
   del_url = server + api_path + object

try:
    r = requests.delete(del_url, headers=headers, verify=False)

look like this:

for object in del_data:
    del_url = server + api_path + object

    try:
        print(del_url)
        r = requests.delete(del_url, headers=headers, verify=False)

That will ensure that for every element in your del_data set, you will execute that command.

2. Don't call your variables object , it's an illegal name to use in Python, due to the fact that it's a built in class.

Re: 404 error in Python script to DELETE objects from FMC using a CSV file

Maneck_UK — Sun, 07 Jun 2020 13:49:04 GMT

I have updated my code to reflect the suggested changes and I can now delete one object from the csv file. Which is a big improvement, thanks. But only the last object in my csv deletes.

Also, if there is an object that is already deleted the, script ends and reports the 404 error. My attempts to update the error msg operation from 400 to 404 (elif status_code == 404:) to help get past the 404 results in an error event:

Error:

Status code: 404
Message: {"error":{"category":"FRAMEWORK","messages":[{"description":"No resource found"}],"severity":"ERROR"}}

But when my script ends because of the object already being deleted I do get a 404 error:

Error:

Status code: 404
Connection error -> 404 Client Error: Not Found for url: https://v-uxfmc01.brunel.ac.uk/api/fmc_config/v1/domain/e276abec-e0f2-11e3-8169-6d9ed49b625f/object/hosts/005056A6-6972-0ed3-0000-090194314100

When using this operation in my working POST object from csv script, the script would continue past an object that already existed.

So now I'm looking for help again please....

Firstly in understanding why only the last object in my csv is used and how to get it to delete all the objects in the file?

and

How to continue past an object that no longer exists?

Many thanks for everyone's help so far, I would not have got this far without you.

Updated script

f = open("del-hostid.csv")
elementsfile = csv.DictReader(f)

for element in elementsfile:
del_data = {
element["objectid"],
}

for element in del_data:
del_url = server + api_path + element
try:
r = requests.delete(del_url, headers=headers, verify=False)
status_code = r.status_code
resp = r.text
log = open('delete_objects.log', 'a')
print(" Status code: "+str(status_code))
json_resp = json.loads(resp)
log.write('\n=====\n')
log.write(json.dumps(json_resp,sort_keys=True,indent=4, separators=(',', ': ')))

if status_code == 201 or status_code == 202:
print(" SUCCESS ")
elif status_code == 400:
print((" Message: ")+ resp + ('\n'))
else:
r.raise_for_status()
print((" Message: ")+ resp + ('\n'))

except requests.exceptions.HTTPError as err:
print("Connection error -> "+str(err))
finally:
if r: r.close()

print('\nLog file "delete_objects.log" appended\n')
input("Press <Enter> to return to CMD prompt")

Re: 404 error in Python script to DELETE objects from FMC using a CSV file

Maneck_UK — Sun, 07 Jun 2020 14:06:40 GMT

@JonasKs Thanks, I wrote my update post without seeing this update (my post took ages to write). This is great help. I will take another look.

Re: 404 error in Python script to DELETE objects from FMC using a CSV file

JonasKs — Sun, 07 Jun 2020 14:27:28 GMT

In order to not break the script if a resource is not found, you can simply add a if status_code == 404 to your logic, like this:

        <... your code>
        elif status_code == 400:
            print(" Message: " + resp + '\n')
        elif status_code == 404:
            print(" Message: " + resp + '\n')
        else:
            r.raise_for_status()
            print(" Message: "+ resp + '\n')
        <...your code>

It's the

r.raise_for_status()

that actually raises an exception, making the program crash. (Unless you catch that exception)

Re: 404 error in Python script to DELETE objects from FMC using a CSV file

Maneck_UK — Sun, 07 Jun 2020 14:35:54 GMT

I've looked over the extra help and I have cracked the deletion part. I just needed to indent the:
for element in del_data:
del_url = server + api_path + element
try:
so that it ran under:
for element in elementsfile:
del_data = {
element["objectid"],
}

f = open("del-hostid.csv")
elementsfile = csv.DictReader(f)

for element in elementsfile:
del_data = {
element["objectid"],
}

for element in del_data:
del_url = server + api_path + element
try:
print(del_url)
r = requests.delete(del_url, headers=headers, verify=False)
status_code = r.status_code
resp = r.text
log = open('delete_objects.log', 'a')
print(" Status code: "+str(status_code))
json_resp = json.loads(resp)
log.write('\n=====\n')
log.write(json.dumps(json_resp,sort_keys=True,indent=4, separators=(',', ': ')))

if status_code == 201 or status_code == 202:
print(" SUCCESS ")
elif status_code == 400:
print((" Message: ")+ resp + ('\n'))
else:
r.raise_for_status()
print((" Message: ")+ resp + ('\n'))

except requests.exceptions.HTTPError as err:
print("Connection error -> "+str(err))
finally:
if r: r.close()

Re: 404 error in Python script to DELETE objects from FMC using a CSV file

Maneck_UK — Sun, 07 Jun 2020 14:39:53 GMT

I have one issue remaining, if anyone could help with it; this would be amazing:

If there is an object that is already deleted the, script ends and reports a 404 error. My attempts to update the error msg operation from 400 to 404 (elif status_code == 404:) to help get past the 404 results in an error event:

Error:

Status code: 404
Message: {"error":{"category":"FRAMEWORK","messages":[{"description":"No resource found"}],"severity":"ERROR"}}

But when my script ends because of the object already being deleted I do get a 404 error:

Error:

When using this operation in my working POST object from csv script, the script would continue past an object that already existed.

Anyone know how to fix my issue in getting past an object that no longer exists?

Thanks in advance

Re: 404 error in Python script to DELETE objects from FMC using a CSV file

JonasKs — Sun, 07 Jun 2020 14:50:46 GMT

By the way, if you click reply and then the insert code button it's easier for us to read your code.

Re: 404 error in Python script to DELETE objects from FMC using a CSV file

Maneck_UK — Sun, 07 Jun 2020 15:01:51 GMT

I didn't know that. Thanks

Re: 404 error in Python script to DELETE objects from FMC using a CSV file

Maneck_UK — Sun, 07 Jun 2020 15:02:50 GMT

f = open("del-hostid.csv")
elementsfile = csv.DictReader(f)

for element in elementsfile:
    del_data = {
        element["objectid"],
    }

    for element in del_data:
        del_url = server + api_path + element
        try:
            print(del_url)
            r = requests.delete(del_url, headers=headers, verify=False)
            status_code = r.status_code
            resp = r.text
            log = open('delete_objects.log', 'a')
            print(" Status code: "+str(status_code))
            json_resp = json.loads(resp)
            log.write('\n=====\n')
            log.write(json.dumps(json_resp,sort_keys=True,indent=4, separators=(',', ': ')))
        
            if status_code == 201 or status_code == 202:
                print(" SUCCESS ")
            elif status_code == 400:
                print((" Message: ")+ resp + ('\n'))
            else:
                r.raise_for_status()
                print((" Message: ")+ resp + ('\n'))
    
        except requests.exceptions.HTTPError as err:
            print("Connection error -> "+str(err))
        finally:
            if r: r.close()

Re: 404 error in Python script to DELETE objects from FMC using a CSV file

JonasKs — Sun, 07 Jun 2020 16:13:30 GMT

f = open("del-hostid.csv")
elementsfile = csv.DictReader(f)

for element in elementsfile:
    del_data = {
        element["objectid"],
    }

for element in del_data:
    del_url = server + api_path + element
    try: 
        print(del_url)
        r = requests.delete(del_url, headers=headers, verify=False)
        status_code = r.status_code
        resp = r.text
        log = open('delete_objects.log', 'a')
        print(" Status code: "+str(status_code))
        json_resp = r.json()
        log.write('\n=====\n')
        log.write(json.dumps(json_resp,sort_keys=True,indent=4, separators=(',', ': ')))
        if status_code == 201 or status_code == 202:
            print(" SUCCESS ")
        elif status_code == 400 or status_code == 404:
            print(" Message: " + resp + '\n')
        else:
            r.raise_for_status()
            print(" Message: " + resp + '\n')

    except requests.exceptions.HTTPError as err:
        # This will only happen if the `r.raise_for_status()` line is hit
        print("Connection error -> "+str(err))

Try this.

I've changed

elif status_code == 400:

elif status_code == 400 or status_code == 404:

and

json_resp = json.loads(resp)

json_resp = r.json()

as well as removed your double for-loop.

Re: 404 error in Python script to DELETE objects from FMC using a CSV file

JonasKs — Sun, 07 Jun 2020 16:56:39 GMT

Weird, my reply got discarded I think?

I'll repost it:

f = open("del-hostid.csv")
elementsfile = csv.DictReader(f)

for element in elementsfile:
    del_data = {
        element["objectid"],
    }

for element in del_data:
    del_url = server + api_path + element
    try: 
        print(del_url)
        r = requests.delete(del_url, headers=headers, verify=False)
        status_code = r.status_code
        resp = r.text
        log = open('delete_objects.log', 'a')
        print(" Status code: "+str(status_code))
        json_resp = r.json()
        log.write('\n=====\n')
        log.write(json.dumps(json_resp,sort_keys=True,indent=4, separators=(',', ': ')))
        if status_code == 201 or status_code == 202:
            print(" SUCCESS ")
        elif status_code == 400 or status_code == 404:
            print(" Message: " + resp + '\n')
        else:
            r.raise_for_status()
            print(" Message: " + resp + '\n')

    except requests.exceptions.HTTPError as err:
        print("Connection error -> "+str(err))

You had a double for-loop which wasn't needed.

# Old : elif status_code == 400:
# Old :     print((" Message: ")+ resp + ('\n'))

# New : elif status_code == 400 or status_code == 404: 
# New :     print(" Message: " + resp + '\n')

I've changed these two things:

# Old : json_resp = json.loads(resp)
# New : json_resp = r.json()

Re: 404 error in Python script to DELETE objects from FMC using a CSV file

Maneck_UK — Sun, 07 Jun 2020 18:54:14 GMT

@JonasKs Top notch help, thanks

I had tried this but it had not worked, it turns out the msg handling failure was due to the incorrect indentation which had stopped my script looping through the csv file.

Re: 404 error in Python script to DELETE objects from FMC using a CSV file

JonasKs — Sun, 07 Jun 2020 19:12:54 GMT

Aha, yes, 4 spaces per indention is the way to go. I really recommend using an editor that help you see those things, such a PyCharm or Visual studio code with the Python plugins. Glad you solved it.

Re: 404 error in Python script to DELETE objects from FMC using a CSV file

Maneck_UK — Sun, 07 Jun 2020 19:20:20 GMT

@JonasKs, damned annoying your earlier post disappeared, the post you wrote about how the script worked through the loops allowed me to work it out and understand it.
I just spent several hours re-writing all of my scripts cleanly so that I could re-test the POST objects, GET host object ID and create .csv and then the all important DELETE objects.
I completed testing using my script prior to your final update from Hotfix and whilst it wasn't as clean as your script I had everything working 🙂
In order to test the new changes I will need to recreate the all the objects again and then run the extractions, remove an object to test the error aspect. All of which I will do, however it will be tomorrow now.
I understand the clean up of the messaging but what does the following achieve please?
# Old : json_resp = json.loads(resp)
# New : json_resp = r.json()

Re: 404 error in Python script to DELETE objects from FMC using a CSV file

JonasKs — Sun, 07 Jun 2020 19:38:32 GMT

First time really replying in here, might've been me messing up something.

So you do this:

resp = r.text
json_resp = json.loads(resp)

What this does is that it converts the JSON into a Python dictionary. How ever, the response object already has this built in. In other words, you can change those two lines into this

json_resp = r.json()

This works since `.json()` is a function that takes the response.text and converts it into a Python dictionary. So it’s doing the exact same thing, it’s just often easier to do, so I wanted to show you 🙂 sorry if it confused you.

Re: 404 error in Python script to DELETE objects from FMC using a CSV file

Maneck_UK — Sun, 07 Jun 2020 20:21:30 GMT

Thanks,
I will try that version tomorrow. I was confused at the start of this post but thanks to your help I have a significantly improved understanding :-).
I also had a post in my more normal reading location in the networking section of the forums, I've tried to link that back to you too, so you get the Kudos from that post also.
Thanks for your time and effort. It is very much appreciated.

Re: 404 error in Python script to DELETE objects from FMC using a CSV file

JonasKs — Mon, 08 Jun 2020 05:33:48 GMT

You’re welcome. 🙂 I appreciate the helpful votes!

Good luck!