cisco ACI - Error 400

[titusroz03]

Dear ACI Experts,

We are running a problem with ACI, none of our BDs are able to get submitted and we are getting the below 400 error for all BDs in our prod tenant. we have mapped all our BDs to EPGs in one-to-one basis.

subnets are configured in EPG and BD with below scope options.

EPG scope option settings

BD Scope option settings

when we raised TAC they are saying to download and do offline modification on scope settings, but this issue is not only with scope settings but throughout the BD, any changes we do in BD we couldn't submit.

We used a TEST BD and moved it to common vrf and tested it works fine, could modify any options. But the same we tried to revert back to prod-vrf but couldn't do it

Can anyone help me with this

[titusroz03]

@RedNectar Thanks for your reply, I already tried this workaround but it didn't help. I deleted the Subnet in EPG and tried to change settings in BD but couldn't submit. I am getting the same error.

Next, I deleted the subnet in BD and trying to create it again, again I am facing the same error.

Subnets are created in EPG for inter VRF leak setup when ACI was running in version 2.0. Now it is upgraded to 5.2 since we are facing this problem.

[titusroz03]

@RedNectar

To answer the question-"Why are you configuring an IP address on BOTH the EPG and the BD?" , it was configured by a TEAM seven years back, who were before me managing this fabric and it was for inter-leaking the route between Vrfs. 

 I tried the above option, but in different EPG since the above BD is critical, but still no luck. I am seeing the same error.

Deleted the EPG subnet and tried to change BD, still same error.

Then Deleted BD subnet and trying to create subnet in BD still same error.

TAC gave us the below solution:

Export tenant configuration,  do offline modification by changing subnet scope same for BDs & EPGs and then push modified configuration.

++ Above action plan has been tested in TAC Lab and it worked.

But we are a bit skeptical/unconfident about the unknown issues when doing this offline procedure. Because we have critical and non-critical BDS in both prod and non-prod tenants and we were suggested to do the scope settings in all the BDs in both the tenants at same time. 

Any suggestions or advice from your end.

[RedNectar]

Hi @titusroz03 ,

The TAC approach sounds sensible. I'm gathering that the problem lies in the fact that when it was first configured in an older version of ACI, it was possible to have different scopes for the BD and EPG. I vaguely remember that this was the case. And in later versions there were safeguards put into the GUI to stop users doing that.  I'm still not sure why you got the same error when you:

• Deleted the EPG subnet and tried to change BD, still same error.
• Then Deleted BD subnet and trying to create subnet in BD still same error.

Now - the logic of modifying the tenant config offline and pushing it again will work because the API does not do the same sanity checks as the GUI. But there are some caveats. Pushing config using the API is ADDITIVE - unless you know how to delete items in the JSON config that you'll push.

Let's make sure that you have a good plan to do the changes. My suggested plan is the following. You may wish to have TAC verify it so you can blame them rather than me if it doesn't work

Since you mention "both tenants", I assume that the problem exists in two tenants.  Personally I don't see any problem with doing the tenants one at a time, but given your advice is to do both tenants together, I'll work with that - it just makes the process a bit more complicated to push the configuration of two tenants together rather than one-at-a-time.

[Edit: I've just spent an hour trying to post the config of two tenants configured in one file - just the very basic config - name and dn. And only the first tenant gets posted, so I've given up on that idea]

Here's my suggested plan:

Step 1: Save the configuration of both tenants (in JSON format)

1. Right-click on each tenant and choose Save as ...
2. 
   

   set the options as above and click Download

3. Repeat for the other tenant.

Step 2: Make sure the scope of each subnet in each BD is set to "public,shared"

1. Open each json file in your favourite editor. I'll assume VS Code for my illustrations.
2. Find the scope attribute under the fvSubnet  object which is one of the child objects of the BD (fvBD)
3. Make sure the scope is set to "public,shared" as shown below
    
4. Repeat for every subnet for every BD in BOTH JSON files (i.e. both tenants)

Step 3: Make sure the scope of each subnet in each EPG is set to "public,shared"

1. I'll assume the VS Code editor is still open
2. Find the scope attribute under the fvSubnet object under every EPG (fvAEPg) and make sure it is set to "public,shared" - much the same as for the BDs, but don't forget that the fvAEPg object is under the fvAp object. See example below.
   
3. Repeat for every subnet for every EPG in BOTH JSON files (i.e. both tenants)
4. Save both files - best to save with a different name so you don't clobber the originals if anything goes wrong.

STEP 4: Test your JSON files. IMPORTANT STEP

You can screw your configuration monumentally if you push your config and it is wrong. Use a ACI Simulator or a lab install to see if you can POST your updated JSON config files without error by

1. right-clicking on any tenant and choosing Post 
2. Make sure the Parnet DN is set to uni/ and choose your updated file to post 
   
3. Click Post and watch for errors, if none, look at the posted config to check it looks OK

STEP 5: Back up your existing config

Nothing else to be said. You've been warned. Very easy to save a snapshot of your whole config (or just one tenant) and to roll back if all goes wrong.

STEP 6: Post to your production fabric (out-of-hours of course)

Unfortunately, you'll have to do this one tenant at a time, (as I explained in my edit above) but that shouldn't be a problem

STEP 7: (optional but recommended) Consider deleting the Subnet under each EPG [edit] or BD [/edit]

[Edit: About 5:30 this morning I woke up with the though that MAYBE you are exporting contracts between Tenants, and that is why you were configuring subnets under the EPG. IF you are indeed exporting contracts from one tenant to another, then the EPG that is providing the contract MUST have the subnet defined under the EPG. If this is the case, I'd advise deleting the subnet under the BD rather than the EPG]

I consider having the same subnet defined in two different places just confusing, especially since you'll never be able to change the scope either while the other exists (you'll get the same error if you try)

HTH

[titusroz03]

@RedNectar   Thanks for the detailed steps of the above process. It will be really helpful for me to take this up with my customer.

While I was checking with TAC , they told that rollback option is not possible in this procedure but as per your advice we can rollback this if something goes wrong after POST in the fabric..? Could you provide more detail on how to roll back if the POST corrected config didn't work..?

[RedNectar]

Hi @titusroz03 ,

OK. I'm guessing that the TAC are saying that a rollback won't work because your existing config has errors, and if you take a snapshot as it is, then try and restore that snapshot (along with its errors) then the restore will probably fail. I hadn't thought about that angle when I suggested the snapshot. As for your request...

---

Could you provide more detail on how to roll back if the POST corrected config didn't work..?

---

I'm afraid that if the POST config fails it will probably make no changes at all, in which case you'll just have to edit the JSON until it does stick (which is why the testing on a simulator or test fabric is so important)

The danger when using POST is that there are fundamental logic errors in the JSON, such as using the wrong BD names where the EPG is linked to a BD. In your case, given that the edits are minimal, this risk is small.  These kind of errors usually occur when you say take the JSON config (for example) of one tenant and try to edit it to push the changes to a different tenant - and you forget to edit ALL of the relationships you need. Don't ask me how I know this ;(

So hang in there, double check your JSON before pushing to production, and do it out of hours. You may wish to consider booking a pre-emptive call with TAC during the process.

[titusroz03]

@RedNectar Thanks once again for all your valuable suggestions. For the roll back can we duplicate the Json files so we can do the changes in one copy and try to POST it and if there is any issues during the POST process or after the POST, then we can roll back the other JSON file...? Is this possible and what would be the unknown challenges in this...?

[RedNectar]

Hi @titusroz03 ,

Your logic sounds like it should work........BUT. The problem is that since your existing config seems to have some errors in it, you might be able to save it OK, but if you were to try and POST it back, it is likely to throw an error.

BACKGROUND: Earlier I tried to create a JSON config with the EPG and BD parameters set as per your original post. When I tried to POST that to ACI using the API, the API refused to accept it - giving a similar message about the "Subnets with matching IP must have the same scope"

If you take the JSON config files for the existing config for your two tenants, try to POST them on a Simulator of the same version as your production system. If no "scope" errors occur, then your logic of:

...then we can roll back the other JSON file...?

should be possible.

Good luck, let us know how it goes. If you want to PM me any JSON files to check first I'll give it a shot (although turn around might be a bit slow - about to go for a short driving/hiking holiday next week)

[titusroz03]

@RedNectar Thank you again for your responses. I have asked my customer for a VM to build a ACI Simulator where I can run both PRE-Change configs and POST change configs before going to production. Will keep you posted.

Meanwhile could you share me the MOQUERY for collecting the subnets under EPGs or any idea for that..? Because we are going to modify only the scope options on EPGs configured with subnets and enabled with Share-VRF, I want to collect those EPG information in both tenants. Could you help me..?

[RedNectar]

Hi @titusroz03 ,

I had to drag up a blog post of mine to answer this one:

But I think this will get the information you need:

apic1# bash ;#You need to be at the BASH prompt so you can enter the ? character in the command
admin@apic1:~> icurl -s "http://localhost/api/node/class/\
fvAEPg.json?\
rsp-prop-include=naming-only\
&query-target=subtree\
&target-subtree-class=fvAEPg\
&rsp-subtree=children\
&rsp-subtree-class=fvSubnet\
&rsp-subtree-include=required" | jq 

There may be an equivalent in moquery but I find icurl more flexible, more accurate and uses syntax that is more like that seen in the GUI

[titusroz03]

@RedNectar  Will check with the above commands,thanks for the same. I want to clarify on an Alternate solution which TAC provided which was to downgrade the controller alone to previous version 4.2(4O) and remediate the Scope issue, but I am little doubtful if this version will be compatible with our current running NX-OS(15.2(8i)), Does downgrading the APIC alone will provide a solution for this...? In 4.2 version I was orally confirmed from my colleague that this scope mismatch error didn't occur, but how can I ensure in written from cisco on this..?

Your thoughts and opinions. How can I check the compatibility of these APIC and leaf switches..?

My Second thought which came to my mind is we have APIC controllers 2 in X DC and one in Y DC and all the three are in one cluster but physically in different Datacenters. Y is our Pre-prod Data center which is less critical than X. Hence, we would need to explore an option to downgrade the Y, APIC alone to 4.2 and make it as standalone controller to make the online remediation through this controller

[RedNectar]

Hi @titusroz03 ,

Downgrading scares me at the best of times. And a LOT of work - HOURS of it. If you can sort it out with pushing modified configs seems much simpler than me. But then I don't get paid by Cisco to make sure it is right, so my advice has to be to do what TAC tells you (I have not always agreed with the TAC approach, but they do have access to information that normal people don't)

The path of making one of your APICs a stand-alone for a period sounds like a design for disaster - an absolute last resort. (Sorry).  Breaking a cluster and re-building it is not trivial. Maybe if you could find an extra APIC to use as a stand-alone controller for testing might be better.

[titusroz03]

@RedNectar Even I agree with your point on offline modification is low risky than downgrading the controllers, but my customer is bit nervous on Exporting and importing the tenant config during the process and they feel that downgrading is quite safer option. Still, it is pending under their decision, we are planning to setup lab for both 5.2 and 4.2 versions to test both the options.

And for the standalone option,even cisco TAC rejected that plan so no comments on that

And on the icurl commands you have shared with me I just want to understand how I can execute those commands and get the output. Will this output be recorded in Json file which I need to export that or will it be displayed once I execute the commands..?Could you help me how to execute it

[RedNectar]

Hi @titusroz03 ,

Re:

---

And on the icurl commands you have shared with me I just want to understand how I can execute those commands and get the output. Will this output be recorded in Json file which I need to export that or will it be displayed once I execute the commands..?Could you help me how to execute it

---

Maybe I should get you to read my blog! - where you will see examples of similar output (if you click on show result)

When you execute:

apic1# bash ;#You need to be at the BASH prompt so you can enter the ? character in the command
admin@apic1:~> icurl -s "http://localhost/api/node/class/\
fvAEPg.json?\
rsp-prop-include=naming-only\
&query-target=subtree\
&target-subtree-class=fvAEPg\
&rsp-subtree=children\
&rsp-subtree-class=fvSubnet\
&rsp-subtree-include=required" | jq 

• The apic1  prompt shows that the commands are issued at the CLI of the APIC
• the bash  command shows that you leave the standard APIC CLI and enter the bash shell - this is necessary only because you need to include a ? character in the command.
• the icurl -s "http://localhost/api/node/class/fvAEPg.json?rsp-prop-include=naming-only&query-target=subtree&target-subtree-class=fvAEPg&rsp-subtree=children&rsp-subtree-class=fvSubnet&rsp-subtree-include=required"  part of the command creates the output at the bash prompt in JSON format - but it will be in a single string almost impossible for a human to read
• the | jq  part of the command takes the output and pipes it through the jq (JSON Query) application to turn the output into something human readable
• From here, you have two options
  
  1. (simplest) simply highlight and copy the output and copy it. Paste it to a file and job done.
  2. Add > epgsWithSubnets.json to the end of the command to create a file called epgsWithSubnets.json in the current directory of the APIC - then copy that off somehow

[titusroz03]

@RedNectar Hope you are doing good. After long time I am back here to clarify some additional points on testing.

I am trying to setup a ACI Simulator for 5.0(8) version and it have a 8 files need to be concatenated into one file.. which is too huge.. Is there any possibilities for having it compressed or any other way to have a small file size.

And second thing is I want to have more knowledge on the Json file editor(VB code editor) which you used for editing the file. Could you share some videos and posts on that..?

[RedNectar]

Hi @titusroz03 ,

I am trying to setup a ACI Simulator for 5.0(8) version and it have a 8 files need to be concatenated into one file.. which is too huge.. Is there any possibilities for having it compressed or any other way to have a small file size.

Oh what a pain! But no, you have to download all the files and then join them together. I agree, an absolute pain. Luckily I have someone who looks after our lab who does it for our team so I son't have to (unless I want to run it on my home kit). Curiously though (I just checked the website) v5.0(2h) has only 4 files. You'll probably not notice any difference in functionality, so I reckon your best bet is to give v5.0(2h) a shot. Although, unless you are trying to match your production system, I'd suggest learning on v5.2 or later because the way Access Policies work changed. (for the better) in v5.2

And second thing is I want to have more knowledge on the Json file editor(VB code editor) which you used for editing the file. Could you share some videos and posts on that..?

---

The editor I use (and most of the rest of the world) is Microsoft's Visual Studio Code editor. Now here's a curious thing. Although it's Microsoft product, it is actually a Github project - in other words anyone can clone it, change it etc. And just recently, Google did just that and have released their own version of an editor - as far a I can see is an exact copy of VS Code. It is called Antigravity.

So if you are a Microsoft fan, download from http://code.visualstudio.com (or https://github.com/microsoft/vscode) but if you want to go the Meta path, try https://antigravity.google/  - however they don't seem to have shared their version on github