Jinja2 Inputs YAML, JSON or Python dic

[Jerems]

Hi Guys,

Do you agree with this statement :

Does YAML or JSON offers better control on datas you pass to Jinja2 in opposition to a simple Python dictionary ?

While writing this question i realized that we could consider of coding this control in the python script itself (beside the dictionary creation).

Thanks for your answer !

Jerems

[Nabsch · Accepted solution]

Hello Jerems,

Using Json/Yaml or python dict for me will not offer a better controls. It depends on what you are trying to achieve. 

For example, let's see you want to generate a configuration that you  have to push on 10 differents interface. You can  choose whatever you want but you will still be able to use a wrong interface name like "TOTO1/0/2".

Jinja is a template engine that allow us to use some data as input and a template in order to render the final document that you want to get.

The question is you have to think about what you are trying to modelize. Is it a list of string for example or a dict or a list of dictionnary.

For example ,if you want to push the same configuration in x interfaces . The best way would be a list . So then in order to use this list you can do it using two way in your python using the list directly or use a dict that contains the list.

Here an example

from jinja2 import Template

template_str = """
{% for intf  in interface_list %}
interface  {{ intf }}
mtu 4000
{% endfor %}
"""

template = Template(template_str)

interface_list = ["TenGigE0/3/0/0","TenGigE0/3/0/1"]

# if you want to modify more element ==> You have add to , variable_name = value and so on . So if you are using a lot of variable
print("Rendering first version", template.render(interface_list=interface_list)) 


# if you want to modify more element ==> You have just to add in the dict
data_to_render = {"interface_list":interface_list} 
print("Rendering second version", template.render(data_to_render))

Here the output that i got

Rendering first version

interface  TenGigE0/3/0/0
mtu 4000

interface  TenGigE0/3/0/1
mtu 4000

Rendering second version

interface  TenGigE0/3/0/0
mtu 4000

interface  TenGigE0/3/0/1
mtu 4000

You can also store your data in a json or yaml that you will load in your code then pass it to the template.

Here an example to convert a python object to yaml or dict. In the example, i used a dict.

import json
import yaml

interface_list = ["TenGigE0/3/0/0","TenGigE0/3/0/1"]
data_to_render = {"interface_list":interface_list} 

print("Yaml : \n", yaml.dump(data_to_render))

print("Json : \n", json.dumps(data_to_render))

The output that i got

Yaml :
 interface_list:
- TenGigE0/3/0/0
- TenGigE0/3/0/1
Json :
 {"interface_list": ["TenGigE0/3/0/0", "TenGigE0/3/0/1"]}

[Claudia de Luna · Accepted solution]

Hi ,

If I'm understanding the statement correctly, I'd have to say I do not agree.

JSON and YAML are serialization formats or ways to save your programatic data structures (lists, dictionaries, and combinations thereof)  to files (that is how I think of them) so they can be saved to storage and then easily and quickly consumed by your scripts.

At the end of the day you will need to convert your JSON or YAML file into a Python data structure.   Now, what is the best data structure for your Jinja2 templating?  That is a very interesting question.

Over the years, I have found that a list of dictionaries works best.

1. A list lets me put for loops in my templates and so if its one thing or more than one thing (list element), my template can handle it.

2. A list of lists works of course but you have to remember what index for what value which is great when you are "in the zone" for your script but 6 months later its not so great because at least for me, I don't remember.

3. So a list of dictionaries (lod) I find works best.   I can still iterate over the elements (one or more) and I can call things out specifically by their keys making my template very readable.

I put together a little repo to play around.
https://github.com/cldeluna/jinja2_example

You have already seen an example of a list from @Nabsch..  Eventually you will want to do more (hence 1-3 above) so lets look at a list of lists:

    payload_list = [
     ["5", "vlan5_1.1.1.0_24"],
     ["7", "vlan7_7.1.1.0_24"],
     ["10", "vlan10_10.1.1.0_24"],
    ]

and its corresponding jinja2 template:

! Layer 2 Vlan Template
! using a List of Lists

{% for line in mylistvar %}
vlan {{ line[0] }}
  name {{ line[1] }}
{% endfor %}

If your list has one item or many items you will get:

--- Template Rendering with a List of Lists Configuration Payload
! Layer 2 Vlan Template
! using a List of Lists


vlan 5
  name vlan5_1.1.1.0_24

vlan 7
  name vlan7_7.1.1.0_24

vlan 10
  name vlan10_10.1.1.0_24

For something this simple thats just fine but I find this template serves me better when I have to come back to it a few months from now:

! Layer 2 Vlan Template
! using a List of Dictionaries for improved readability

{% for line in mylistvar %}
vlan {{ line['vlan'] }}
  name {{ line['name'] }}
{% endfor %}

This not an idea example because of its simplicity but I hope you can see that given a far more complicated configuration referring to values by their keys makes for a much more readable template and will shortcut the inevitable time spent trying to figure out what you did 6 months ago.

In the little example script in the GitHub Repo j2_payload_comparison.py I conclude with saving the list of dictionaries payload to a JSON file and reading it back in.

[
    {
        "vlan": "5",
        "name": "vlan5_1.1.1.0_24"
    },
    {
        "vlan": "7",
        "name": "vlan7_7.1.1.0_24"
    },
    {
        "vlan": "10",
        "name": "vlan10_10.1.1.0_24"
    }
]

JSON and YAML will let you save your Python data structures to a file and easily read those data structures back in as data structures.  That is their value.

JSON vs YAML is of course another question.  I use YAML if I need to add comments and if I have to share the payload with others who are not very familiar with scripting because its a bit easier to read (less intimidating) if you have not been exposed to this.
Normally, if I've built out a configuration payload, I will save it to JSON so I have a machine readable snapshot of the payload I used to generate configs or whatever.

I will mention that for complicated templates, lets say a full configuration, I will build a dictionary of lists where the key is the config section, say "SVI" and the value of that key is a list of dictionaries holding the configuration data for all of the SVIs for that device.

Hopefully I've not made your head hurt too much!

If I have, this little post about "Decomposing Data Structures" might help.

Hope some of this is useful to you!  Happy Templating!

[Jerems]

Thousand thanks to both of you !

Cheers,

Jerems