DevOps

  Inspiration and Purpose   The idea for this project came from our team’s desire to attach audio versions of the blogs and articles we post to the Cisco Community. This will enable those who are visually impaired and give everyone else the option to listen instead of reading. I also wanted to have a place to store all my blog and article documents, other than a folder on my desktop. This source of truth (SOT) of text templates and their associated audio MP3s will also serve as an archive for future use of this knowledge. The continuous integration (CI) pipeline which I’m about to share is worth sharing because it accomplishes two things in particular, which you may have assumed GitHub Actions would do out of the box, but for which I had to do  some research to find and implement niche tools . These two things are:   1) Detect exactly which file(s) have changed in a Push to GitHub and assign the properties of those files to variables. 2) After running a script using those variables, automatically Commit and Push back into the GitHub repository.     Setting up the Directories in GitHub   We almost always post in the Developer Hub area of the Cisco Community, so I just needed to model that structure in GitHub. Here’s the contents of the Developer Hub, as seen online:   I created a new GitHub repository (repo) named “community-content-pipeline” and added a directory called “developer-hub” where I replicated the structure from the website.     Now, whenever I post a new article or blog in the Developer Hub, I will also upload my original docx document here, in the appropriate subfolder. Now, let’s automate some CI magic!     GitHub Actions Workflow   In order to setup a GitHub Action, we create a YAML file in a /.github/workflows/ directory we create in our GitHub repo. Mine is laid out like this: community-content-pipeline/.github/workflows/make_mp3_on_push.yml   The first thing I did, besides naming it, was to set this workflow to only run on a PUSH to the repo and only if the files being pushed are located in the developer-hub directory and end in ‘.docx’ (this is the standard file format currently used by Microsoft Office – Word).   name: Create MP3 from .docx upon Push to a developer-hub directory on: push: #only runs when files in this path are changed, and they must end in .docx paths: - developer-hub/**.docx   Now, I needed to find a way to see exactly which file changed in the Push and assign it to a variable for working with it in the Python script we’ll compose later. This action does just the trick, and it’s found on GitHub Marketplace as well: trilom/file-changes-action   Here’s how I’ve used it in the workflow to see which files are added or modified and feed them to the Python script we’ll make:   - id: file_changes # see which file changed in the push, in '' string output uses: trilom/file-changes-action@v1.2.3 with: outout: '' fileOutput: '' - name: test # print modified and added files run: | echo 'modified file(s) = ${{ steps.file_changes.outputs.files_modified}}' echo 'added file(s) = ${{ steps.file_changes.outputs.files_added}}' - name: execute py script # run file env: MODIFIED_FILE: ${{ steps.file_changes.outputs.files_modified}} ADDED_FILE: ${{ steps.file_changes.outputs.files_added }} run: | python3 action_txt_2_mp3.py   Ok, the workflow is almost done. Now, I needed to tell the workflow to commit the changes the Python script will make and push them back into the repo. That is accomplished, thanks to the following GitHub Action, also found on the GitHub Marketplace: ad-m/github-push-action    Here's that part of the workflow:   - name: Commit files # transfer the new files back into the repository run: | git config --local user.name "xanderstevenson" git add . git commit -m "Updating the repository, with mp3 in the ./mp3s folder" - name: Push changes # push the output folder to your repo uses: ad-m/github-push-action@master with: github_token: ${{ secrets.GITHUB_TOKEN }} force: true   Alright! We’re almost there. Now, we just need to write the Python script we’ve invoked in the workflow and we’ll be good to go!     Python Script to Convert Text-to-Speech   Remember the variables ‘MODIFIED_FILE’ and ‘ADDED_FILE’ we created in the workflow? Let’s import them into our Python script.  if __name__ == "__main__": #import environmental variable from the GitHub actions workflow # get the file that changed or was added and strip the brackets and quotes the_modified_filename = os.getenv('MODIFIED_FILE') the_modified_filename = the_modified_filename.replace('[','').replace(']','').replace('"','') the_added_filename = os.getenv('ADDED_FILE') the_added_filename = the_added_filename.replace('[','').replace(']','').replace('"','')   Now, we’ve got the name of the file that changed. But we still need its basepath as Python sees it. Here’s the method I used to do that: base_path = os.path.dirname()   We have the filename and basepath and now I want to create a directory named after the file that changed. This was how I got that to happen: # get the filename of the document file_name = str(os.path.basename(txt_filepath).rsplit(".", 1)[0]) file_name = file_name.replace(" ", "-") txt_dirname = mp3_base_path + f"/{file_name}/" if not os.path.exists(txt_dirname): os.makedirs(txt_dirname)   Translating a docx to a MP3 was not working, so I found I had to convert the DOCX to a TXT first. I installed docx2txt for that and used it as follows. I also needed to format the resulting txt io remove URLs and empty lines, which could make the MP3 sound annoying when read in text-to-speech: try: # convert docx to txt my_text = docx2txt.process(docx_filepath) # remove all URLs from text my_text = re.sub(r"http\S+", "", my_text) # create and write text to txt file with open(txt_file, "w") as text_file: print(my_text, file=text_file) #remove original docx os.remove(docx_filepath) ### removing empty lines # Read lines as a list workable_txt = open(txt_file, "r") lines = workable_txt.readlines() workable_txt.close() # Weed out blank lines with filter lines = filter(lambda x: not x.isspace(), lines) # Write workable_txt = open(txt_file, "w") workable_txt.write("".join(lines)) workable_txt.close() ###   Finally, we feed the TXT file to the gTTS module I’ve installed and save the MP3 to a new sub-directory we create called ‘mp3s. # open and read .txt file with open(txt_file, 'r') as f: the_text = f.read() # conversion to MP3 mp3 = gTTS(the_text, lang="en", tld=accent) # strip filename from filepath file_name = str(os.path.basename(txt_file).rsplit('.', 1)[0]) # if the 'mp3s' directory does not exist, create it if not os.path.exists(txt_dirname + '/mp3s/'): os.makedirs(txt_dirname + '/mp3s/') time.sleep(3) # name and save mp3 mp3_filename = txt_dirname + '/mp3s/' + file_name mp3_filename += '.mp3' mp3.save(mp3_filename)   At this point in the workflow, the following will be committed and pushed to the repo:   A new folder, named for your .docx file, inside of which will be: A downloadable copy of the original DOCX document A readable TXT version of the document A downloadable MP3, inside of an 'mp3s' sub-folder     Demo   Let’s see this baby in action! I have a DOCX document, named “DevSecOps-Defined”, which contains text, images, links and URLs.  To read the DevSecOps Defined article, just click on the image.   All I need to do is upload and commit it to the appropriate directory in my repo, which in this case is community-content-pipeline/developer-hub/developer-security/security-knowledge-base/   That will start the build process that is detailed in our Workflow. GitHub Actions does the rest.     The entire Build, Test and Report process for CI took GitHub Actions about 30 seconds to complete; not too bad! Here are the results: a new directory named “DevSecOps-Defined” with our DOCX and TXT files inside….     ….as well as the mp3s folder with the audio inside...         Special Thanks, Source Code and Related Projects    Special thanks to John Capobianco of Cisco, whose wiktrola project provided inspiration (and a bit of code ; ) for this project.     For the complete code to my project, visit the following repository: https://github.com/xanderstevenson/community-content-pipeline     People may want control over the accent used for text-to-speech. For that, I have created the following:   https://github.com/xanderstevenson/txt_2_mp3   This will give you the ability to choose from the following accents for the MP3: 1. English (Australia) 2. English (United Kingdom) 3. English (United States) 4. English (Canada) 5. English (India) 6. English (Ireland) 7. English (South Africa)   * An audio version of this article is attached, MP3 format/zipped * Python code was linted with Pylint.   ... View more

(YouTube link - in case embedded video won't play)   The third edition of the 'DevOps Shop' series, 'Linter Wonderland + AI Assistant for Developers', is live now! Linting is a simple, yet effective, way to test and clean up your code during the build phase of continuous integration. If you are currently skipping linting, you could be living in a ‘linter winter’ which could be making your code, at best - inefficient, and at worst - a security liability. In this video, you will see tried, tested and true ‘lint hints’ and how automation can make linting a relatively painless process. You’ll also learn about various Python linters, and how the benefits of linting for security and automation should have you ‘sprinting for linting’. Last, but not least, we will introduce and demonstrate Tabnine, the powerful AI assistant for developers you can use in your IDE, on your laptop, on a server behind your firewall, or in the cloud. Tabnine is powered by machine learning models pre-trained on open-source code with permissive licenses and it features whole and full-function code completion. Privacy is paramount with Tabnine, so you code will remain private...and it offers a FREE plan too! Here is the Cisco Blog about Tabnine, mentioned in the video: https://blogs.cisco.com/tag/tabnine   Previous editions of DevOps Shop   1. Intro to Cisco Modeling Labs 2. 9 Easy-to-Use Python Modules for DevOps     ** An MP3 version of this blog is attached   ... View more

Do you hear about DevOps just as much as we do? Is it a title? A tool? A new cheese? And where do SRE's (Site Reliability Engineers) come into the mix?? ... View more

The second edition of the 'DevOps Shop' series, titled '9 Easy-to-Use Python Modules for DevOps', is live now! Python is arguably the most important programming language for DevOps. It enables automation, can adapt to many use cases, and comes with a vast amount of built-in modules. In this video, I present and demo nine of the best and most user-friendly Python modules (Requests, os, sys, re, Fabric, subprocess, Platform, Beautiful Soup, smptplib), with code examples and resources for further learning and practice. Check it out now!   ... View more

Introduction In this article you will see three API-based workflow automations using the Cisco DNA Center REST APIs. Inventory Collection – Python SDK Device Configuration – Ansible Software Upgrades – Terraform Each of the use cases will be implemented using a different Cisco DNA Center Library. This article will help you to learn how to call the Cisco DNA Center APIs using the Northbound Python SDK, Ansible and Terraform.  The Cisco DNA Center libraries will simplify your software development and allow you to focus on your automation workflows rather than each API call. All these use cases will be using GitHub to pull and push data to repos. What is DNAC?  Cisco Digital Network Architecture offers centralized, intuitive management that makes it fast and easy to design, provision, and apply policies across your network environment. The Cisco DNA Center GUI provides end-to-end network visibility and uses network insights to optimize network performance and deliver the best user and application experience. Cisco DNA Center is a network management solution. You can use Northbound REST APIs to collect, update or delete records that cisco DNA center has about sites or network devices. Northbound REST APIs help to automate lot of workflows and operations associated with the GUI. Cisco DNAC is enabled to receive event notifications like system health or Network issues. Cisco DNA Center platform is used to create value-added applications that leverage the native capabilities of Cisco DNA Center. For more details check –https://developer.cisco.com/docs/dna-center/#!cisco-dna-center-platform-overview/cisco-dna-center-platform-overview. Cisco DNA Center provides two tools to use the APIs. One is the Try It option, which needs no coding. Second is the Code Preview which helps to test your code on the platform. The main task for the network engineer and the DevOps engineer would be to develop some standardized templates for the workflow of DNAC. In the first use case you are going to run an application developed using python SDK which is going to perform inventory collection. Will identify the network devices that are compliant and non-compliant with DNAC. Once the inventory collection is done the output will be pushed to GitHub. Second use case will be on Device configuration using Ansible. Based on the inventory details from GitHub the playbook is going to filter out the compliant devices that need to be configured. Third use case will be on analyzing the non-compliant devices based on the software perspective using Terraform and hence performing the necessary software upgrades. These automation tasks will help to have consistency across Cisco DNA Center clusters. It also helps to have enhanced security and visibility. Git repositories as a single source of truth to deliver Infrastructure-as-Code: Automations and integrations Intent-based configurations documented as code GitOps for Cisco DNA Center helps to monitor the code changes, can do intent configurations, can track the network state and version control. Inventory Collection – Python SDK  Includes all Cisco DNA Center REST APIs Support for Cisco DNA Center version 2.3.3.x First SDK version August 2019 Average 1,500+ downloads per month For more details on refer: https://dnacentersdk.readthedocs.io/en/latest/ https://github.com/cisco-en-programmability/dnacentersdk Python SDK is basically a wrapper for all the cisco DNAC APIs. In this use case of automation workflow python SDK will collect the inventory of the network devices, will identify the device state whether they are reachable or managed, the role of the devices etc. You can save the collected data in any format including JSON, YAML, CSV. Cisco DNA Center-Python SDK has more lines of code on getting the inventory information parsed in the format that is needed for the next two use cases. After saving the data locally it will be pushed into GitHub repo. You can have the “device_inventory.json” file that contains the inventory data collected from the DNAC  like role, device_family, device_id, site_id etc which can be used by the Ansible.Also can have the inventory data collected as “non_compliant_devices. yaml” file to be used for Terraform use case. A walk through on screenshots taken from the demo.  If you navigate the DNAC lab setup you can see 13 devices as shown below. GitHub url - https://github.com/cisco-en-programmability/dnacenter_day_n   On GitHub navigate to the file named inventory_collection_sdk.py under /python_sdk folder. Clone the repo. When you run the script  you can see the files generated under inventory folder. The above  output will be pushed to GitHub repo. If you check the repo can monitor when was the last change in inventory happened. When did the script last run. When was the inventory updated etc. Once you have the inventory data available under GitHub you can use that data for other applications. Let’s take the second use case of Device Configuration using Ansible.  In this case the device inventory as well as the deployment templates are downloaded from GitHub first. From a configuration perspective the devices are checked against the intent. Exactly to what all network devices that are matched only for those devices provisioning will be done. Let’s explore some terms used in Ansible Modules.  YAML - Yet Another Markup Language or YAML Ain’t Markup Language. Ansible uses YAML to define playbook configurations and variable files. It is human readable and may be used with many programming languages. Collection - A packaging format for bundling and distributing Ansible content, including plugins, roles, modules, and more. Module/Plugin - Code, typically written in Python, that will perform some action on a host. Cisco DNA Center modules provide the documentation, the plugins execute the actions Ansible One Slide. Playbook - Repeatable, re-usable, simple configuration management that will push a new configuration or confirm the existing configuration. They are composed of plays and tasks. Play – Execution of a set of tasks to a host or group of hosts – lab Cisco DNA Center vs production Cisco DNA Centers. Task – Execute a module with specific arguments. When a task has been executed on all target machines, Ansible moves on to the next task. Task Lists and Blocks – re-usable groups of tasks that are executed based on specific conditions or counts. Ref: https://docs.ansible.com On this url you can find the required collection of playbooks https://galaxy.ansible.com/cisco/dnac.  Following are its specifications. Complete set of Ansible modules for all Cisco DNA Center REST APIs First library version Dec 2020 Certified by RedHat v 2.9 – 04/2021 Community certified - 05/2022 Support for Cisco DNA Center version 2.3.3.x 7,300 + downloads It’s the intent modules and plugins that makes the calls based on the designed intent and automates the workflow. The steps required to install the collection and run the playbook is mentioned in this url – https://galaxy.ansible.com/cisco/dnac You may  navigate to the repo were the playbooks on DNAC is published. https://github.com/cisco-en-programmability/dnacenter-ansible To know which are all the modules that are needed to call an API you need to check the parameters corresponding to your requirement. Then select the best fit that is mentioned in the GitHub collection as the name suggests. Same is the case with templates. You need to select the API that is used to deploy the template.  Also select the corresponding required module. Template Deployment using Ansible Playbook.  The network engineer creates deployment template and publishes it on GitHub. Please note only approved templates will be deployed into the network infrastructure. Ansible playbook that is downloaded from GitHub will pull the templates as well as the device inventory. It also identifies which are all the devices managed by cisco DNAC and will match the filter on features that needs to be configured. Through cisco DNAC only you will be pushing the configurations into network devices. Note: you are not going to configure the network device directly using ansible. It all happens via DNAC. These steps must be considered to run the Ansible playbook created by the developer and the network engineers. In the below picture the filter mentions an intent. The template on the left will be configured only to devices that are cloud services routers. The template on the right will be pushed to all the devices which has got Access as the role. The benefit is you don’t have to maintain any inventory. The intent will match what is needed based on the devices of network state you mention. You are running an automation tool to identify all the devices that will match the intent which are in compliance. To run the Ansible Playbook all you need is to specify the deployment name as a parameter. You don’t have to even know that whether the deployment template exists on GitHub or not.  While the play book is running if it finds a deployment template will download it and configure as per the stated intent. So, all you need is to just change the deployment template as per your requirement and there is no need to change the playbook. If the devices are matched and reachable by the DNAC then only device configuration will happen. A walk through on screenshots taken from the demo. you have no ansible project created . If you check GitHub you can see some already validated and published templates. You can clone the Ansible playbook from GitHub, which could be edited as needed for automation workflows. From terminal window you may run the play book using the below command. While running the playbook you can view various tasks like updating the template folder, creating a new project, downloading the inventory and filtering  the devices that need to be configured. Finally, you can see the project was created and the template that you intended to deploy was published. In the third use case you are going to see the automation workflow for software upgrades using Terraform.   Terraform is an infrastructure provisioning tool. You are going to use the capabilities of terraform to manage a network infrastructure by calling the Cisco DNA Center REST APIs. Let’s explore some terms used in Terraform: HCL (HashiCorp Configuration Language) - Establishes the syntax Terraform uses for things like arguments, blocks, literal values, and expressions, and writing plans.  Provider – Plugins responsible for understanding API interactions with other platforms and exposing resources based on their APIs. Data Source – Allows Terraform to use (read) information defined outside of Terraform. Example: providers, local-only Resource - Are the most important element in the Terraform language. Each resource block describes one or more infrastructure objects – devices, interfaces, operations. Terraform One Slide Init - The command is used to initialize a working directory containing Terraform configuration files. This is the first command that should be run, and it is safe to run this command multiple times. It will install the required providers and modules. Plan - compares the managed infrastructure state to the configuration, and it determines which changes are necessary. It presents a human-readable summary to the user. Apply – Makes changes to real infrastructure in order to make it match the desired state. It may use saved plans or creates a new plan and asks for approval. Ref: https://www.terraform.io/docs In this use case you are going to use two providers. DNAC which provides the software. GitHub which provides the inventory of devices that are not compliant. Providers will be using either Data Source or Resource. If you want to have information from DNAC regarding a site you will be using the data source. For example, GET API is a data source. But if you are going to perform some actions like software deployment or PnP then those actions will require a Resource. Based on the functionality required you need to map to the correct API. Terraform will show you first what all things will be changed but no changes will be performed over the network devices. If you like those changes using the command terraform apply you can deploy the changes. You can find the Terraform Provider details published on the registry at the below url: https://registry.terraform.io/providers/cisco-en-programmability/dnacenter Note: Not all the DNAC API’s are good for Terraform provisioning. Based on the API functionality you can select among which automation tools will be helpful. You may find the details on GitHub. In this use case your main task is to distribute the software hosted on DNAC to the network devices that are non-compliant based on the inventory data. There are a few steps required to Initialize the Terraform Working Directory. Next step will be to create an execution plan. This will help to see how the changes may look like and what all things will be changed. Next comes the execution plan where terraform will make the necessary API calls, download the latest information, and apply it to the network infrastructure. You can see the screenshot of the details on plan execution which gives the output file on what was changed. Let’s have a walk through on the screenshots taken while running the demo. In Github url - https://github.com/cisco-en-programmability/dnacenter_day_n/tree/main/terraform/modules/distribution you can see the main.tf and output.tf files. Using the command #terraform init you are going to install the providers that helps to make the API calls. By Plan you are going to see the changes and by Apply command the deployment will be done. To verify you can see that you have a new image installed in the flash memory of the non-compliant devices. A short note on GitHub Security There are various kinds of code scanning tools available on GitHub platform. You may perform code scanning and confirm that no mistakes are present from security perspective. This will make sure that when you clone the code no vulnerabilities are injected to your environment. In the case if you are using any open source libraries in the code try to use the tool “Dependencies Alerts” which will help to alert if any vulnerabilities is present on your code or not. credit: cisco live presentation- ClickOps to GitOps - Cisco DNA Center Infrastructure-as-Code Use Cases – DEVNET-2739 Summary Any network automation use case may be implemented with any Cisco DNA Center library Python SDK is very versatile, excellent choice for complex use cases, and integrations Ansible and Terraform are great options for: Simple use cases, or end-to-end automations with Data Center and Cloud When DevOps engineers use these automation tools Partner with developers to create your automations workflows Code re-use - build your team’s library: modules, tasks lists, plans Cisco DNA Center – Developer Resources https://developer.cisco.com/codeexchange/github/repo/wwt/dnac_site_settings_example https://www.youtube.com/watch?v=Lzt-akHiAlc https://developer.cisco.com/dnacenter/ http://cs.co/EN-Programmability-Videos https://github.com/cisco-en-programmability https://dnacentersdk.readthedocs.io/en/latest/ https://galaxy.ansible.com/cisco/dnac https://github.com/cisco-en-programmability/dnacenter-go-sdk  https://registry.terraform.io/providers/cisco-en-programmability/dnacenter   ... View more