Network Automation with Plug and Play (PnP) – Part 8

[aradford]

Introduction

If you have been working with Network Plug and Play (PnP) for a while, you might find some of these tools useful.  I am going to assume you have seen my earlier blogs, so have a good understanding of the PnP process, the API and how they work.  The scripts are purely educational, and you should adapt them for your own environment.

Installation

The tools are published at the following repository https://github.com/aradford123/PnPTools.git

Instructions for installing on a MAC/Linux machine follow:

git clone https://github.com/aradford123/PnPTools.git

I have written each as a standalone application.  I recommend you use a virtual environment to install the requirements. Note: these scripts use the uniq library, which requires python3.

cd PnPTools

virtualenv -p python3 env

source env/bin/activate

Remember, if you exit your shell or logout, you will need to run the command "source env/bin/activate" each time to activate the virtual environment.

#1 PnPWatch

This utility shows you the steps and progress of the PnP process for a device.

To install and run do the following:

cd PnPWatch/

pip install -r requirements.txt

To run the script you need to add your controller and the credentials for it.  You can do this in two ways:

1. edit the apic_config.py file
2. use the APIC, APIC_USER and APIC_PASSWORD environment variables.  For example the shell command "export APIC='sandboxapic.cisco.com'" would set the controller to the DevNet cloud controller. Be very careful with quotes.

You also provide a serial number for the device you are watching.  You need to have a rule for the device, or it needs to be in the unclaimed devices list.

$ ./src/watch_provision.py FDO1732Q00B

Watching unclaimed for serial:FDO1732Q00B

19:49:35: Duration (0) Unclaimed

19:49:43: Duration (9) Waiting for Resource

19:50:02: Duration (28) Start Provisioning

19:51:08: Duration (94) Deploying Device Certificate

19:52:09: Duration (155) Deploying Config

19:54:09: Duration (275) Provisioned

19:54:09: Completed (275): PROVISIONED

each step of the process is shown, along with the number of seconds it took to execute.  For example, it took 155 seconds for the "Deploying Config" step.

#2 PnPSync

This utility keeps your library of configuration, template and images files synced up to APIC-EM. As you are not able to edit directly on the controller, you can edit the disk based version of these files then sync them up to the controller.  The SHA1 hash is checked first to ensure an upload is required.  Any missing files are uploaded (POST), and any existing files are updated (PUT)

You will need to perform the same steps as the first example above.

There is an extra variable which is the directory containing the folders for the configuration, template and image files.  Those folders are called "configs", "templates" and "images".

$ ./src/pnp_file_sync.py

NameSpace:config:valid

Updated File:4451.txt (c0991ef7-2274-40a8-8b21-7a690fc73193)

NameSpace:template:valid

Uploaded File:new-template.txt (967e44ab-bd66-40bb-9645-71d69b7db0a6)

NameSpace:image:valid

Skipping File:c2960x-universalk9-mz.152-4.E.bin (2f4380b4-64f5-401c-9464-ad5925f783a9) SHA1hash:67ff12708d66c188e998e78641d4f2f18e29c539

Skipping File:cat3k_caa-universalk9.SPA.03.06.05.E.152-2.E5.bin (f2b2c594-73de-45cc-9c70-ed0315e5b0ff) SHA1hash:c87f5ea36cd66d4558af1729c9d41f5058da4123

This shows the file "configs/4451.txt" was updated.

File "templates/new-template.txt" was uploaded.

File "images /c2960x-universalk9-mz.152-4.E.bin" was left unchanged as the SHA1 hash was identical.

#3 PnPConfigTemplator

This is an "industrialized" version of the initial example I wrote about in earlier blogs. There are two main differences:

1) It does not append a 4 digit number onto the end of projects and filenames

2) It uses nested jinja2 templates to generate configuration files for switch stacks.  There is a base template, that gets extended through macros and loops.

You will need to do the same steps as #1 to install the requirements and update the variables for the APIC-EM, username and password.

$ ./10_create_and_upload.py ./work_files/hosts.csv

Using device file: ./work_files/hosts.csv

Using template directory: work_files/templates

##########################

Variables: {'HOSTNAME': 'fourswitch', 'imageFile': 'cat3k_caa-universalk9.SPA.03.06.05.E.152-2.E5.bin', 'VoiceVlan': '300', 'ManagementIP': '10.10.10.220', 'stackCount': '4', 'DISTRO': 'distribution2-2-2', 'site': 'Sydney', 'serialNumber': '12345678901', 'USERVLAN': '100', 'platformId': '3850-stack', 'management': '90', 'template': 'four_switch.jnja'}

created file: work_files/configs/fourswitch-config

creating project:Sydney

Configuration File_id: b01989ab-19e7-47ee-88ec-812a90fc479c

Creating Rule [

  {

    "eulaAccepted": true,

    "serialNumber": "12345678901",

    "licenseLevel": "ipbase",

    "platformId": "3850-stack",

    "memberCount": "4",

    "imageId": "f2b2c594-73de-45cc-9c70-ed0315e5b0ff",

    "pkiEnabled": true,

    "configId": "b01989ab-19e7-47ee-88ec-812a90fc479c",

    "hostName": "fourswitch"

  }

]

Rule Status: {"message":"Success creating new site device(rule)","ruleId":"1aff050c-ba3f-456f-976d-e9a3531f544c"}

<SNIPPED>

The script has quite a lot of debugging in it to show you what is happening.

I can see:

• variables used for the template: Variables: {'HOSTNAME'…
• configuration file being created: created file: work_files/configs/fourswitch-config
• - project being created: creating project:Sydney
• - Configuration file ID:Configuration File_id: b01989ab-19e7-47ee-88ec-812a90fc479c
• - The rule being uploaded: Creating Rule […..
• - Rule status: Rule Status: {"

These rules assume that an image file "cat3k_caa-universalk9.SPA.03.06.05.E.152-2.E5.bin"is present on the controller.  You can either edit the file "work_files/hosts.csv" to remove it, upload a dummy file for testing, or a real file for production.

How do the templates work

These examples are using jinja2 templates.    A sample configuration file is below.  All of the base IOS commands are in the file "work_files/templates/base.jnja". There are "macros" defined for the different port types in the file "work_files/templates/macros.jnja".

The base file has a "block" in it called "interfaces" {% block interfaces %}.  The template fills out that block (you could have other blocks as well, if required).

Jinja2 supports for loops {% for stack_num in range(1,3) %} .  In this case the variable "stack_num" will be set to 1 and 2.

"stack_num" is used to define the interface range, and then the macro "user_port" has the configurations for a user port.

{% from "macros.jnja" import user_port, ap_port, uplink_trunk with context%}

{% extends "base.jnja" %}

{% block interfaces %}

{% for stack_num in range(1,3) %}
interface range g{{ stack_num }}/0/1 - 40
{{ user_port() }}

int range g{{ stack_num }}/0/45 - 48
{{ ap_port() }}

int ten{{ stack_num }}/1/4
{{ uplink_trunk() }}

{% endfor %}
{% endblock %}

There are many ways this could be optimized and extended.  This example is just to illustrate the basic concepts.

Cleanup.

Running

./12_clean_up_all.py  work_files/hosts.csv

will remove all of the files/projects/rules.  This is for testing purposes

What Next?

I will keep adding to these utilities as I get time.  They are not officially (or unofficially J) supported, just examples of tools to make life easier.

In the meantime, if you would like to learn more about this, you could come hang out with us in The Cisco Devnet DNA Community. We’ll have a continuous stream of blogs like this and you can ask questions and we’ll get you answers.

In addition, we have a Github repository where you can get examples related to PnP.

Thanks for reading,

@adamradford123

[raziahme]

We are a Plug and Plan Runbook for Cisco internal team, would you allow us to use you blog content?

[aradford]

sure.  All public information.

[DAVID]

Just discovered your blog on APIC-EM and it's a great resource.  My hurdle is how to I use PNP and EEM to configure the router to download the vlan.dat file so that when the APIC-EM pushes out the config to the router that the router will have the necessary vlan.dat file to recognize the vlans

[aradford]

Hi David,

there are two options.  You can use an EEM script to download the vlans.  I have examples of "self destructing" EEM scripts in other blogs.

The other option is to change the VTP mode to transparent, so vlan.dat is not required.

[DAVID]

Thanks for the help.  I'm learning my way through the EEM, and what I have at the end of the configuration file that APIC-EM pushes to the router looks something like this.

event manager applet POST_PNP

event none sync yes

action 1.0 cli command "enable"

action 1.1 cli command "config t"

action 1.2 cli command "aaa authorization exec default group ISE-Servers local"

action 1.3 cli command "aaa authorization network default group ISE-Servers"

action 1.4 cli command "aaa authorization auth-proxy default group ISE-Servers"

action 1.5 cli command "vlan2"

action 1.6 cli command "vlan3"

action 1.7 cli command "vlan4"

action 1.8 cli command "vlan5"

action 1.9 cli command "vlan13"

action 2.0 cli command "vlan100"

action 2.1 cli command "exit"

action 2.2 cli command "no event manager applet POST_PNP"

action 2.3 cli command "end"

end

conf t

event manager applet reload

event none sync yes

action 1.0 cli command "enable"

action 2.0 cli command "wr"

action 4.0 reload

exit

end

event manager run reload

I am testing out PNP in the lab so I am able to watch the console of my test router but test router just sits there there and APIC -EM shows the status as in progress.  Is it better to combine the two applets so the commands are saved to the start-up config and there are no event manager entries in the config after a successful reload?

[aradford]

Hi David,

one question, why are you getting EEM to reload the device?

PnP process does a "wr mem" and then returns a status to complete the process.  If this gets interupted then it will not complete.

As an aside, i normally have my EEM script fire after the PnP process completes.

Adam

[DAVID]

As I am new to EEM and APIC-EM, I don't trust myself so I want be assured that I have a process that is fool-proof and predictable.  If this does indeed work, I will this process to replace over 500 891W routers with little to no touch.  I hope that is...

My goal is to simply use APIC-EM and EEM if necessary to configure PNP so that I can replace aging Cisco 891W routers with new Cisco ISR c1100's.  I am still working my way through why APIC continues to throw up the ERROR_HEALTH_CHECK_TIMER_EXPIRED

Failed health check since device is stuck in non-terminal state PROVISIONING_CONFIG for more than threshold time: 0 hours, 16 minutes, 0 seconds

I am running APIC-EM ver 1.6.1.30163.

What I am doing is that I have configured a test 891W with the required option 43 scope.  This 891W would mimic the same router in the remote location.  I have the int gig0/0/1 connected to the fa0 interface of the 891W so that the C1111 would get the necessary ip addressing from the 891W and communicate with the APIC-EM and it does.  I am able to see the unclaimed router, claim it and add it to the project and cross my fingers.  In short, the new router will simply replace old router and on-site tech will just move cables from old router to new router.  Config is just now on new router.  All ip addressing, ZBFW's ,policies, ACL's  are the same just on new router.

[DAVID]

As I am new to EEM and APIC-EM, I don't trust myself so I want be assured that I have a process that is fool-proof and predictable.  If this does indeed work, I will this process to replace over 500 891W routers with little to no touch.  I hope that is...

My goal is to simply use APIC-EM and EEM if necessary to configure PNP so that I can replace aging Cisco 891W routers with new Cisco ISR c1100's.  I am still working my way through why APIC continues to throw up the ERROR_HEALTH_CHECK_TIMER_EXPIRED

Failed health check since device is stuck in non-terminal state PROVISIONING_CONFIG for more than threshold time: 0 hours, 16 minutes, 0 seconds

I am running APIC-EM ver 1.6.1.30163.

What I am doing is that I have configured a test 891W with the required option 43 scope.  This 891W would mimic the same router in the remote location.  I have the int gig0/0/1 connected to the fa0 interface of the 891W so that the C1111 would get the necessary ip addressing from the 891W and communicate with the APIC-EM and it does.  I am able to see the unclaimed router, claim it and add it to the project and cross my fingers.  In short, the new router will simply replace old router and on-site tech will just move cables from old router to new router.  Config is just now on new router.  All ip addressing, ZBFW's ,policies are the same just on new router.

[DAVID]

I believe I have tracked my APIC PNP failures to the actual configuration file.  While the configuration file is nothing more than a text file created in Notepad and this file is easily able to be copied via USB to a Cisco router with no errors when APIC-EM attempts to parse the configuration file APIC continues to throw up errors and yet the router does indeed have the configuration is now on the router while the APIC-EM shows errors for the device and the device still remains in the unprovisioned  state rather than provisioned state.

Why this is happening and what I can do to correct it?

[aradford]

This error normally means something in the configuration is preventing the router from communicating with APIC-EM.

When the config is applied to the router, the PnP agent needs to sync back with APIC-EM to let it know the config was successful and the configuration was saved successfully.

The best way to test this is to log in to the console of the router and see if you can ping APIC-EM.  You might also need to make sure a HTTPS session can be established from the router to APIC-EM (if you are using a FW).

"debug pnp all" on the router should also tell you if there are errors.

The simplest way to confirm the configuration file is to copy/paste it into a clean router and make sure the syntax is ok.  This should get picked up by the pnp process too.

In short:

1) verify the configuration file is ok

2) if you have a router in "HEALTH CHECK EXPIRED" make sure it can connect to APIC-EM

Just as an aside, are you still doing a reboot in your EEM script?  This will likely break PnP as PnP wants to save the config then tell APIC-EM that the config was saved successfully.

Adam

[DAVID]

Are there any gothca's in any configuration that APIC-EM PNP does not like such as the "AAA Authorization commands that I will need to configure EEM to run?

The only reason right now that I am even trying to reboot is just the added assurance that the process actually does work and I do have a working configuration on the new router.

[DAVID]

I went through the event history of the device from the last attempt and noticed these errors:

Received response from pnp agent for message correlatorId: CiscoPnP-1.0-20-163-7F6E918738-18 but with error code : ZTD_CMD_ERROR Response String: ERROR:PnP Service Error 1413:Invalid input detected service log:

% Invalid input detected at '^' marker. ip cef ^ % Invalid input detected at '^' marker.

no ipv6 cef ^ % Invalid input detected at '^' marker. epm access-control open ^ % Invalid input detected at '^' marker.

class-map type inspect match-any wan-to-pos ^ % Invalid input detected at '^' marker.

match access-group name wan-to-pos ^ % Invalid input detected at '^' marker. class-map type inspect match-any pos-to-wan ^ % Invalid input detected at '^' marker. match access-group name pos-to-wan

and so on.

What is the invalid input detector?  I can copy the same configuration from a USB to the Cisco c1111 router with no errors and the router loads the config and runs fine.

[aradford]

That is strange.

I can see how it causes an issue if the syntax is incorrect.

However, you should see the same issue if you paste the config from the console too.

The only thing I can think of it is something to do with the order of the commands.  e.g. if the first attempt via PnP failed, when you copy the commands from USB, the command order issue is not exposed as due to the pnp process.

Have you tried to paste the config up to just before the "ip cef" command and then the "ip cef" command, onto a router that has been "wr erase" and reset to factor defaults.

[aradford]

There is a got-cha with the aaa authorisation.  I have documented it in my blog.

Network Automation with Plug and Play (PnP) – Part 7

The issue is the userid needs to be set for the eem commands.  However, in later versions of IOS, this work around should not be required.  You can provide a username and password for the device in the PnP rule, and you do not need the EEM scripts.

If you are running older IOS, then you will need the EEM script and the workaround.  I looked back and it seems you are running 1100 routers, that will run recent IOS.  You should be able to use the username/password in the rule.

The EEM script will remove itself after it has run.

1. The EEM script needs to also be authorized.  In this example, I have provided and user "sdn2" and a privilege level "15".  This user is not a local user, it is only defined on my tacacs server as that is where authorization is occurring.
   • event manager session cli username sdn2 privilege 15
2. Move the "aaa" commands inside the EEM script that runs 30seconds after being created.
   • action 2.1 cli command "aaa authorization commands 1 default group ISE-T if-authenticated"
   • action 2.2 cli command "aaa authorization commands 15 default group ISE-T if-authenticated"

[DAVID]

When I would copy a text file from USB it is simply copy usb0: filename bootflash:filename followed by copy bootflash:filename startup config and then a reload and I am done.  Currently this way is repeatable, and repeatable every time. I don't get any errors to track down.  No endless Googling.  No opening TAC cases to troubleshoot.  I could have done by hand more routers this way than I could done just one router with APIC at this rate.

I think I have reset a router back to factory default at least twenty times only to fail somewhere during the provisioning process.