Showing results for 
Search instead for 
Did you mean: 

Network Automation with Plug and Play (PnP) – Part 6

Cisco Employee

Continuing the story-- Stacking

This blog covers another advanced PnP use case: Stacking.   Stacking can be used with any of the earlier examples, here it is with an earlier advanced use case (from blog 5): Etherchannel + trunk + Non Vlan 1 for management (NV1) + Non vlan1 native vlan.

For all of these examples I am using a 3650 switch running 3.6.5 code, but you could use versions of 16.3.1 and 3.7.4 (For other platforms such 2960x please see release notes for details. Not all 2960 support stacking).

Make sure you do not hit any keys on the console while the switch is booting, as this can interrupt the PnP process.

Setup DHCP server

The first thing needed is a mechanism for the switch to discover the controller.  In our examples we are going to use DHCP, but you could also use DNS etc. as covered in earlier blogs.  Here is a sample configuration for an IOS switch.  The controller IP address is  Note also the use of the "5A1D" in the option 43 string.  The "D" displays debug messages for PnP on the console of the PnP switch.

ip dhcp pool ZTD-switches



option 43 ascii "5A1D;B2;K4;I10.10.10.140;J80"


Stacking + EtherChannel + NV1 for management + NV1 Native

The switches are going to be connected by two links bound together in an ether-channel.

stacking diagram.png

First a bit about stacking.

Stacking is a way of combining switches together in a unified system using a special "stackwise" port at the back of the device.  In this example I am using 3650 switches.  Each switch is referred to as a "member".  There are three types of members:

  • Active/Master: This switch "runs" the stack. It holds the running configuration. There is only ever one active switch. It is elected and can be influenced using priority.  By default lowest MAC address is master.
  • Standby: backup master switch.  Again, there is only ever one backup switch.  It takes over if the master fails.
  • Member: not a master or a standby.

The default election process is to use the lowest Mac address.

You can have up to 9 switches in a stack.  See the stacking configuration guide for more information about stacking.

Switch Configurations

The upstream switch needs to have an ether channel configured.  To avoid issues when the PnP switch first comes up, the "no port-channel standalone-disable" command is required.  If this is left out the channel will be disabled as it has not been configured on the PnP switch at boot up.

In addition the native VLAN has been set to 999.

The "pnp startup-vlan 14" command is required to create a new management VLAN on the PnP switch. By default VLAN 1 would be used.

pnp startup-vlan 14

interface Port-channel1

switchport trunk native vlan 999

switchport mode dynamic desirable

no port-channel standalone-disable

interface GigabitEthernet1/0/5

description PNP switch 3650->g1/0/1

switchport mode dynamic desirable

switchport trunk native vlan 999

channel-protocol lacp

channel-group 1 mode passive

interface GigabitEthernet1/0/6

description 2nd link to 3650 etherchannel test

switchport trunk native vlan 999

switchport mode dynamic desirable

channel-protocol lacp

channel-group 1 mode passive

The configuration of the PnP switch includes the EtherChannel.  It also has native VLAN of 999.

We also need to create VLAN 999 on the switch.  In this case, due to the version of IOS involved (3.6.5), it cannot be created from PnP configuration.  We are using a "trick" self-destructing EEM script.  The script creates VLAN 999, and then removes itself.  NOTE: If using 16.3.1 code, this would not be required.

hostname 3650-dhcp

enable password xxx


username xxx password 0 xxx


ip http server

ip http secure-server

snmp-server community xxx RO

interface Port-channel1

switchport mode dynamic desirable

switchport trunk native vlan 999

no port-channel standalone-disable


int range g1/0/1,g1/0/3

switchport mode dynamic desirable

switchport trunk native vlan 999

channel-protocol lacp

channel-group 1 mode active



line con 0

line vty 0 4

login local

transport input ssh telnet

line vty 5 15

login local

transport input ssh telnet


!need this to create vlan 999 on IOS versions that do not do it automatically – self destructing EEM script

event manager applet POST_PNP

event timer countdown time 30

action 1.0 cli command "enable"

action 1.1 cli command "config t"

action 1.2 cli command "vlan 999"

action 2.0 cli command "exit"

action 2.1 cli command "no event manager applet POST_PNP"

action 2.2 cli command "end"

action 2.3 cli command "exit


As the switch boots up, the console shows two switches listed, which indicates a stack will be formed. In this case switch #1 has the serial number FDO1732Q00B and switch #2 has serial number FDO1735Q0G5

  Switch #1 has a lower mac address than #2, so is elected master.

0K bytes of  at webui:.

Base Ethernet MAC Address          : 6c:41:6a:0d:e7:80

Motherboard Assembly Number        : 73-15776-03

Motherboard Serial Number          : FDO1731142C

Model Revision Number              : A0

Motherboard Revision Number        : A0

Model Number                       : WS-C3650-48PQ

System Serial Number               : FDO1732Q00B

Switch 02


Switch uptime                      : 6 minutes

Base Ethernet MAC Address          : c0:67:af:ed:bd:80

Motherboard Assembly Number        : 73-15125-05

Motherboard Serial Number          : FDO17340LEC

Model Revision Number              : A0

Motherboard Revision Number        : A0

Model Number                       : WS-C3650-48PD

System Serial Number               : FDO1735Q0G5

If the switch contacts the controller without a PnP rule being present, the master serial number would appear in unclaimed devices.

The PnP process will recognise either serial number in a rule.   To prove this, I am going to create a PnP rule that matches the backup switch (#2) FDO1735Q0G5.

I also need to indicate the number of stack members, the licence for the stack (all members need to have the same license) and accept the EULA (licence agreement).

create rule.png

Here is the rule. Notice that it is using the serial number of what will be the backup switch.  I could have also included an image to upgrade the stack.  In that case, the new image would be downloaded, the stack rebooted, then the configuration downloaded.


At this point, the PnP process is going to proceed identically to the single switch scenario.

"pnp startup-vlan 14" will use CDP to create VLAN 14 on the stack, get it to use DHCP to get an IP Address, and put all active interfaces in VLAN 14.

The DHCP option 43 attribute will indicate the controller to connect to.  A rule has been defined in advance, so the PnP process will kick off automatically.

The configuration will be downloaded, and the EEM script will create VLAN 999, before removing itself.

The running configuration, certificates and licences will also be synced to the standby.

I wrote a little script to monitor the steps (from the PnP status API) and record the times taken (in seconds).  It takes 11mins to reboot the stack (706 seconds) and another 7 minutes to deploy the device (including a certificate).

$ python ./ FDO1735Q0G5

Watching unclaimed for serial:FDO1735Q0G5

16:39:33: Duration (0) Pending

16:51:19: Duration (706) Getting Device Info

16:51:27: Duration (714) Establishing Secure Channel

16:51:41: Duration (728) Getting Device Info

16:51:54: Duration (741) Clearing Certificate Bundle

16:51:59: Duration (746) Getting Device Info

16:54:45: Duration (913) Waiting for Resource

16:55:48: Duration (975) Deploying Device Certificate

16:55:52: Duration (980) Deploying Config

16:58:57: Duration (1164) Provisioned

  16:58:59: Completed (1164):

Taking a look at the PnP application on the controller, it shows the stack has been provisioned correctly.


Clicking on the stacking link (in red box), the controller knows both switches that are part of the stack. If I had a larger stack, all members would be shown.

switch -details - provisioned.png

As in the single switch mode, you can see the status of the ether-channel.  Both ports are active and a part of the ether-channel

3650-dhcp# show int port-channel 1 etherchannel

Port-channel1   (Primary aggregator)

Age of the Port-channel   = 0d:00h:04m:53s

Logical slot/port   = 12/1          Number of ports = 2

HotStandBy port = null

Port state          = Port-channel Ag-Inuse

Protocol            = LACP

Port security       = Disabled

Standalone          = Enabled (independent mode)

Ports in the Port-channel:

Index Load   Port     EC state        No of bits


  0     00 Gi1/0/1  Active             0

  0 00     Gi1/0/3  Active             0

Time since last port bundled:    0d:00h:04m:49s    Gi1/0/3

This also shows VLAN999 is now the native VLAN

3650-dhcp#show int port-channel 1 trunk

Port Mode Encapsulation  Status        Native vlan

Po1 desirable        802.1q         trunking      999

Port Vlans allowed on trunk

Po1 1-4094

Port Vlans allowed and active in management domain

Po1 1,14,999

Port Vlans in spanning tree forwarding state and not pruned

  Po1         1,14,999

Practical Tips

Member Number

The stack interfaces are based on the member number.  For example member number 1, will have interfaces gig1/0/1-48 and member #2 will be gig2/0/1-48  etc. If the switches are physically stacked in a different order to the logical member numbers, this can be confusing. The member number can be overwritten, but a magic EEM script is required to do this.  You can delay the boot time of the switches by 20sec to get the to join the stack in the order you want (top to bottom).

Clean Up Tips

If you are testing in a lab, you are likely to be reusing switches for your testing.  A switch is only ever "out of the box" on the first occasion, so you will need to reset it back to defaults.  Most people are familiar with the "wr er; reload" sequence that erases the configuration and reloads the device.

There are also a couple of other things needed to remove other state information stored outside the configuration file.  This is not usually required, but do this if you are having issues with testing.

# remove the certificates on active and standby

delete /force nvram:*.cer

delete /force stby-nvram:*.cer

# remove vlan data based from active and standby

delete /force flash-1:vlan.dat

delete /force flash-2:vlan.dat

#  remove certificates from memory.  NOTE: you will not be able to SSH after this

conf t

crypto key zeroize



Need spare flash for upgrading software.

It is important to check there is enough spare flash if upgrading.   Make sure you check both the active and standby.  You can reference the master as "flash-1:" and the standby as "flash-2:".  Here is an example of a failed upgrade (and full flash) on the standby.

Switch#dir flash-2:

Directory of flash-2:/

15490 -rw-        1236   Jul 2 2016 12:08:52 +00:00  packages.conf.00-

15497 -rw-         676  Oct 15 2016 04:57:08 +00:00  vlan.dat

15493 -rw-        1236   Jul 2 2016 04:39:03 +00:00  packages.conf.01-

15495 -rw-        1244   Jul 2 2016 04:19:23 +00:00  packages.conf.02-

15492 -rw-        8042  Oct 14 2016 07:35:49 +00:00  pnp-reset-config.cfg

30979 -rw-    83297376   Jul 2 2016 12:08:47 +00:00 cat3k_caa-base.SPA.03.06.05E.pkg

30980 -rw-     4012732   Jul 2 2016 12:08:47 +00:00 cat3k_caa-drivers.SPA.03.06.05E.pkg

15502 -rw-       10838  Jun 25 2016 09:10:07 +00:00  pre-qos-initial

15496 -rw-        1236  Oct 14 2016 07:49:50 +00:00  packages.conf

15506 -rw-     2097152  Oct 15 2016 05:12:22 +00:00 nvram_config

30984 -rw-   111260480   Jul 2 2016 12:08:48 +00:00 cat3k_caa-wcm.SPA.

46468 drwx        4096  Nov 27 2015 07:36:17 +00:00  onep

69697 drwx        4096  Oct 11 2015 07:00:57 +00:00  vman_fdb

30981 -rw-    33786860   Jul 2 2016 12:08:47 +00:00 cat3k_caa-infra.SPA.03.06.05E.pkg

30982 -rw-    43048256   Jul 2 2016 12:08:47 +00:00 cat3k_caa-iosd-universalk9.SPA.152-2.E5.pkg

30983 -rw-    28343180   Jul 2 2016 12:08:47 +00:00 cat3k_caa-platform.SPA.03.06.05E.pkg

15491 -rw-   303753780   Sep 8 2016 19:13:33 +00:00 cat3k_caa-universalk9.SPA.03.06.05.E.152-2.E5.bin

15494 -rw-   328157104  Oct 14 2016 07:47:28 +00:00 cat3k_caa-universalk9.SPA.03.07.04.E.152-3.E4.bin

46467 drwx        4096  Oct 11 2015 07:00:57 +00:00  virtual-instance

77441 drwx        4096  Oct 15 2016 04:56:02 +00:00  dc_profile_dir

77443 drwx        4096  Oct 11 2015 07:02:23 +00:00  wnweb_store

30978 -rw-    81992100  Oct 14 2016 07:49:47 +00:00 cat3k_caa-base.SPA.03.07.04E.pkg

15498 -rw-    82665136   Jul 2 2016 04:19:18 +00:00 cat3k_caa-base.SPA.03.06.04.E.pkg

30985 -rw-     5667516  Oct 14 2016 07:49:47 +00:00 cat3k_caa-drivers.SPA.03.07.04E.pkg

15499 -rw-     4913852   Jul 2 2016 04:19:18 +00:00 cat3k_caa-drivers.SPA.03.06.04.E.pkg

15501 -rw-    33784816   Jul 2 2016 04:19:18 +00:00 cat3k_caa-infra.SPA.03.06.04.E.pkg

15503 -rw-    43021636   Jul 2 2016 04:19:18 +00:00 cat3k_caa-iosd-universalk9.SPA.152-2.E4.pkg

15504 -rw-    27417488   Jul 2 2016 04:19:18 +00:00 cat3k_caa-platform.SPA.03.06.04.E.pkg

15505 -rw-   111180608   Jul 2 2016 04:19:19 +00:00 cat3k_caa-wcm.SPA.

30986 -rw-    36656136  Oct 14 2016 07:49:47 +00:00 cat3k_caa-infra.SPA.03.07.04E.pkg

30987 -rw-    46349632  Oct 14 2016 07:49:47 +00:00 cat3k_caa-iosd-universalk9.SPA.152-3.E4.pkg

30988 -rw-    28904332  Oct 14 2016 07:49:47 +00:00 cat3k_caa-platform.SPA.03.07.04E.pkg

30989 -rw-   128582464  Oct 14 2016 07:49:48 +00:00 cat3k_caa-wcm.SPA.

15500 -rw-    48214016  Oct 15 2016 05:12:05 +00:00 cat3k_caa-universalk9.SPA.03.07.02.E.152-3.E2.bin

1622147072 bytes total (0 bytes free)

This can be resolved with the "software clean" command.

What Next?

This blog covered an advanced deployment models for network plug and play.  Other blogs in the series have covered simpler deployment models and the API and how to automate the creation, upload of configuration files as well as the automation of rules.  In future I will cover some of new enhancements coming in the 1.3 release, including configuration templates, native in APIC-EM.

In the meantime, if you would like to learn more about this, you could come hang out with us in The Cisco Devnet DNA Community. We’ll have a continuous stream of blogs like this and you can ask questions and we’ll get you answers.  In addition, we have a Github repository where you can get examples related to PnP.

Thanks for reading,



In this post, you mentioned a section about how stack member numbers are written to the stack and that it is not related to the physical order of stacking in some cases. I am having issues with this as I am trying to get these pnp deployments as zero touch as possible. For the moment, I have been hard coding 1/1/1 and 1/1/2 as my uplinks back to our distributions, but because the member numbers are different sometimes, I end up either having to come back and move uplinks or swap member numbers. Is there a reliable way, out of the box, to have the switches come up in the order that they are physically stacked or at least a way to know which switch will by default become member 1? In my experience, it appears as the lowest serial number will become member 1 in the stack while the lowest mac still dictates the active member in the stack. Thoughts?

Cisco Employee

Hi Zak,

the only way to "stick the order" right now is to power on the stack top to bottom.  Wait 10-20sec between powering on stack members from top to bottom.

serial number is used to elect a master.

We are looking at some enhancements to this to make it more deterministic.


David Hild

If I may throw in my two cents, the method Adam describes is the only way I have found to ensure the switches come up in a "controlled" way, to getting the switch numbers assigned the way I want them. Of course this experience was pre-PnP. To Zak's point, it certainly would be nice to have an automated way to establish the switch IDs from the APIC-EM prior to pushing the config to the stack instead of having to manually intervene. We will anxiously await the enhancements.


I agree with David and I am also anxiously waiting for January's patch just to see what has been fixed and altered to hopefully make my pnp experience much smoother


Is the Network PnP application available on PC Windows ?



If you are talking about APIC-EM, it is linux-based appliance. Configuration is done over a web interface or API.

What Adam is showcasing here is the PnP functionality of APIC-EM.

best regards,

-- Jose

Cisco Employee

Thanks Jose


Thanks Jose.

I tested the Cisco PnP mobile app in iPhone for branch deployment and I would like to know if it is available on PC Windows ?


Cisco Employee

Hi Seb,

let me check on the plans for windows for the app.


Cisco Employee

Hi Seb,

we have not seen a lot of interest in Windows mobile app (so far) compared to IPhone/Android.  You could "make a wish" on the APIC to get the Product Managers interest.



Hi Adam,

OK, Thanks.

Joakim Backlund

Anyone seen a solution on this issue predefining switch stack order? Booting them in sequence doesnt work reliable from my experiences.

If there ain’t an official solution does anyone have a working eem script to resolve this they can share?

Cisco Employee

2min wait between switch boot should work as that is the timeout to form a stack.  Painful, but it does work.

IOS 16.x supports the infrastructure to renumber switches, but that is not exposed in the App yet.

Joakim Backlund

What will happen with large stacks where the PnP process have already started the deployment of the configuration of the stack before all stack members are up and running? Won't we end up with members of the stack with correct switch numbers but with no interface configuration?

We are running stacks up to 8 members in total. That would mean that the last member will be up and running approx 14 minutes after the first switch in the stack.

I guess PnP will choke the same way the CLI does when I copy and paste myself. Or?

Cisco Employee

No.  You can add stack members to the config and have the interfaces available.


switch 1 provision ws-c3650-48pq

switch 2 provision ws-c3650-48pd

switch 3 provision ws-c3650-48pd

and you will be able to download the configuration for the interfaces.

You can just provision a single switch #1 with all of the config for the stack in this way.

I know it is not ideal and we are fixing it.

Content for Community-Ad