cancel
Showing results for 
Search instead for 
Did you mean: 
cancel

Network Automation with Plug and Play (PnP) – Part 4

6896
Views
35
Helpful
9
Comments
Cisco Employee

Continuing the story

My last few blogs showed examples of using APIC-EM REST API to upload configuration files and create rules for PnP devices.

I am often asked about the different deployment models for switches.  There are a few different concepts to take into account, VLANs, management VLANs, trunks, EtherChannels etc.

If you have a very simple network with VLAN 1 for management and are not using any of the features above, PnP just works, so no need to keep reading.

This blog post demystifies the different deployment models for edge switches.  We will cover three basic deployment models:

  • "Flat" with non-VLAN 1 (NV1) for management
  • Trunked with NV1 for management  and a static IP address
  • Ether Channel with NV1 for management

For all of these examples I am using a 3650 switch running 16.3.1 code, but you could use versions of 3.6.5 and 3.7.4 (For other platforms such 2960x please see release notes for details).

Make sure you do not hit any keys on the console while the switch is booting, as this can interrupt the PnP process.

The first thing we need is a mechanism for the switch to discover the controller.  In our examples we are going to use DHCP, but you could also use DNS etc as covered in earlier blogs.  Here is a sample configuration for an IOS switch.  The controller IP address is 10.10.10.140.  Note also the use of the "5A1D"  in the option 43 string.  The "D" displays debug messages for PnP on the console of the PnP switch.

Setup DHCP server

ip dhcp pool ZTD-switches

network 10.10.14.0 255.255.255.0

default-router 10.10.14.1

option 43 ascii "5A1D;B2;K4;I10.10.10.140;J80"

  remember

Lets take a look at the first of the three scenarios.

1. Flat deployment – NV1 for management

The two switches are in a "flat" configuration.  Only one VLAN is defined on the PnP switch and the management interface is in that VLAN.

#1.png

This is the configuration on the upstream switch. The "pnp startup-vlan 14" command is required to create a new management VLAN on the PnP switch. By default VLAN 1 would be used.

pnp startup-vlan 14

interface GigabitEthernet1/0/5

description PNP switch 3650->g1/0/1

  switchport access vlan 14

The configuration for the PnP switch is very simple.

hostname 3650-dhcp

enable password xxxx

!

username xxx password 0 xxxx

!

ip http server

ip http secure-server

snmp-server community xxx RO

!

!

!

!

line con 0

line vty 0 4

login local

transport input ssh telnet

line vty 5 15

login local

transport input ssh telnet

!

end

The debug logs show the new VLAN (14) being configured.  This happens via a CDP negotiation between the upstream switch and the PnP switch.

Cisco IOS Software [Denali], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.3.1, RELEASE SOFTWARE (fc3)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2016 by Cisco Systems, Inc.

Compiled Tue 02-Aug-16 17:33 by mcpre

*Oct 6 01:24:19.193: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up

*Oct 6 01:24:20.074: %SYS-6-BOOTTIME: Time taken to reboot after reload =  332 seconds

*Oct 6 01:24:20.193: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up

*Oct 6 01:24:21.258: %LINK-5-CHANGED: Interface Vlan1, changed state to administratively down

*Oct 6 01:24:28.299: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on GigabitEthernet1/0/1 (1), with 3850-core GigabitEthernet1/0/5 (14).

*Oct 6 01:24:29.204: %SYS-5-CONFIG_I: Configured from console by tty100

*Oct  6 01:24:29.666: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan14, changed state to down

*Oct  6 01:24:52.796: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan14, changed state to up

*Oct 6 01:24:58.352: %PNPA-DHCP Op-43 Msg: Process state = READY

*Oct 6 01:24:58.352: %PNPA-DHCP Op-43 Msg: OK to process message

*Oct 6 01:24:58.353: XML-UPDOWN: PNPA_DHCP_OP43 XML Interface(102) UP. PID=359

*Oct 6 01:24:58.354: %PNPA-DHCP Op-43 Msg: _pdoon.1.ntf.don=359

*Oct 6 01:24:58.354: %PNPA-DHCP Op-43 Msg: _pdoop.1.org=[A1D;B2;K4;I10.10.10.140;J80]

*Oct 6 01:24:58.354: %PNPA-DHCP Op-43 Msg: _pdgfa.1.inp=[B2;K4;I10.10.10.140;J80]

*Oct 6 01:24:58.354: %PNPA-DHCP Op-43 Msg: _pdgfa.1.B2.s12=[ ipv4 ]

*Oct 6 01:24:58.355: %PNPA-DHCP Op-43 Msg: _pdgfa.1.K4.htp=[ transport http ]

*Oct 6 01:24:58.355: %PNPA-DHCP Op-43 Msg: _pdgfa.1.Ix.srv.ip.rm=[ 10.10.10.140 ]

*Oct 6 01:24:58.390: %PNPA-DHCP Op-43 Msg: _pdgfa.1.Jx.srv.rt.rm=[ port 80 ]

*Oct 6 01:24:58.390: %PNPA-DHCP Op-43 Msg: _pdoop.1.ztp=[pnp-zero-touch] host=[] ipad=[10.10.10.140] port=80

*Oct 6 01:24:58.390: %PNPA-DHCP Op-43 Msg: _pors.done=1

*Oct 6 01:24:58.390: %PNPA-DHCP Op-43 Msg: _pdokp.1.kil=[PNPA_DHCP_OP43] pid=359 idn=[Vlan14]

*Oct 6 01:24:58.390: XML-UPDOWN: Vlan14 XML Interface(102) SHUTDOWN(101). PID=359

*Oct 6 01:24:59.298: %PNPA-DHCP Op-43 Msg: Op43 has 5A. It is for PnP

*Oct 6 01:24:59.298: %PNPA-DHCP Op-43 Msg: After stripping extra characters in front of 5A, if any: 5A1D;B2;K4;I10.10.10.140;J80 op43_len: 28

*Oct  6 01:24:59.298: %PNPA-DHCP Op-43 Msg: _pdoon.2.ina=[Vlan14]

*Oct 6 01:24:59.298: %PNPA-DHCP Op-43 Msg: _papdo.2.cot=[5A1D;B2;K4;I10.10.10.140;J80] lot=[5A1D;B2;K4;I10.10.10.140;J80]

*Oct 6 01:24:59.298: %PNPA-DHCP Op-43 Msg: Process state = READY

*Oct 6 01:24:59.298: %PNPA-DHCP Op-43 Msg: OK to process message

*Oct 6 01:24:59.299: XML-UPDOWN: PNPA_DHCP_OP43 XML Interface(102) UP. PID=359

*Oct 6 01:24:59.299: %PNPA-DHCP Op-43 Msg: _pdoon.2.ntf.don=359

*Oct 6 01:24:59.301: %PNPA-DHCP Op-43 Msg: _pdoop.2.org=[A1D;B2;K4;I10.10.10.140;J80]

*Oct 6 01:24:59.301: %PNPA-DHCP Op-43 Msg: _pdgfa.2.inp=[B2;K4;I10.10.10.140;J80]

*Oct 6 01:24:59.301: %PNPA-DHCP Op-43 Msg: _pdgfa.2.B2.s12=[ ipv4 ]

*Oct 6 01:24:59.301: %PNPA-DHCP Op-43 Msg: _pdgfa.2.K4.htp=[ transport http ]

*Oct 6 01:24:59.301: %PNPA-DHCP Op-43 Msg: _pdgfa.2.Ix.srv.ip.rm=[ 10.10.10.140 ]

*Oct 6 01:24:59.301: %PNPA-DHCP Op-43 Msg: _pdgfa.2.Jx.srv.rt.rm=[ port 80 ]

*Oct 6 01:24:59.302: %PNPA-DHCP Op-43 Msg: _pdoop.2.ztp=[pnp-zero-touch] host=[] ipad=[10.10.10.140] port=80

*Oct 6 01:24:59.302: %PNPA-DHCP Op-43 Msg: _pors.done=1

*Oct 6 01:24:59.302: %PNPA-DHCP Op-43 Msg: _pdokp.2.kil=[PNPA_DHCP_OP43] pid=359 idn=[Vlan14]

*Oct  6 01:24:59.302: XML-UPDOWN: Vlan14 XML Interface(102) SHUTDOWN(101). PID=359

*Oct  6 01:24:59.411: %DHCP-6-ADDRESS_ASSIGN: Interface Vlan14 assigned DHCP address 10.10.14.3, mask 255.255.255.0, hostname

% Generating 2048 bit RSA keys, keys will be non-exportable... got vend id vend spec. info ret: succeed

*Oct 6 01:25:13.341: %PNP-6-HTTP_CONNECTING: PnP Discovery trying to connect to PnP server http://10.10.10.140:80/pnp/HELLO

*Oct 6 01:25:13.351: %PNP-6-HTTP_CONNECTED: PnP Discovery connected to PnP server http://10.10.10.140:80/pnp/HELLO

[OK] (elapsed time was 9 seconds)

Before the configuration is applied to the switch via PnP, you can see that the CDP "pnp startup-vlan" command has completed.  It has moved the active port into VLAN 14 and created VLAN 14 on the switch and enabled DHCP.

Switch#show run int g1/0/1

Building configuration...

Current configuration : 100 bytes

!

interface GigabitEthernet1/0/1

switchport access vlan 14

macro description CISCO_SMI_EVENT

end

This shows the creation of VLAN 14, and the shutdown of VLAN1.

show ip int br

Interface              IP-Address      OK? Method Status                Protocol

Vlan1                  unassigned      YES unset administratively down down   

Vlan14                 10.10.14.3      YES DHCP up                    up

Once the configuration is complete, the uplink connection is in access mode using VLAN 14.  The only real change downloaded in the configuration was the switch hostname "3650-dhcp".

3650-dhcp#show int g1/0/1 switchport

Name: Gi1/0/1

Switchport: Enabled

Administrative Mode: dynamic auto

Operational Mode: static access

Administrative Trunking Encapsulation: dot1q

Operational Trunking Encapsulation: native

Negotiation of Trunking: On

Access Mode VLAN: 14 (VLAN0014)

2. Trunk – NV1 for management – static IP address

The upstream switch and PnP switch are going to be connected by a trunk port.

#2.png

In this scenario, the upstream switch has a trunk mode desirable, and no VLANs defined.

interface GigabitEthernet1/0/5

description PNP switch 3650->g1/0/1

switchport mode dynamic desirable

In this example, the DHCP address is going to be overwritten by a permanent static management IP address.  Note: when you do this, you also need to provide a default route (ip route 0.0.0.0 0.0.0.0 10.10.14.1), otherwise the PnP device will not be able to contact the controller after the configuration has been downloaded.

hostname 3650-dhcp

enable password xxxx

!

username xxx password 0 xxxx

!

ip http server

ip http secure-server

snmp-server community xxx RO

!

vlan 2222

vlan 2223

int vlan 14

ip address 10.10.14.100 255.255.255.0

ip route 0.0.0.0 0.0.0.0 10.10.14.1

!

!

line con 0

line vty 0 4

login local

transport input ssh telnet

line vty 5 15

login local

transport input ssh telnet

!

end

The DHCP address has been overwritten by the static address in the configuration file.

3650-dhcp#show run int vlan14

Building configuration...

Current configuration : 63 bytes

!

interface Vlan14

ip address 10.10.14.100 255.255.255.0

end

The uplink interface is in trunk mode and has both the management VLANs as well as the locally defined VLANs on it.

3650-dhcp#show interfaces g1/0/1  trunk

Port Mode Encapsulation  Status        Native vlan

Gi1/0/1 auto             802.1q         trunking 1

Port Vlans allowed on trunk

Gi1/0/1 1-4094

Port Vlans allowed and active in management domain

Gi1/0/1 1,14,2222-2223

Port        Vlans in spanning tree forwarding state and not pruned

Gi1/0/1     1,14,2222-2223

Protected: false

Unknown unicast blocked: disabled

Unknown multicast blocked: disabled

  Appliance trust: none

3. EtherChannel + NV1 for management

  The switches are going to be connected by two links bound together in an ether-channel.

#3.png

The upstream switch needs to have an ether channel configured.  To avoid issues when the PnP switch first comes up, the "no port-channel standalone-disable" command is required.  If this is left out the channel will be disabled as it has not been configured on the PnP switch at boot up.

interface Port-channel1

switchport mode dynamic desirable

no port-channel standalone-disable

interface GigabitEthernet1/0/5

description PNP switch 3650->g1/0/1

switchport mode dynamic desirable

channel-protocol lacp

channel-group 1 mode passive

interface GigabitEthernet1/0/6

description 2nd link to 3650 etherchannel test

switchport mode dynamic desirable

channel-protocol lacp

channel-group 1 mode passive

The configuration of the PnP switch includes the EtherChannel:

hostname 3650-dhcp

enable password xxx

!

username xxx password 0 xxx

!

ip http server

ip http secure-server

snmp-server community xxx RO

interface Port-channel1

switchport mode dynamic desirable

no port-channel standalone-disable

!

int range g1/0/1,g1/0/3

switchport mode dynamic desirable

switchport trunk allowed vlan except 1

channel-protocol lacp

channel-group 1 mode active

!

line con 0

line vty 0 4

login local

transport input ssh telnet

line vty 5 15

login local

transport input ssh telnet

!

end

Looking at debugs, you can see both interfaces are up, and then the port channel comes up, after the configuration has been downloaded to the PnP switch.  Again, VLAN 14 is used for the management VLAN.

Oct 5 21:58:54.638: %PKI-6-PKCS12IMPORT_SUCCESS: PKCS #12 Successfully Imported.

Oct 5 21:59:07.138: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to down

Oct 5 21:59:07.155: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/3, changed state to down

Oct 5 21:59:08.138: %LINK-3-UPDOWN: Interface Vlan14, changed state to down

Oct  5 21:59:09.071: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel1, changed state to up

.Oct 5 21:59:09.094: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up

.Oct 5 21:59:09.139: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan14, changed state to down

.Oct 5 21:59:09.188: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/3, changed state to up

.Oct 5 21:59:10.085: %LINK-3-UPDOWN: Interface Port-channel1, changed state to up

.Oct 5 21:59:11.241: %LINK-3-UPDOWN: Interface Vlan14, changed state to up

.Oct  5 21:59:12.242: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan14, changed state to up

Looking at the PnP switch, we can see the Management interface is using VLAN 14 and DHCP to obtain an IP address.

3650-dhcp#show ip int br

Interface              IP-Address      OK? Method Status                Protocol

Vlan1                  unassigned      YES unset administratively down down   

Vlan14                 10.10.14.3      YES DHCP up                    up

Can also see the status of the ether-channel.  Both ports are active and a part of the ether-channel.

3650-dhcp#show etherchannel 1 port-channel

         Port-channels in the group:

         ---------------------------

Port-channel: Po1    (Primary Aggregator)

------------

Age of the Port-channel   = 0d:00h:27m:46s

Logical slot/port   = 12/1          Number of ports = 2

HotStandBy port = null

Port state          = Port-channel Ag-Inuse

Protocol            = LACP

Port security       = Disabled

Standalone          = Enabled (independent mode)

Ports in the Port-channel:

Index Load   Port     EC state        No of bits

------+------+------+------------------+-----------

  0     00 Gi1/0/1  Active             0

  0 00     Gi1/0/3  Active             0

This also shows VLAN1 is no longer sent over the ether-channel trunk link

3650-dhcp#show int port-channel 1 switchport

Name: Po1

Switchport: Enabled

Administrative Mode: dynamic desirable

Operational Mode: trunk

Administrative Trunking Encapsulation: dot1q

Operational Trunking Encapsulation: dot1q

Negotiation of Trunking: On

Access Mode VLAN: 14 (VLAN0014)

Trunking Native Mode VLAN: 1 (default)

Administrative Native VLAN tagging: enabled

Voice VLAN: none

Administrative private-vlan host-association: none

Administrative private-vlan mapping: none

Administrative private-vlan trunk native VLAN: none

Administrative private-vlan trunk Native VLAN tagging: enabled

Administrative private-vlan trunk encapsulation: dot1q

Administrative private-vlan trunk normal VLANs: none

Administrative private-vlan trunk associations: none

Administrative private-vlan trunk mappings: none

Operational private-vlan: none

Trunking VLANs Enabled: 2-4094

Pruning VLANs Enabled: 2-1001

Protected: false

Unknown unicast blocked: disabled

Unknown multicast blocked: disabled

Appliance trust: none

Time since last port bundled:    0d:00h:27m:44s    Gi1/0/3

What Next?

This blog covered three standard deployment models for network plug and play.  Other blogs in the series have covered the API and how to automate the creation, upload of configuration files as well as the automation of rules.  In future I will cover some of new enhancements coming in the 1.3 release, including configuration templates, native in APIC-EM.

In the meantime, if you would like to learn more about this, you could come hang out with us in The Cisco Devnet DNA Community. We’ll have a continuous stream of blogs like this and you can ask questions and we’ll get you answers. In addition, we have a Github repository where you can get examples related to PnP.

Thanks for reading,

@adamradford123

9 Comments
Cisco Employee

Awesome post Adam...  excellent information here.

Cisco Employee

Thanks Craig, a few more posts in the works.  Stacking and Etherchannel with Non Native vlan1 is next.

Enthusiast

Hi Adam,

The command "switchport mode dynamic desirable"on Trunk port is not supported on Cisco 800 Routers.

What is the workaround ?

Thanks

/Sebastien

Cisco Employee

Hi Sebastien,

no work around i am aware of.  Is this on both WAN and LAN ports?

Adam

Beginner

Very good posting, a quick question about flat deployment: So upstream switch will be configured as DHCP server?

Cisco Employee

It can be, but is not essential.

Sent from my iPhone

Beginner

Hi, great post.

Can you set a MGMT-ip in the same subnet as the switch first recieve a DHCP address from?

Say you have a subnet 192.168.1.0/24, where 192.168.1.0-100 is excluded from DHCP for aligning static addresses, and 192.168.1.100-254 is used in a DHCP scope.

Switch first recieves IP from DHCP, let's say 192.168.1.101, and starts connecting to APIC-EM.

You then want to set a statically assigned address to the switch, for example 192.168.1.5, specified in the APIC-EM template.

Is this possible, and can APIC-EM then continue connecting to the switch on the new IP?

I've tried to create a new VLAN, say VLAN 5, and then set a static IP in the same range as the already obtained DHCP-address, and this results in a error like "192.168.1.0 overlaps with Vlan1.

Cisco Employee

Thanks for the feedback.

Sure, you can do this.

You just need to include the following lines to remove the IP address from vlan 1 first.  you only need to do this if the addresses overlap.  Make sure these line come before the vlan 5 definition and IP address assignment.

int vlan 1

no ip address

The reason it is an issue is that a "router" cannot have two interfaces on the same network (unless you are using VRF).

All communication is from device to APIC-EM, so changing IP address/interface etc is fine.  I have tested this many times.

Adam

Beginner

Great post.

 

I'm using a 3650 a PnP device that is connected to a 9300 that has an etherchannel + NV1 (pnp startup-vlan) configured. I'm reusing the DHCP option 43 from other pools where I have on the same 9300 that are working.

The 3650 gets an IP address, but never contacts DNA Center. At least I never see it there...

 

The 3650 is running 16.8.1a. 

 

I'm able to ping the 3650 from DNAC on its DHCP assigned IP in the NV1 scope.

 

I also followed the instructions in LTRNMS-2007

 

This is what I'm seeing on the console of 3650:

 

*Aug 20 12:31:41.067: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/3, changed state to up
*Aug 20 12:31:42.978: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
*Aug 20 12:31:43.585: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/3, changed state to up
*Aug 20 12:31:45.265: %PKI-2-NON_AUTHORITATIVE_CLOCK: PKI timers have not been initialized due to non-authoritative system clock. Ensure system clock is configured/updated.
*Aug 20 12:31:45.351: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named CISCO_IDEVID_SUDI_LEGACY has been generated or imported
*Aug 20 12:31:45.498: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named CISCO_IDEVID_SUDI has been generated or imported
*Aug 20 12:31:50.513: %SYS-5-CONFIG_I: Configured from console by vty0
*Aug 20 12:31:50.805: %SPANTREE-2-RECV_PVID_ERR: Received BPDU with inconsistent peer vlan id 9 on GigabitEthernet1/0/3 VLAN1.
*Aug 20 12:31:50.805: %SPANTREE-2-BLOCK_PVID_PEER: Blocking GigabitEthernet1/0/3 on VLAN0009. Inconsistent peer vlan.
*Aug 20 12:31:50.805: %SPANTREE-2-BLOCK_PVID_LOCAL: Blocking GigabitEthernet1/0/3 on VLAN0001. Inconsistent local vlan.
*Aug 20 12:31:51.530: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan9, changed state to up
*Aug 20 12:31:52.377: %LINK-5-CHANGED: Interface Vlan1, changed state to administratively down
*Aug 20 12:32:05.806: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking GigabitEthernet1/0/3 on VLAN0009. Port consistency restored.
*Aug 20 12:32:05.806: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking GigabitEthernet1/0/3 on VLAN0001. Port consistency restored.
*Aug 20 12:32:06.666: %DHCP-6-ADDRESS_ASSIGN: Interface Vlan9 assigned DHCP address 10.71.128.52, mask 255.255.255.240, hostname 

Ping from DNAC:

 

$ ping 10.71.128.52
PING 10.71.128.52 (10.71.128.52) 56(84) bytes of data.
64 bytes from 10.71.128.52: icmp_seq=1 ttl=250 time=1.61 ms
64 bytes from 10.71.128.52: icmp_seq=2 ttl=250 time=1.44 ms
64 bytes from 10.71.128.52: icmp_seq=3 ttl=250 time=1.39 ms
64 bytes from 10.71.128.52: icmp_seq=4 ttl=250 time=1.45 ms
64 bytes from 10.71.128.52: icmp_seq=5 ttl=250 time=1.30 ms
64 bytes from 10.71.128.52: icmp_seq=6 ttl=250 time=1.39 ms
64 bytes from 10.71.128.52: icmp_seq=7 ttl=250 time=1.35 ms
64 bytes from 10.71.128.52: icmp_seq=8 ttl=250 time=1.35 ms
^C
--- 10.71.128.52 ping statistics ---
8 packets transmitted, 8 received, 0% packet loss, time 7008ms
rtt min/avg/max/mdev = 1.308/1.415/1.611/0.099 ms

[Tue Aug 20 12:27:02 UTC] maglev@10.169.254.1 (maglev-master-1) ~
$ 

Any suggestions are quite welcome.

 

Thanks.

This widget could not be displayed.