Continuing the story
My last few blogs showed examples of using APIC-EM REST API to upload configuration files and create rules for PnP devices.
I am often asked about the different deployment models for switches. There are a few different concepts to take into account, VLANs, management VLANs, trunks, EtherChannels etc.
If you have a very simple network with VLAN 1 for management and are not using any of the features above, PnP just works, so no need to keep reading.
This blog post demystifies the different deployment models for edge switches. We will cover three basic deployment models:
- "Flat" with non-VLAN 1 (NV1) for management
- Trunked with NV1 for management and a static IP address
- Ether Channel with NV1 for management
For all of these examples I am using a 3650 switch running 16.3.1 code, but you could use versions of 3.6.5 and 3.7.4 (For other platforms such 2960x please see release notes for details).
Make sure you do not hit any keys on the console while the switch is booting, as this can interrupt the PnP process.
The first thing we need is a mechanism for the switch to discover the controller. In our examples we are going to use DHCP, but you could also use DNS etc as covered in earlier blogs. Here is a sample configuration for an IOS switch. The controller IP address is 10.10.10.140. Note also the use of the "5A1D" in the option 43 string. The "D" displays debug messages for PnP on the console of the PnP switch.
Setup DHCP server
ip dhcp pool ZTD-switches
network 10.10.14.0 255.255.255.0
default-router 10.10.14.1
option 43 ascii "5A1D;B2;K4;I10.10.10.140;J80"
remember
Lets take a look at the first of the three scenarios.
1. Flat deployment – NV1 for management
The two switches are in a "flat" configuration. Only one VLAN is defined on the PnP switch and the management interface is in that VLAN.
This is the configuration on the upstream switch. The "pnp startup-vlan 14" command is required to create a new management VLAN on the PnP switch. By default VLAN 1 would be used.
pnp startup-vlan 14
interface GigabitEthernet1/0/5
description PNP switch 3650->g1/0/1
switchport access vlan 14
The configuration for the PnP switch is very simple.
hostname 3650-dhcp
enable password xxxx
!
username xxx password 0 xxxx
!
ip http server
ip http secure-server
snmp-server community xxx RO
!
!
!
!
line con 0
line vty 0 4
login local
transport input ssh telnet
line vty 5 15
login local
transport input ssh telnet
!
end
The debug logs show the new VLAN (14) being configured. This happens via a CDP negotiation between the upstream switch and the PnP switch.
Cisco IOS Software [Denali], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.3.1, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2016 by Cisco Systems, Inc.
Compiled Tue 02-Aug-16 17:33 by mcpre
*Oct 6 01:24:19.193: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up
*Oct 6 01:24:20.074: %SYS-6-BOOTTIME: Time taken to reboot after reload = 332 seconds
*Oct 6 01:24:20.193: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
*Oct 6 01:24:21.258: %LINK-5-CHANGED: Interface Vlan1, changed state to administratively down
*Oct 6 01:24:28.299: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on GigabitEthernet1/0/1 (1), with 3850-core GigabitEthernet1/0/5 (14).
*Oct 6 01:24:29.204: %SYS-5-CONFIG_I: Configured from console by tty100
*Oct 6 01:24:29.666: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan14, changed state to down
*Oct 6 01:24:52.796: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan14, changed state to up
*Oct 6 01:24:58.352: %PNPA-DHCP Op-43 Msg: Process state = READY
*Oct 6 01:24:58.352: %PNPA-DHCP Op-43 Msg: OK to process message
*Oct 6 01:24:58.353: XML-UPDOWN: PNPA_DHCP_OP43 XML Interface(102) UP. PID=359
*Oct 6 01:24:58.354: %PNPA-DHCP Op-43 Msg: _pdoon.1.ntf.don=359
*Oct 6 01:24:58.354: %PNPA-DHCP Op-43 Msg: _pdoop.1.org=[A1D;B2;K4;I10.10.10.140;J80]
*Oct 6 01:24:58.354: %PNPA-DHCP Op-43 Msg: _pdgfa.1.inp=[B2;K4;I10.10.10.140;J80]
*Oct 6 01:24:58.354: %PNPA-DHCP Op-43 Msg: _pdgfa.1.B2.s12=[ ipv4 ]
*Oct 6 01:24:58.355: %PNPA-DHCP Op-43 Msg: _pdgfa.1.K4.htp=[ transport http ]
*Oct 6 01:24:58.355: %PNPA-DHCP Op-43 Msg: _pdgfa.1.Ix.srv.ip.rm=[ 10.10.10.140 ]
*Oct 6 01:24:58.390: %PNPA-DHCP Op-43 Msg: _pdgfa.1.Jx.srv.rt.rm=[ port 80 ]
*Oct 6 01:24:58.390: %PNPA-DHCP Op-43 Msg: _pdoop.1.ztp=[pnp-zero-touch] host=[] ipad=[10.10.10.140] port=80
*Oct 6 01:24:58.390: %PNPA-DHCP Op-43 Msg: _pors.done=1
*Oct 6 01:24:58.390: %PNPA-DHCP Op-43 Msg: _pdokp.1.kil=[PNPA_DHCP_OP43] pid=359 idn=[Vlan14]
*Oct 6 01:24:58.390: XML-UPDOWN: Vlan14 XML Interface(102) SHUTDOWN(101). PID=359
*Oct 6 01:24:59.298: %PNPA-DHCP Op-43 Msg: Op43 has 5A. It is for PnP
*Oct 6 01:24:59.298: %PNPA-DHCP Op-43 Msg: After stripping extra characters in front of 5A, if any: 5A1D;B2;K4;I10.10.10.140;J80 op43_len: 28
*Oct 6 01:24:59.298: %PNPA-DHCP Op-43 Msg: _pdoon.2.ina=[Vlan14]
*Oct 6 01:24:59.298: %PNPA-DHCP Op-43 Msg: _papdo.2.cot=[5A1D;B2;K4;I10.10.10.140;J80] lot=[5A1D;B2;K4;I10.10.10.140;J80]
*Oct 6 01:24:59.298: %PNPA-DHCP Op-43 Msg: Process state = READY
*Oct 6 01:24:59.298: %PNPA-DHCP Op-43 Msg: OK to process message
*Oct 6 01:24:59.299: XML-UPDOWN: PNPA_DHCP_OP43 XML Interface(102) UP. PID=359
*Oct 6 01:24:59.299: %PNPA-DHCP Op-43 Msg: _pdoon.2.ntf.don=359
*Oct 6 01:24:59.301: %PNPA-DHCP Op-43 Msg: _pdoop.2.org=[A1D;B2;K4;I10.10.10.140;J80]
*Oct 6 01:24:59.301: %PNPA-DHCP Op-43 Msg: _pdgfa.2.inp=[B2;K4;I10.10.10.140;J80]
*Oct 6 01:24:59.301: %PNPA-DHCP Op-43 Msg: _pdgfa.2.B2.s12=[ ipv4 ]
*Oct 6 01:24:59.301: %PNPA-DHCP Op-43 Msg: _pdgfa.2.K4.htp=[ transport http ]
*Oct 6 01:24:59.301: %PNPA-DHCP Op-43 Msg: _pdgfa.2.Ix.srv.ip.rm=[ 10.10.10.140 ]
*Oct 6 01:24:59.301: %PNPA-DHCP Op-43 Msg: _pdgfa.2.Jx.srv.rt.rm=[ port 80 ]
*Oct 6 01:24:59.302: %PNPA-DHCP Op-43 Msg: _pdoop.2.ztp=[pnp-zero-touch] host=[] ipad=[10.10.10.140] port=80
*Oct 6 01:24:59.302: %PNPA-DHCP Op-43 Msg: _pors.done=1
*Oct 6 01:24:59.302: %PNPA-DHCP Op-43 Msg: _pdokp.2.kil=[PNPA_DHCP_OP43] pid=359 idn=[Vlan14]
*Oct 6 01:24:59.302: XML-UPDOWN: Vlan14 XML Interface(102) SHUTDOWN(101). PID=359
*Oct 6 01:24:59.411: %DHCP-6-ADDRESS_ASSIGN: Interface Vlan14 assigned DHCP address 10.10.14.3, mask 255.255.255.0, hostname
% Generating 2048 bit RSA keys, keys will be non-exportable... got vend id vend spec. info ret: succeed
*Oct 6 01:25:13.341: %PNP-6-HTTP_CONNECTING: PnP Discovery trying to connect to PnP server http://10.10.10.140:80/pnp/HELLO
*Oct 6 01:25:13.351: %PNP-6-HTTP_CONNECTED: PnP Discovery connected to PnP server http://10.10.10.140:80/pnp/HELLO
[OK] (elapsed time was 9 seconds)
Before the configuration is applied to the switch via PnP, you can see that the CDP "pnp startup-vlan" command has completed. It has moved the active port into VLAN 14 and created VLAN 14 on the switch and enabled DHCP.
Switch#show run int g1/0/1
Building configuration...
Current configuration : 100 bytes
!
interface GigabitEthernet1/0/1
switchport access vlan 14
macro description CISCO_SMI_EVENT
end
This shows the creation of VLAN 14, and the shutdown of VLAN1.
show ip int br
Interface IP-Address OK? Method Status Protocol
Vlan1 unassigned YES unset administratively down down
Vlan14 10.10.14.3 YES DHCP up up
Once the configuration is complete, the uplink connection is in access mode using VLAN 14. The only real change downloaded in the configuration was the switch hostname "3650-dhcp".
3650-dhcp#show int g1/0/1 switchport
Name: Gi1/0/1
Switchport: Enabled
Administrative Mode: dynamic auto
Operational Mode: static access
Administrative Trunking Encapsulation: dot1q
Operational Trunking Encapsulation: native
Negotiation of Trunking: On
Access Mode VLAN: 14 (VLAN0014)
2. Trunk – NV1 for management – static IP address
The upstream switch and PnP switch are going to be connected by a trunk port.
In this scenario, the upstream switch has a trunk mode desirable, and no VLANs defined.
interface GigabitEthernet1/0/5
description PNP switch 3650->g1/0/1
switchport mode dynamic desirable
In this example, the DHCP address is going to be overwritten by a permanent static management IP address. Note: when you do this, you also need to provide a default route (ip route 0.0.0.0 0.0.0.0 10.10.14.1), otherwise the PnP device will not be able to contact the controller after the configuration has been downloaded.
hostname 3650-dhcp
enable password xxxx
!
username xxx password 0 xxxx
!
ip http server
ip http secure-server
snmp-server community xxx RO
!
vlan 2222
vlan 2223
int vlan 14
ip address 10.10.14.100 255.255.255.0
ip route 0.0.0.0 0.0.0.0 10.10.14.1
!
!
line con 0
line vty 0 4
login local
transport input ssh telnet
line vty 5 15
login local
transport input ssh telnet
!
end
The DHCP address has been overwritten by the static address in the configuration file.
3650-dhcp#show run int vlan14
Building configuration...
Current configuration : 63 bytes
!
interface Vlan14
ip address 10.10.14.100 255.255.255.0
end
The uplink interface is in trunk mode and has both the management VLANs as well as the locally defined VLANs on it.
3650-dhcp#show interfaces g1/0/1 trunk
Port Mode Encapsulation Status Native vlan
Gi1/0/1 auto 802.1q trunking 1
Port Vlans allowed on trunk
Gi1/0/1 1-4094
Port Vlans allowed and active in management domain
Gi1/0/1 1,14,2222-2223
Port Vlans in spanning tree forwarding state and not pruned
Gi1/0/1 1,14,2222-2223
Protected: false
Unknown unicast blocked: disabled
Unknown multicast blocked: disabled
Appliance trust: none
3. EtherChannel + NV1 for management
The switches are going to be connected by two links bound together in an ether-channel.
The upstream switch needs to have an ether channel configured. To avoid issues when the PnP switch first comes up, the "no port-channel standalone-disable" command is required. If this is left out the channel will be disabled as it has not been configured on the PnP switch at boot up.
interface Port-channel1
switchport mode dynamic desirable
no port-channel standalone-disable
interface GigabitEthernet1/0/5
description PNP switch 3650->g1/0/1
switchport mode dynamic desirable
channel-protocol lacp
channel-group 1 mode passive
interface GigabitEthernet1/0/6
description 2nd link to 3650 etherchannel test
switchport mode dynamic desirable
channel-protocol lacp
channel-group 1 mode passive
The configuration of the PnP switch includes the EtherChannel:
hostname 3650-dhcp
enable password xxx
!
username xxx password 0 xxx
!
ip http server
ip http secure-server
snmp-server community xxx RO
interface Port-channel1
switchport mode dynamic desirable
no port-channel standalone-disable
!
int range g1/0/1,g1/0/3
switchport mode dynamic desirable
switchport trunk allowed vlan except 1
channel-protocol lacp
channel-group 1 mode active
!
line con 0
line vty 0 4
login local
transport input ssh telnet
line vty 5 15
login local
transport input ssh telnet
!
end
Looking at debugs, you can see both interfaces are up, and then the port channel comes up, after the configuration has been downloaded to the PnP switch. Again, VLAN 14 is used for the management VLAN.
Oct 5 21:58:54.638: %PKI-6-PKCS12IMPORT_SUCCESS: PKCS #12 Successfully Imported.
Oct 5 21:59:07.138: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to down
Oct 5 21:59:07.155: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/3, changed state to down
Oct 5 21:59:08.138: %LINK-3-UPDOWN: Interface Vlan14, changed state to down
Oct 5 21:59:09.071: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel1, changed state to up
.Oct 5 21:59:09.094: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
.Oct 5 21:59:09.139: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan14, changed state to down
.Oct 5 21:59:09.188: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/3, changed state to up
.Oct 5 21:59:10.085: %LINK-3-UPDOWN: Interface Port-channel1, changed state to up
.Oct 5 21:59:11.241: %LINK-3-UPDOWN: Interface Vlan14, changed state to up
.Oct 5 21:59:12.242: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan14, changed state to up
Looking at the PnP switch, we can see the Management interface is using VLAN 14 and DHCP to obtain an IP address.
3650-dhcp#show ip int br
Interface IP-Address OK? Method Status Protocol
Vlan1 unassigned YES unset administratively down down
Vlan14 10.10.14.3 YES DHCP up up
Can also see the status of the ether-channel. Both ports are active and a part of the ether-channel.
3650-dhcp#show etherchannel 1 port-channel
Port-channels in the group:
---------------------------
Port-channel: Po1 (Primary Aggregator)
------------
Age of the Port-channel = 0d:00h:27m:46s
Logical slot/port = 12/1 Number of ports = 2
HotStandBy port = null
Port state = Port-channel Ag-Inuse
Protocol = LACP
Port security = Disabled
Standalone = Enabled (independent mode)
Ports in the Port-channel:
Index Load Port EC state No of bits
------+------+------+------------------+-----------
0 00 Gi1/0/1 Active 0
0 00 Gi1/0/3 Active 0
This also shows VLAN1 is no longer sent over the ether-channel trunk link
3650-dhcp#show int port-channel 1 switchport
Name: Po1
Switchport: Enabled
Administrative Mode: dynamic desirable
Operational Mode: trunk
Administrative Trunking Encapsulation: dot1q
Operational Trunking Encapsulation: dot1q
Negotiation of Trunking: On
Access Mode VLAN: 14 (VLAN0014)
Trunking Native Mode VLAN: 1 (default)
Administrative Native VLAN tagging: enabled
Voice VLAN: none
Administrative private-vlan host-association: none
Administrative private-vlan mapping: none
Administrative private-vlan trunk native VLAN: none
Administrative private-vlan trunk Native VLAN tagging: enabled
Administrative private-vlan trunk encapsulation: dot1q
Administrative private-vlan trunk normal VLANs: none
Administrative private-vlan trunk associations: none
Administrative private-vlan trunk mappings: none
Operational private-vlan: none
Trunking VLANs Enabled: 2-4094
Pruning VLANs Enabled: 2-1001
Protected: false
Unknown unicast blocked: disabled
Unknown multicast blocked: disabled
Appliance trust: none
Time since last port bundled: 0d:00h:27m:44s Gi1/0/3
What Next?
This blog covered three standard deployment models for network plug and play. Other blogs in the series have covered the API and how to automate the creation, upload of configuration files as well as the automation of rules. In future I will cover some of new enhancements coming in the 1.3 release, including configuration templates, native in APIC-EM.
In the meantime, if you would like to learn more about this, you could come hang out with us in The Cisco Devnet DNA Community. We’ll have a continuous stream of blogs like this and you can ask questions and we’ll get you answers. In addition, we have a Github repository where you can get examples related to PnP.
Thanks for reading,
10 Comments