Continuing the story
My initial blogs showed examples of using APIC-EM REST API to upload configuration files and create rules for PnP devices. More recently, I showed some of the common deployment models for switches, taking into account, VLANs, management VLANs, trunks, EtherChannels etc.
This blog covers an advanced use case. Etherchannel + trunk + Non Vlan 1 for management (NV1) + Non vlan1 native vlan.
For all of these examples I am using a 3650 switch running 16.3.1 code, but you could use versions of 3.6.5 and 3.7.4 (For other platforms such 2960x please see release notes for details).
Make sure you do not hit any keys on the console while the switch is booting, as this can interrupt the PnP process.
Setup DHCP server
The first thing we need is a mechanism for the switch to discover the controller. In our examples we are going to use DHCP, but you could also use DNS etc. as covered in earlier blogs. Here is a sample configuration for an IOS switch. The controller IP address is 10.10.10.140. Note also the use of the "5A1D" in the option 43 string. The "D" displays debug messages for PnP on the console of the PnP switch.
ip dhcp pool ZTD-switches
network 10.10.14.0 255.255.255.0
default-router 10.10.14.1
option 43 ascii "5A1D;B2;K4;I10.10.10.140;J80"
remember
EtherChannel + NV1 for management + NV1 Native
The switches are going to be connected by two links bound together in an ether-channel.
The upstream switch needs to have an ether channel configured. To avoid issues when the PnP switch first comes up, the "no port-channel standalone-disable" command is required. If this is left out the channel will be disabled as it has not been configured on the PnP switch at boot up.
In addition the native VLAN has been set to 999.
The "pnp startup-vlan 14" command is required to create a new management VLAN on the PnP switch. By default VLAN 1 would be used.
pnp startup-vlan 14
interface Port-channel1
switchport trunk native vlan 999
switchport mode dynamic desirable
no port-channel standalone-disable
interface GigabitEthernet1/0/5
description PNP switch 3650->g1/0/1
switchport mode dynamic desirable
switchport trunk native vlan 999
channel-protocol lacp
channel-group 1 mode passive
interface GigabitEthernet1/0/6
description 2nd link to 3650 etherchannel test
switchport trunk native vlan 999
switchport mode dynamic desirable
channel-protocol lacp
channel-group 1 mode passive
The configuration of the PnP switch includes the EtherChannel. It also has native VLAN of 999.
NOTE: remember to define VLAN 999 otherwise you will have issues after you deploy the configuration.
hostname 3650-dhcp
enable password xxx
!
username xxx password 0 xxx
!
ip http server
ip http secure-server
snmp-server community xxx RO
interface Port-channel1
switchport mode dynamic desirable
switchport trunk native vlan 999
no port-channel standalone-disable
!
int range g1/0/1,g1/0/3
switchport mode dynamic desirable
switchport trunk native vlan 999
channel-protocol lacp
channel-group 1 mode active
vlan 999
!
!
!
!
line con 0
line vty 0 4
login local
transport input ssh telnet
line vty 5 15
login local
Once the switch has booted, you will see the following log messages (remember to use the "5A1D" string as mentioned earlier to get the debugs). Both VLAN 1 and 999 are initially blocked, but VLAN 14 is created as the management VLAN. VLAN 14 is used to communicate with APIC-EM, and you can see the switch successfully connects to the controller.
You will also see warning messages about the Native VLAN mismatch, and you can ignore them for now. Once the configuration is downloaded to the PnP switch, these will go away.
*Oct 10 22:16:27.480: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up
*Oct 10 22:16:27.816: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/3, changed state to up
*Oct 10 22:16:28.414: %SYS-6-BOOTTIME: Time taken to reboot after reload = 339 seconds
*Oct 10 22:16:31.489: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
*Oct 10 22:16:31.825: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/3, changed state to up
*Oct 10 22:16:38.810: %SPANTREE-2-RECV_PVID_ERR: Received BPDU with inconsistent peer vlan id 999 on GigabitEthernet1/0/1 VLAN1.
*Oct 10 22:16:38.810: %SPANTREE-2-BLOCK_PVID_PEER: Blocking GigabitEthernet1/0/1 on VLAN0999. Inconsistent peer vlan.
*Oct 10 22:16:38.811: %SPANTREE-2-BLOCK_PVID_LOCAL: Blocking GigabitEthernet1/0/1 on VLAN0001. Inconsistent local vlan.
*Oct 10 22:16:38.811: %SPANTREE-2-RECV_PVID_ERR: Received BPDU with inconsistent peer vlan id 999 on GigabitEthernet1/0/3 VLAN1.
*Oct 10 22:16:38.811: %SPANTREE-2-BLOCK_PVID_PEER: Blocking GigabitEthernet1/0/3 on VLAN0999. Inconsistent peer vlan.
*Oct 10 22:16:38.812: %SPANTREE-2-BLOCK_PVID_LOCAL: Blocking GigabitEthernet1/0/3 on VLAN0001. Inconsistent local vlan.
*Oct 10 22:16:40.072: %LINK-5-CHANGED: Interface Vlan1, changed state to administratively down
*Oct 10 22:16:47.124: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on GigabitEthernet1/0/3 (1), with 3850-core GigabitEthernet1/0/6 (999).
*Oct 10 22:16:47.254: %SYS-5-CONFIG_I: Configured from console by tty100
*Oct 10 22:16:47.291: %SYS-5-CONFIG_I: Configured from console by tty100
*Oct 10 22:16:50.145: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan14, changed state to up
*Oct 10 22:16:56.126: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on GigabitEthernet1/0/1 (1), with 3850-core GigabitEthernet1/0/5 (999).
*Oct 10 22:17:05.127: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on GigabitEthernet1/0/1 (1), with 3850-core GigabitEthernet1/0/5 (999).
*Oct 10 22:17:14.127: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on GigabitEthernet1/0/3 (1), with 3850-core GigabitEthernet1/0/6 (999).
*Oct 10 22:17:21.376: %PNPA-DHCP Op-43 Msg: Process state = READY
*Oct 10 22:17:21.376: %PNPA-DHCP Op-43 Msg: OK to process message
*Oct 10 22:17:21.377: XML-UPDOWN: PNPA_DHCP_OP43 XML Interface(102) UP. PID=279
*Oct 10 22:17:21.377: %PNPA-DHCP Op-43 Msg: _pdoon.1.ntf.don=279
*Oct 10 22:17:21.378: %PNPA-DHCP Op-43 Msg: _pdoop.1.org=[A1D;B2;K4;I10.10.10.140;J80]
*Oct 10 22:17:21.378: %PNPA-DHCP Op-43 Msg: _pdgfa.1.inp=[B2;K4;I10.10.10.140;J80]
*Oct 10 22:17:21.378: %PNPA-DHCP Op-43 Msg: _pdgfa.1.B2.s12=[ ipv4 ]
*Oct 10 22:17:21.378: %PNPA-DHCP Op-43 Msg: _pdgfa.1.K4.htp=[ transport http ]
*Oct 10 22:17:21.378: %PNPA-DHCP Op-43 Msg: _pdgfa.1.Ix.srv.ip.rm=[ 10.10.10.140 ]
*Oct 10 22:17:21.378: %PNPA-DHCP Op-43 Msg: _pdgfa.1.Jx.srv.rt.rm=[ port 80 ]
*Oct 10 22:17:21.379: %PNPA-DHCP Op-43 Msg: _pdoop.1.ztp=[pnp-zero-touch] host=[] ipad=[10.10.10.140] port=80
*Oct 10 22:17:21.379: %PNPA-DHCP Op-43 Msg: _pors.done=1
*Oct 10 22:17:21.379: %PNPA-DHCP Op-43 Msg: _pdokp.1.kil=[PNPA_DHCP_OP43] pid=279 idn=[Vlan14]
*Oct 10 22:17:21.379: XML-UPDOWN: Vlan14 XML Interface(102) SHUTDOWN(101). PID=279
*Oct 10 22:17:21.493: %DHCP-6-ADDRESS_ASSIGN: Interface Vlan14 assigned DHCP address 10.10.14.3, mask 255.255.255.0, hostname
*Oct 10 22:17:23.129: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on GigabitEthernet1/0/1 (1), with 3850-core GigabitEthernet1/0/5 (999).
*Oct 10 22:17:32.130: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on GigabitEthernet1/0/3 (1), with 3850-core GigabitEthernet1/0/6 (999).% Generating 2048 bit RSA keys, keys will be non-exportable... got vend id vend spec. info ret: succeed
*Oct 10 22:17:39.848: %PNP-6-HTTP_CONNECTING: PnP Discovery trying to connect to PnP server http://10.10.10.140:80/pnp/HELLO
*Oct 10 22:17:39.858: %PNP-6-HTTP_CONNECTED: PnP Discovery connected to PnP server http://10.10.10.140:80/pnp/HELLO
*Oct 10 22:17:41.142: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on GigabitEthernet1/0/1 (1), with 3850-core GigabitEthernet1/0/5 (999).
[OK] (elapsed time was 6 seconds)
protocol on Interface Vlan14, changed state to up
Looking at the initial configuration created as a result of the "pnp startup-vlan 14" command, both active interfaces have been placed in VLAN 14. Interface VLAN 14 was created and set to use DHCP.
interface GigabitEthernet1/0/1
switchport access vlan 14
macro description CISCO_SMI_EVENT
!
interface GigabitEthernet1/0/3
switchport access vlan 14
macro description CISCO_SMI_EVENT
!
interface Vlan14
ip address dhcp
Both interfaces are in trunk mode.
Switch#show int g1/0/1 trunk
Port Mode Encapsulation Status Native vlan
Gi1/0/1 auto 802.1q trunking 1
Port Vlans allowed on trunk
Gi1/0/1 1-4094
Port Vlans allowed and active in management domain
Gi1/0/1 1,14,999
Port Vlans in spanning tree forwarding state and not pruned
Gi1/0/1 14S
Switch#show int g1/0/3 trunk
Port Mode Encapsulation Status Native vlan
Gi1/0/3 auto 802.1q trunking 1
Port Vlans allowed on trunk
Gi1/0/3 1-4094
Port Vlans allowed and active in management domain
Gi1/0/3 1,14,999
Port Vlans in spanning tree forwarding state and not pruned
Gi1/0/3 none
At this point, I now claim the device and push the configuration to it. I could have also used a pre defined rule, but I wanted to show the intermediate steps.
Looking at debugs, you can see both interfaces are up, and then the port channel comes up, after the configuration has been downloaded to the PnP switch. Again, VLAN 14 is used for the management VLAN.
Oct 10 23:12:59.673: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to down
Oct 10 23:12:59.689: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/3, changed state to down
Oct 10 23:13:00.619: %LINK-3-UPDOWN: Interface Port-channel1, changed state to up
Oct 10 23:13:00.674: %LINK-3-UPDOWN: Interface Vlan14, changed state to down
Oct 10 23:13:01.578: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
Oct 10 23:13:01.620: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel1, changed state to up
Oct 10 23:13:01.728: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/3, changed state to up
Oct 10 23:13:03.267: %LINK-3-UPDOWN: Interface Vlan14, changed state to up
We can see the status of the ether-channel. Both ports are active and a part of the ether-channel
3650-dhcp#show etherchannel 1 port-channel
Port-channels in the group:
---------------------------
Port-channel: Po1 (Primary Aggregator)
------------
Age of the Port-channel = 0d:00h:17m:14s
Logical slot/port = 12/1 Number of ports = 2
HotStandBy port = null
Port state = Port-channel Ag-Inuse
Protocol = LACP
Port security = Disabled
Standalone = Enabled (independent mode)
Ports in the Port-channel:
Index Load Port EC state No of bits
------+------+------+------------------+-----------
0 00 Gi1/0/1 Active 0
0 00 Gi1/0/3 Active 0
This also shows VLAN999 is now the native VLAN.
3650-dhcp#show int port-channel 1 switchport
Name: Po1
Switchport: Enabled
Administrative Mode: dynamic desirable
Operational Mode: trunk
Administrative Trunking Encapsulation: dot1q
Operational Trunking Encapsulation: dot1q
Negotiation of Trunking: On
Access Mode VLAN: 14 (VLAN0014)
Trunking Native Mode VLAN: 999 (VLAN0999)
Administrative Native VLAN tagging: enabled
Voice VLAN: none
Administrative private-vlan host-association: none
Administrative private-vlan mapping: none
Administrative private-vlan trunk native VLAN: none
Administrative private-vlan trunk Native VLAN tagging: enabled
Administrative private-vlan trunk encapsulation: dot1q
Administrative private-vlan trunk normal VLANs: none
Administrative private-vlan trunk associations: none
Administrative private-vlan trunk mappings: none
Operational private-vlan: none
Trunking VLANs Enabled: ALL
Pruning VLANs Enabled: 2-1001
Protected: false
Unknown unicast blocked: disabled
Unknown multicast blocked: disabled
Appliance trust: none
What Next?
This blog covered an advanced deployment models for network plug and play. Other blogs in the series have covered simpler deployment models and the API and how to automate the creation, upload of configuration files as well as the automation of rules. In future I will cover switch stacking.
In the meantime, if you would like to learn more about this, you could come hang out with us in The Cisco Devnet DNA Community. We’ll have a continuous stream of blogs like this and you can ask questions and we’ll get you answers. In addition, we have a Github repository where you can get examples related to PnP.
Thanks for reading,