Solved: Re: {PNP}Streamlining Zero-Touch - Page 2

zlantz · ‎02-06-2017

Currently, we are using the dhcp method for pnp discovery. This has been working well, however, our current deployments depend on having 2 uplinks to our upstream switch. On the uplink switch, we use the first link as a trunk and pass specific vlans through, while the second link we have been using as an access port into our pnp specific vlan, 1400. This works no problem, but we need to remember to go back to our upstream switch and change that access port to match the 1st link and add it to the existing port-channel for redundancy. I would like to eliminate this final step and truly make this zero-touch after install. I found the PNP stacking blog posts by Adam helpful. This one in particular sparked my interest: Network Automation with Plug and Play (PnP) – Part 6.

Using a 6807 as my upstream switch and 2 3650s as by deployment, I used the following config and was unable to get this deployment to discover:

6807:

pnp start-up vlan 1400

int po77

description APIC-EM-TEST

switchport

switchport mode trunk

switchport trunk allowed vlan 1,1000,1400

no port-channel standalone-disable

int gi 2/46-47

description TEST-APIC-EM

switchport

switchport mode trunk

switchport trunk allowed vlan 1,1000,1400

udld port

channel-group 77 mode active

Based on the article linked above, cdp should create vlan 1400 on the deployment and have it pull from DHCP. Then it would assign all active interfaces on the stack to 1400, in this case, 1/1/1 and 2/1/1. Unfortunately, 1400 is never created on the 3650s and discovery never happens. The main 1400 vlan lives on the 6807 and has a helper that points at an MS DHCP. Option 43 lives there and has been working with our current deployment standard. Just looking to see if I misunderstood something or misconfigured something.

Thanks,

Zak

zlantz · ‎02-10-2017

No problem. I want to see this product succeed and the more I use it and learn its functions the easier it makes my job.

Ok, that makes sense. So we use an SVI for vlan 1000 for our management on all our devices and with that we just set the IPs statically on that SVI. I also include this in my deployment template for PNP. The problem with using that vlan for PNP and management is that we didn't necessarily want to include that vlan in DHCP at all, but we would have to in order to do pnp with DHCP.

Ok I'll let you know how that works out when I do the upgrade to the 6k next week.

aradford · ‎02-10-2017

ok. you have two options:

1) reserve a small part of the address space for DHCP (just to PnP), then switch to static IP when you provision. This will work fine, and is quite common.

2) Switch to another vlan (e.g. 1000) when you provision with a static IP. This is a little more tricky as you need to make sure the vlan gets created. depending on the switch platform, this may require some EEM help.

Adam

zlantz · ‎02-15-2017

I think what we will end up doing is sticking with our current 1400 vlan and seeing how far that takes us. We already have it in place and the very small amount of 3560CX that we have can be PNP in my test environment prior to racking them. We have maybe a dozen of them.

I did run into an issue late last week while testing this. I can no longer provision a certain switch that I had been using quite a bit for testing. It was getting pnp startup and making the vlan no problem, but pnp trace shows that it connects and doesn't get anything, like I didn't have a rule in place. I double checked the serial and my rule and it all looked fine. Tried a different switch with the same configuration just different serial # and it worked fine. I did a few API calls and even though I can see the rule in the GUI, the calls return no rule for that serial #. I even tried to put the rule in with a PUT and that did not work either. Seems like there may be a limit to how many times you can use the same serial number? Or is there a way to clear previous entries for that serial #? I also tried putting a rule in a completely different project and this still did not work.

aradford · ‎02-15-2017

Hi Zak,

there is no limit to the number of times you can PnP a device.

How well did you clean it? Did you remove all the old certificates etc, or just "wr er".

If you remove the device from a rule, does it come up as unclaimed (which it should)?

Adam

zlantz · ‎02-15-2017

It does not show up under unplanned devices for me to claim, but the pnp start up vlan works and from the pnp trace it looks like it hits the controller fine, but just never gets anything. I'll get the pnp trace for you to verify.

I use the following script every time I wipe a switch for testing:

configure terminal

crypto key zeroize

yes

!

end

test crypto pki trustpool reset

delete nvram:*.cer

delete stby-nvram:*.cer

write memory

write erase

delete flash:vlan.dat

erase nvram:

Reload

aradford · ‎02-15-2017

that looks good.

can you share the API calls you made and the screen shot of the device in the UI please?

Just to be clear:

The API call you need to verify the rule is created is (where d168aa1a-bf61-46c9-b5d6-7ae4e27c48c8 is your projectID... yours will be different).

https://adam-iwan/api/v1/pnp-project/d168aa1a-bf61-46c9-b5d6-7ae4e27c48c8/device?offset=1&limit=10

The only time an entry is created in /pnp-device is when it is provisioned

https://adam-iwan/api/v1/pnp-device?matchDeviceState=true&offset=1&limit=10

Adam

zlantz · ‎02-17-2017

Here is what I had done before and what I just did again:

I created the rule in the GUI:

Then I tried to see if I could see it by serial number:

Above, both times I did this, nothing would get returned when I search specifically for that serial #

However, this time, I was able to see it in the project by querying with the project ID:

However, this rule never progresses out of pending, but the ports are advertising the correct pnp startup vlan:

APP TLV (Gig 2/44), Configured tlv type: 4099, value: 1

APP TLV (Gig 2/44), Configured tlv type: 4103, value: 44

APP TLV (Gig 2/45), Configured tlv type: 4099, value: 1

APP TLV (Gig 2/45), Configured tlv type: 4103, value: 44

And the switch is seen in CDP neighbors:

Switch Gig 2/45 157 S I WS-C3650- Gig 1/1/2

Switch Gig 2/44 132 S I WS-C3650- Gig 1/1/1

VLAN 44 is created successfully on the switch and here is the pnp trace:

Switch#sh pnp trace

[02/17/17 13:30:52.100 UTC 1 399] Info: Startup config does not exists

[02/17/17 13:30:52.101 UTC 2 399] start_pnpa_discovery: PnP Discovery trial number[1]

[02/17/17 13:30:52.101 UTC 3 399] start_pnpa_discovery: Initiating PnP discovery manager

[02/17/17 13:30:52.101 UTC 4 399] pnpa_discovery_autoinstall_pid_create: waiting for autoinstall

[02/17/17 13:31:45.468 UTC 5 399] pnpa_discovery_autoinstall_pid_create:Received autoinstall complete status

[02/17/17 13:31:50.462 UTC 6 399] pnpa_autonomic_discovery: Starting autonomic discovery

[02/17/17 13:31:50.462 UTC 7 399] pnpa_autonomic_discovery: Starting autonomic discovery

[02/17/17 13:31:55.463 UTC 8 399] pnpa_disc_dhcp_option_43: op43 strict protocol: Yes, must secure: No

[02/17/17 13:31:55.463 UTC 9 399] pnpa_disc_dhcp_option_43: op43 profile pnp-zero-touch

[02/17/17 13:31:55.463 UTC A 399] pnpa_disc_dhcp_option_43: op43 ipaddr 10.255.72.116

[02/17/17 13:31:55.463 UTC B 399] pnpa_disc_dhcp_option_43: op43 transport 1

[02/17/17 13:31:55.463 UTC C 399] pnpa_disc_dhcp_option_43: transport http

[02/17/17 13:31:55.463 UTC D 399] pnpa_validate_port_type: Port is 80

[02/17/17 13:31:55.463 UTC E 399] pnpa_disc_dhcp_option_43: op43 port 80

[02/17/17 13:31:55.463 UTC F 399] pnpa_validate_ip_type: op43 iptype ipv4

[02/17/17 13:31:55.464 UTC 10 399] pnp_httpc_register: PnP httpc registered

[02/17/17 13:31:55.464 UTC 11 399] get_pnp_work_req_url: Port is 80

[02/17/17 13:31:55.464 UTC 12 399] pnp_httpc_send_get: url http://10.255.72.116:80/pnp/HELLO

[02/17/17 13:31:55.464 UTC 13 399] pnp_httpc_send_get: HTTP SEND SUCCESS

[02/17/17 13:31:55.479 UTC 14 312] pnp_http_resp_data_alloc: PnP response data alloc 4096 bytes

[02/17/17 13:31:55.479 UTC 15 312] pnp_resp_data: request status Response data recieved, successfully

[02/17/17 13:31:55.479 UTC 16 312] pnp_resp_data: DATA STARTS HERE

[02/17/17 13:31:55.479 UTC 17 312] pnp_resp_data: DATA ENDS HERE

[02/17/17 13:31:55.479 UTC 18 312] pnp_resp_data: Status of this transaction is 200

[02/17/17 13:31:55.479 UTC 19 312] pnp_resp_data: Length of data handed over 21

[02/17/17 13:31:55.479 UTC 1A 312] pnp_resp_data: session id      : 3

[02/17/17 13:31:55.480 UTC 1B 312] pnp_resp_data: transaction id : 1

[02/17/17 13:31:55.480 UTC 1C 312] pnp_resp_data: status_code      : 200

[02/17/17 13:31:55.480 UTC 1D 312] pnp_resp_data: status_string    : OK

[02/17/17 13:31:55.480 UTC 1E 312] pnp_resp_data: content_type    : application/json;charset=UTF-8

[02/17/17 13:31:55.480 UTC 1F 312] pnp_resp_data: content_encoding :

[02/17/17 13:31:55.480 UTC 20 312] pnp_resp_data: content_length : 21

[02/17/17 13:31:55.480 UTC 21 312] pnp_resp_data: Location        :

[02/17/17 13:31:55.480 UTC 22 312] pnp_resp_data: Server          : Jetty(9.0.z-SNAPSHOT)

[02/17/17 13:31:55.481 UTC 23 312] pnp_resp_data: Data has not been cached

[02/17/17 13:31:55.481 UTC 24 312] pnp_http_resp_data_free: pnp response data freed

[02/17/17 13:31:55.481 UTC 25 399] pnp_httpc_send_get: HTTP send success()

[02/17/17 13:31:55.481 UTC 26 399] send_work_req: HTTP SEND GET REQUEST SUCCESS

[02/17/17 13:31:55.481 UTC 27 399] HA registry indicates presence of standby

[02/17/17 13:31:55.481 UTC 28 399] HA, config safe check [NOT OK], for configuring[try:0]

[02/17/17 13:31:55.481 UTC 29 399] HA, config safe check [NOT OK], for configuring[try:1]

[02/17/17 13:31:55.482 UTC 2A 399] pnpa_dhcp_discovery:PnP profile config unsuccessful

[02/17/17 13:31:55.488 UTC 2B 399] pnpa_disc_dhcp_option_43: op43 strict protocol: Yes, must secure: No

[02/17/17 13:31:55.488 UTC 2C 399] pnpa_disc_dhcp_option_43: op43 profile pnp-zero-touch

[02/17/17 13:31:55.488 UTC 2D 399] pnpa_disc_dhcp_option_43: op43 ipaddr 10.255.72.116

[02/17/17 13:31:55.488 UTC 2E 399] pnpa_disc_dhcp_option_43: op43 transport 1

[02/17/17 13:31:55.488 UTC 2F 399] pnpa_disc_dhcp_option_43: transport http

[02/17/17 13:31:55.488 UTC 30 399] pnpa_validate_port_type: Port is 80

[02/17/17 13:31:55.488 UTC 31 399] pnpa_disc_dhcp_option_43: op43 port 80

[02/17/17 13:31:55.488 UTC 32 399] pnpa_validate_ip_type: op43 iptype ipv4

[02/17/17 13:31:55.488 UTC 33 399] pnp_httpc_register: PnP Already registered with httpc

[02/17/17 13:31:55.488 UTC 34 399] get_pnp_work_req_url: Port is 80

[02/17/17 13:31:55.488 UTC 35 399] pnp_httpc_send_get: url http://10.255.72.116:80/pnp/HELLO

[02/17/17 13:31:55.489 UTC 36 399] pnp_httpc_send_get: HTTP SEND SUCCESS

[02/17/17 13:31:55.494 UTC 37 312] pnp_http_resp_data_alloc: PnP response data alloc 4096 bytes

[02/17/17 13:31:55.494 UTC 38 312] pnp_resp_data: request status Response data recieved, successfully

[02/17/17 13:31:55.494 UTC 39 312] pnp_resp_data: DATA STARTS HERE

[02/17/17 13:31:55.494 UTC 3A 312] pnp_resp_data: DATA ENDS HERE

[02/17/17 13:31:55.494 UTC 3B 312] pnp_resp_data: Status of this transaction is 200

[02/17/17 13:31:55.494 UTC 3C 312] pnp_resp_data: Length of data handed over 21

[02/17/17 13:31:55.494 UTC 3D 312] pnp_resp_data: session id      : 3

[02/17/17 13:31:55.494 UTC 3E 312] pnp_resp_data: transaction id : 2

[02/17/17 13:31:55.494 UTC 3F 312] pnp_resp_data: status_code      : 200

[02/17/17 13:31:55.494 UTC 40 312] pnp_resp_data: status_string    : OK

[02/17/17 13:31:55.494 UTC 41 312] pnp_resp_data: content_type    : application/json;charset=UTF-8

[02/17/17 13:31:55.494 UTC 42 312] pnp_resp_data: content_encoding :

[02/17/17 13:31:55.494 UTC 43 312] pnp_resp_data: content_length : 21

[02/17/17 13:31:55.494 UTC 44 312] pnp_resp_data: Location        :

[02/17/17 13:31:55.494 UTC 45 312] pnp_resp_data: Server          : Jetty(9.0.z-SNAPSHOT)

[02/17/17 13:31:55.494 UTC 46 312] pnp_resp_data: Data has not been cached

[02/17/17 13:31:55.494 UTC 47 312] pnp_http_resp_data_free: pnp response data freed

[02/17/17 13:31:55.494 UTC 48 399] pnp_httpc_send_get: HTTP send success()

[02/17/17 13:31:55.494 UTC 49 399] send_work_req: HTTP SEND GET REQUEST SUCCESS

[02/17/17 13:31:55.494 UTC 4A 399] HA registry indicates presence of standby

[02/17/17 13:31:55.494 UTC 4B 399] HA, config safe check [NOT OK], for configuring[try:0]

[02/17/17 13:31:55.494 UTC 4C 399] HA, config safe check [NOT OK], for configuring[try:1]

[02/17/17 13:31:55.494 UTC 4D 399] pnpa_dhcp_discovery:PnP profile config unsuccessful

[02/17/17 13:32:00.490 UTC 4E 399] pnpa_dns_discovery: Starting PnP DNS discovery

[02/17/17 13:32:00.491 UTC 4F 399] pnpa_disc_dhcp_dns: domain name University.liberty.edu on interface Vlan44

[02/17/17 13:32:09.493 UTC 50 399] pnpa_disc_dhcp_dns:Host name address resolution failed

[02/17/17 13:32:09.493 UTC 51 399] pnpa_dns_discovery:DNS discovery not successful

[02/17/17 13:32:09.493 UTC 52 399] pnpa_dns_discovery: Starting PnP DNS discovery

[02/17/17 13:32:09.493 UTC 53 399] pnpa_disc_dhcp_dns: domain name University.liberty.edu on interface Vlan44

[02/17/17 13:32:09.493 UTC 54 399] pnpa_disc_dhcp_dns:Host name address resolution failed

[02/17/17 13:32:09.493 UTC 55 399] pnpa_dns_discovery:DNS discovery not successful

[02/17/17 13:32:14.492 UTC 56 399] pnpa_cco_discovery: starting CCO discovery

[02/17/17 13:32:23.493 UTC 57 399] pnpa_cco_discovery: Hostname resolution failed

[02/17/17 13:32:23.493 UTC 58 399] pnpa_cco_discovery: starting CCO discovery

[02/17/17 13:32:23.493 UTC 59 399] pnpa_cco_discovery: Hostname resolution failed

[02/17/17 13:32:28.493 UTC 5A 399] pnpa_discovery_manager: Starting NAPP

[02/17/17 13:32:28.493 UTC 5B 399] pnpa_napp_discovery: starting NAPP discovery

[02/17/17 13:32:28.493 UTC 5C 399] set_intf_state_for_napp: Setting interface state for NAPP

[02/17/17 13:32:28.493 UTC 5D 399] pnp_pxc_recv_init: enter

[02/17/17 13:32:28.494 UTC 5E 399] pnp_pxc_send_init: pnp_pxc_xmt_process created. listener_pid=284

[02/17/17 13:32:28.494 UTC 5F 175] pnp_pxc_rcv_process: Begin

[02/17/17 13:32:28.494 UTC 60 175] pnp_pxd_new_workspace: New packet workspace 0x3B63F1EC

[02/17/17 13:32:28.494 UTC 61 175] pnp_pxc_create_recv_socket: enter

[02/17/17 13:32:28.499 UTC 62 175] pnp_pxc_create_recv_socket: done

[02/17/17 13:32:28.499 UTC 63 175] pnp_pxc_rcv_loop: enter

[02/17/17 13:32:28.500 UTC 64 284] pnp_pxc_xmt_process: Entering sending loop.

[02/17/17 13:32:28.563 UTC 65 284] pnp_pxc_xmt_loop: str_port_len = 4

[02/17/17 13:32:28.563 UTC 66 284] pnp_pxc_xmt_loop: client_port_no = 1029

[02/17/17 13:32:28.563 UTC 67 284] pnp_pxc_xmt_loop: str_client_ip_len = 7

[02/17/17 13:32:28.563 UTC 68 284] pnp_pxc_xmt_loop: pxy_client_ip = 0.0.0.0

[02/17/17 13:32:28.563 UTC 69 284] pnp_pxc_xmt_loop: String hello = OP1140:PID:WS-C3650-48PS,VID:V04,SN:FDO2038Z03POP1213:WS-C3650-48PSOP133:1.0OP144:1029OP157:0.0.0.0

[02/17/17 13:32:28.569 UTC 6A 284] pnp_pxc_xmt_loop: sendto returns 99

[02/17/17 13:32:28.569 UTC 6B 284] pnp_pxc_xmt_loop: PROXY CLIENT: sent data = OP1140:PID:WS-C3650-48PS,VID:V04,SN:FDO2038Z03POP1213:WS-C3650-48PSOP133:1.0OP144:1029OP157:0.0.0.0

[02/17/17 13:33:28.571 UTC 6C 284] pnp_pxc_rcv_cleanup: setting FAIL discovery

[02/17/17 13:33:28.571 UTC 6D 284] pnp_pxc_xmt_loop - Killing proxy client sending and receiving processes

[02/17/17 13:33:28.572 UTC 6E 284] send_timeout_response: Timeout status response sent for client 'PID:WS-C3650-48PS,VID:V04,SN:FDO2038Z03P' : OPRESPSTATUS:OP1140:PID:WS-C3650-48PS,VID:V04,SN:FDO2038Z03POP123:1.0OP13OPTRANSS19:TRANSACTION TIMEOUTOPSTATUS33:PROXY CLIENT RETRY COUNT EXCEEDEDOPERRMSG75:Proxy client retry count exceeded. Proxy client process is no longer active

[02/17/17 13:33:28.572 UTC 6F 284] create_pnp_pxc_timeout_response_socket: Socket open succeeded

[02/17/17 13:33:28.578 UTC 70 284] destroy_pnp_pxc_timeout_response_socket: Closing socket

[02/17/17 13:33:28.578 UTC 71 284] pnp_pxc_xmt_process: Sending loop exited.

[02/17/17 13:33:28.578 UTC 72 284] pnp_pxc_xmt_cleanup: enter

[02/17/17 13:33:28.578 UTC 73 284] pnp_pxc_xmt_cleanup: Closing socket

[02/17/17 13:33:28.578 UTC 74 284] pnp_pxc_xmt_cleanup: Killing process

[02/17/17 13:33:28.579 UTC 75 399] pnp_wait_on_napp:Received NAPP complete status

[02/17/17 13:33:33.584 UTC 76 399] Info: PnP profile configuration was unsuccessful

[02/17/17 13:33:33.585 UTC 77 399] start_pnpa_discovery: PnP discovery process failed

[02/17/17 13:33:33.585 UTC 78 399] start_pnpa_discovery: User has entered console.Not retrying PnP.

120 entries printed

Vlan is created during the process:

44 VLAN0044 active

SVI for 44 is created as well:

interface Vlan44

ip address dhcp

And the active interfaces are set to vlan 44

interface GigabitEthernet1/1/1

switchport access vlan 44

macro description CISCO_SMI_EVENT

interface GigabitEthernet1/1/2

switchport access vlan 44

macro description CISCO_SMI_EVENT

aradford · ‎02-17-2017

Hi Zak,

what you are seeing with the API is correct. There is not entry in /pnp-device until the device start provisioning. You will see this also if you reset an existing rule. The output from /pnp-device will be [].

What mechanism are you using for discovery?

[02/17/17 13:32:09.493 UTC 53 399] pnpa_disc_dhcp_dns: domain name University.liberty.edu on interface Vlan44

[02/17/17 13:32:09.493 UTC 54 399] pnpa_disc_dhcp_dns:Host name address resolution failed

[02/17/17 13:32:09.493 UTC 55 399] pnpa_dns_discovery:DNS discovery not successful

02/17/17 13:33:33.584 UTC 76 399] Info: PnP profile configuration was unsuccessful

[02/17/17 13:33:33.585 UTC 77 399] start_pnpa_discovery: PnP discovery process failed

It looks like the DNS resolution failed (assuming that is how you are doing discovery)?

Hence there was no way to discover the controller.

Adam

zlantz · ‎02-17-2017

I didn't know that about that API. Thanks for the info

I am using DHCP with option 43 and it works within the same vtp and vlan with other switches

Actually, so there is another bit of the story I believe you should know that might relate. A long time ago when we first got option 43 set up, I was finding that my deployments would connect to some external cisco site and pulling down a config that would just change the hostname to NS1. When I would do a pnp trace, it was resolving a cisco url to grab this random config. So to remedy this, I changed my DNS servers in that scope to point at the APIC-EM controller. I also did this for the tftp option within that scope. I would just delete those options from that scope, but they are globally set and therefore can not be deleted out of one scope without removing the global option all together.

aradford · ‎02-17-2017

ok. For some reason the DHCP discovery is not working. What version of code is the switch running?

zlantz · ‎02-17-2017

Check my edit above. Might help with that additional info.
Switches are running 3.7.3

aradford · ‎02-17-2017

There is an issue in 3.7.3 and below that makes DHCP problematic with NV1.

Release Notes for Cisco Network Plug and Play, Release 1.3x - Cisco

Just to confirm this, can you upgrade this switch to 3.7.4 and see if that resolves the issue?

Adam

zlantz · ‎02-17-2017

On one hand, I would like 3.7.4 to fix this issue and it would make sense that I've never seen this issue before because we just got pnp startup vlan working. On the other hand, we would like to stay on 3.7.3 because that has been our most stable build within the 3.7.x train and we are not ready for 16 code

I'll let you know if this fixes it. I am upgrading the switches now.

zlantz · ‎02-17-2017

*sigh, it works with 3.7.4 and startup vlan. At least I know now!

aradford · ‎02-17-2017

Thanks for confirming. That issue is only intermittent which is why it took a while to track down.

Adam