topic Re: Can't get switch to initate PnP - "Info: Startup config exists" in Controllers

Can't get switch to initate PnP - "Info: Startup config exists"

BOB NIEDERMAN — Fri, 01 Mar 2019 12:35:32 GMT

I've got a 3850 at 3.6.3E. I use this process to render the switch to factory defaults:

configure terminal

crypto key zeroize

no crypto pki certificate pool

no pnp profile pnp-zero-touch

end

delete nvram:*.cer

( it never finds this file )

delete stby-nvram:*.cer

(if the device has stack members, which it doesn't it's just a single switch)

write erase

reload

as shown here: Solution Guide for Cisco Network Plug and Play - Cisco

I reload the switch. The 3850 switch never makes a DHCP discovery packet according to my packet traces. I've verified that the DHCP process would work if the switch would just start it, by configuring 3850 port connected to the 3650 for "ip address dhcp" and then shut/no shut that interface.

In fact, 'sho pnp trace' shows this:

apic-test-3850#sh pnp trace

[01/18/17 23:39:04.219 UTC 1 442] Info: Startup config exists

[01/18/17 23:39:04.219 UTC 2 442] Info: HW reset status is: 0

[01/18/17 23:39:04.219 UTC 3 442] start_pnpa_discovery: Discovery did not start due to presence of startup config or is a standby

3 entries printed

Re: Can't get switch to initate PnP - "Info: Startup config exists"

zlantz — Thu, 19 Jan 2017 15:08:58 GMT

This is my version of the wipe script that I use and has been working well for me.

configure terminal

crypto key zeroize

yes

end

test crypto pki trustpool reset

delete nvram:*.cer

delete stby-nvram:*.cer

write memory

write erase

delete flash:vlan.dat

erase nvram:

Reload

I leave the spacing there for it to confirm the steps that are taken

Re: Can't get switch to initate PnP - "Info: Startup config exists"

ngoldwat — Thu, 19 Jan 2017 15:09:05 GMT

Can you verify that CCP is not present anywhere in the file system?

Re: Can't get switch to initate PnP - "Info: Startup config exists"

BOB NIEDERMAN — Thu, 19 Jan 2017 16:02:13 GMT

Hi, Nick,

Can you be more specific as to CCP in file system? Is that CPP? I've looked for both:

apic-test-3850#

apic-test-3850#dir /all /recursive all-filesystems | in ccp

525 -r-- 180892 <no date> [/tmp/sw/rp/0/0/rp_infra/mount/usr/binos/lib/libffm_fed_api_wccp]

526 -r-- 19904 <no date> [/tmp/sw/rp/0/0/rp_infra/mount/usr/binos/lib/libffm_fed_api_wccp]

529 -r-- 119788 <no date> [/tmp/sw/rp/0/0/rp_infra/mount/usr/binos/lib/libffm_wccp_msg.so]

530 -r-- 12772 <no date> [/tmp/sw/rp/0/0/rp_infra/mount/usr/binos/lib/libffm_wccp_msg.so]

637 -r-- 797260 <no date> [/tmp/sw/rp/0/0/rp_infra/mount/usr/binos/lib/libffm_wccp_tbl.so]

638 -r-- 85272 <no date> [/tmp/sw/rp/0/0/rp_infra/mount/usr/binos/lib/libffm_wccp_tbl.so]

527 -r-- 435324 <no date> [/tmp/sw/rp/0/0/rp_infra/mount/usr/binos/lib/libffm_wccp_types.s]

528 -r-- 46284 <no date> [/tmp/sw/rp/0/0/rp_infra/mount/usr/binos/lib/libffm_wccp_types.s]

apic-test-3850#dir /all /recursive all-filesystems | in cpp

6637 -r-- 395 Jan 18 2017 23:35:31 +00:00 dhcppool.odm

6635 -r-- 405 Jan 18 2017 23:35:31 +00:00 dhcppoolv6.odm

apic-test-3850#dir /all /recursive all-filesystems | in CCP

apic-test-3850#dir /all /recursive all-filesystems | in CPP

apic-test-3850#

Re: Can't get switch to initate PnP - "Info: Startup config exists"

ngoldwat — Thu, 19 Jan 2017 16:10:40 GMT

It looks like it is not present. I am aware of an issue when it is and

the workaround is to delete it.

Nick

Re: Can't get switch to initate PnP - "Info: Startup config exists"

BOB NIEDERMAN — Thu, 19 Jan 2017 17:58:44 GMT

Using Zak's commands I've gotten slightly further.

Now sho pnp trace gives this:

[01/19/17 16:25:44.715 UTC 1 442] Info: Startup config does not exists

[01/19/17 16:25:44.715 UTC 2 442] start_pnpa_discovery: PnP Discovery trial number[1]

[01/19/17 16:25:44.715 UTC 3 442] start_pnpa_discovery: Initiating PnP discovery manager

[01/19/17 16:25:44.715 UTC 4 442] pnpa_discovery_autoinstall_pid_create: waiting for autoinstall

[01/19/17 16:26:10.216 UTC 5 442] pnpa_discovery_autoinstall_pid_create:Received autoinstall complete status

[01/19/17 16:26:15.216 UTC 6 442] pnpa_autonomic_discovery: Starting autonomic discovery

[01/19/17 16:26:15.216 UTC 7 442] pnpa_autonomic_discovery: Starting autonomic discovery

[01/19/17 16:26:20.214 UTC 8 442] pnpa_disc_dhcp_option_43: no op43 profile

[01/19/17 16:26:20.214 UTC 9 442] pnpa_disc_dhcp_option_43: no op43 profile

[01/19/17 16:26:25.216 UTC A 442] pnpa_dns_discovery: Onto DNS discovery. DHCP discovery failed

[01/19/17 16:26:25.216 UTC B 442] pnpa_disc_dhcp_dns: failed to obtain the domain name

[01/19/17 16:26:25.216 UTC C 442] pnpa_dns_discovery: Onto DNS discovery. DHCP discovery failed

[01/19/17 16:26:25.217 UTC D 442] pnpa_disc_dhcp_dns: failed to obtain the domain name

[01/19/17 16:26:30.217 UTC E 442] pnpa_cco_discovery: starting CCO discovery

[01/19/17 16:26:39.214 UTC F 442] pnpa_cco_discovery: Hostname resolution failed

[01/19/17 16:26:39.214 UTC 10 442] pnpa_cco_discovery: starting CCO discovery

[01/19/17 16:26:39.214 UTC 11 442] pnpa_cco_discovery: Hostname resolution failed

[01/19/17 16:26:44.217 UTC 12 442] pnpa_discovery_manager: Starting NAPP

[01/19/17 16:26:44.217 UTC 13 442] pnpa_napp_discovery: starting NAPP discovery

[01/19/17 16:26:44.218 UTC 14 442] set_intf_state_for_napp: Setting interface state for NAPP

[01/19/17 16:26:44.218 UTC 15 442] pnp_pxc_recv_init: enter

[01/19/17 16:26:44.218 UTC 16 442] pnp_pxc_send_init: pnp_pxc_xmt_process created. listener_pid=284

[01/19/17 16:26:44.219 UTC 17 282] pnp_pxc_rcv_process: Begin

[01/19/17 16:26:44.219 UTC 18 282] pnp_pxd_new_workspace: New packet workspace 0x3A8198A0

[01/19/17 16:26:44.219 UTC 19 282] pnp_pxc_create_recv_socket: enter

[01/19/17 16:26:44.224 UTC 1A 282] pnp_pxc_create_recv_socket: done

[01/19/17 16:26:44.224 UTC 1B 282] pnp_pxc_rcv_loop: enter

[01/19/17 16:26:44.224 UTC 1C 284] pnp_pxc_xmt_process: Entering sending loop.

[01/19/17 16:26:44.277 UTC 1D 284] pnp_pxc_xmt_loop: str_port_len = 4

[01/19/17 16:26:44.277 UTC 1E 284] pnp_pxc_xmt_loop: client_port_no = 1026

[01/19/17 16:26:44.277 UTC 1F 284] pnp_pxc_xmt_loop: str_client_ip_len = 7

[01/19/17 16:26:44.277 UTC 20 284] pnp_pxc_xmt_loop: pxy_client_ip = 0.0.0.0

[01/19/17 16:26:44.278 UTC 21 284] pnp_pxc_xmt_loop: String hello = OP1138:PID:WS-C3850-48U,VID:E0,SN:FOC1824X0PBOP1212:WS-C3850-48UOP133:1.0OP144:1026OP157:0.0.0.0

[01/19/17 16:26:44.283 UTC 22 284] pnp_pxc_xmt_loop: sendto returns 96

[01/19/17 16:26:44.283 UTC 23 284] pnp_pxc_xmt_loop: PROXY CLIENT: sent data = OP1138:PID:WS-C3850-48U,VID:E0,SN:FOC1824X0PBOP1212:WS-C3850-48UOP133:1.0OP144:1026OP157:0.0.0.0

[01/19/17 16:27:44.274 UTC 24 284] pnpa_discovery_reset_intf_state: Resetting interface state for NAPP

[01/19/17 16:27:44.274 UTC 25 284] pnp_pxc_rcv_cleanup: setting FAIL discovery

[01/19/17 16:27:44.274 UTC 26 284] pnp_pxc_xmt_loop - Killing proxy client sending and receiving processes

[01/19/17 16:27:44.274 UTC 27 284] send_timeout_response: Timeout status response sent for client 'PID:WS-C3850-48U,VID:E0,SN:FOC1824X0PB' : OPRESPSTATUS:OP1138:PID:WS-C3850-48U,VID:E0,SN:FOC1824X0PBOP123:1.0OP13OPTRANSS19:TRANSACTION TIMEOUTOPSTATUS33:PROXY CLIENT RETRY COUNT EXCEEDEDOPERRMSG75:Proxy client retry count exceeded. Proxy client process is no longer active

[01/19/17 16:27:44.275 UTC 28 284] create_pnp_pxc_timeout_response_socket: Socket open succeeded

[01/19/17 16:27:44.280 UTC 29 284] destroy_pnp_pxc_timeout_response_socket: Closing socket

[01/19/17 16:27:44.280 UTC 2A 284] pnp_pxc_xmt_process: Sending loop exited.

[01/19/17 16:27:44.280 UTC 2B 284] pnp_pxc_xmt_cleanup: enter

[01/19/17 16:27:44.280 UTC 2C 284] pnp_pxc_xmt_cleanup: Closing socket

[01/19/17 16:27:44.281 UTC 2D 284] pnp_pxc_xmt_cleanup: Killing process

[01/19/17 16:27:44.281 UTC 2E 442] pnp_wait_on_napp:Received NAPP complete status

[01/19/17 16:27:44.281 UTC 2F 442] Info: PnP profile configuration was unsuccessful

[01/19/17 16:27:44.281 UTC 30 442] start_pnpa_discovery: PnP discovery process failed

[01/19/17 16:27:44.281 UTC 31 442] start_pnpa_discovery: User had entered console.Not retrying PnP.

49 entries printed

The 3850 is up, the upstream 3650 has dhcp defined:

ip dhcp pool 192.168.1.1_24

network 192.168.1.0 255.255.255.0

default-router 192.168.1.1

option 60 ascii "ciscopnp"

option 43 ascii "5A1N;B2;K4;I10.30.171.31;J80;Z10.30.171.1"

When I do a "normal" dhcp - eg the connected port on the 3850 is defined with "ip address dhcp" it seems to work correctly:

*Jan 19 17:36:40.365: DHCP: rcvd pkt source: 192.168.1.1, destination: 255.255.255.255
*Jan 19 17:36:40.365:    UDP sport: 43, dport: 44, length: 331
*Jan 19 17:36:40.365:    DHCP op: 2, htype: 1, hlen: 6, hops: 0
*Jan 19 17:36:40.365:    DHCP server identifier: 192.168.1.1
*Jan 19 17:36:40.365:         xid: 777, secs: 0, flags: 8000
*Jan 19 17:36:40.365:         client: 0.0.0.0, your: 192.168.1.7
*Jan 19 17:36:40.365:         srvr:   0.0.0.0, gw: 0.0.0.0
*Jan 19 17:36:40.365:         options block length: 83

*Jan 19 17:36:40.365: DHCP Offer Message   Offered Address: 192.168.1.7
*Jan 19 17:36:40.365: DHCP: Lease Seconds: 86400    Renewal secs: 43200    Rebind secs:   75600
*Jan 19 17:36:40.365: DHCP: Server ID Option: 192.168.1.1
*Jan 19 17:36:40.366: DHCP: offer received from 192.168.1.1
*Jan 19 17:36:40.366: DHCP: SRequest attempt # 1 for entry:
*Jan 19 17:36:40.366: Temp IP addr: 192.168.1.7 for peer on Interface: GigabitEthernet1/0/2
*Jan 19 17:36:40.366: Temp sub net mask: 255.255.255.0
*Jan 19 17:36:40.366:    DHCP Lease server: 192.168.1.1, state: 4 Requesting
*Jan 19 17:36:40.366:    DHCP transaction id: 777
*Jan 19 17:36:40.366:    Lease: 86400 secs, Renewal: 0 secs, Rebind: 0 secs
*Jan 19 17:36:40.366:    Next timer fires after: 00:00:03
*Jan 19 17:36:40.366:    Retry count: 1   Client-ID: cisco-0008.2f92.6e56-Gi1/0/2
*Jan 19 17:36:40.366:    Client-ID hex dump: 636973636F2D303030382E326639322E
*Jan 19 17:36:40.367:                        366535362D4769312F302F32
*Jan 19 17:36:40.367: DHCP: SRequest- Server ID option: 192.168.1.1
*Jan 19 17:36:40.367: DHCP: SRequest- Requested IP addr option: 192.168.1.7
*Jan 19 17:36:40.367: DHCP: SRequest placed lease len option: 86400
*Jan 19 17:36:40.367: DHCP: SRequest placed class-id option: 636973636F706E70
*Jan 19 17:36:40.367: DHCP: SRequest: 317 bytes
*Jan 19 17:36:40.367: DHCP: SRequest: 317 bytes
*Jan 19 17:36:40.367:             B'cast on GigabitEthernet1/0/2 interface from 0.0.0.0
*Jan 19 17:36:40.373: DHCP: Received a BOOTREP pkt
*Jan 19 17:36:40.373: DHCP: Scan: Message type: DHCP Ack
*Jan 19 17:36:40.373: DHCP: Scan: Server ID Option: 192.168.1.1 = C0A80101
*Jan 19 17:36:40.373: DHCP: Scan: Lease Time: 86400
*Jan 19 17:36:40.373: DHCP: Scan: Renewal time: 43200
*Jan 19 17:36:40.373: DHCP: Scan: Rebind time: 75600
*Jan 19 17:36:40.373: DHCP: Scan: Subnet Address Option: 255.255.255.0
*Jan 19 17:36:40.373: DHCP: Scan: Router Option: 192.168.1.1
*Jan 19 17:36:40.373: DHCP: Scan: Vendor specific option 43: <snip>
*Jan 19 17:36:40.374: DHCP: rcvd pkt source: 192.168.1.1, destination: 255.255.255.255
*Jan 19 17:36:40.374:    UDP sport: 43, dport: 44, length: 331
*Jan 19 17:36:40.374:    DHCP op: 2, htype: 1, hlen: 6, hops: 0
*Jan 19 17:36:40.374:    DHCP server identifier: 192.168.1.1
*Jan 19 17:36:40.374:         xid: 777, secs: 0, flags: 8000
*Jan 19 17:36:40.374:         client: 0.0.0.0, your: 192.168.1.7
*Jan 19 17:36:40.374:         srvr:   0.0.0.0, gw: 0.0.0.0
*Jan 19 17:36:40.374:         options block length: 83

*Jan 19 17:36:40.374: DHCP Ack Message
*Jan 19 17:36:40.374: DHCP: Lease Seconds: 86400    Renewal secs: 43200    Rebind secs:   75600
*Jan 19 17:36:40.374: DHCP: Server ID Option: 192.168.1.1
*Jan 19 17:36:43.375: DHCP: Releasing ipl options:
*Jan 19 17:36:43.376: DHCP: Applying DHCP options:
*Jan 19 17:36:43.376:   Setting default_gateway to 192.168.1.1
*Jan 19 17:36:43.376:   Adding default route 192.168.1.1
*Jan 19 17:36:43.376: DHCPC: Notifying other components about option 43
*Jan 19 17:36:43.376: DHCP Client Pooling: ***Allocated IP address: 192.168.1.7
*Jan 19 17:36:43.425: Allocated IP address = 192.168.1.7 255.255.255.0

Re: Can't get switch to initate PnP - "Info: Startup config exists"

BOB NIEDERMAN — Thu, 19 Jan 2017 18:25:41 GMT

Packet traces for the normal dhcp cyle show this:

Packet trace for the PNP reload shows only an initial release:

Re: Can't get switch to initate PnP - "Info: Startup config exists"

ngoldwat — Thu, 19 Jan 2017 18:36:05 GMT

Try removing 5A1N;B2;K4;I10.30.171.31;J80;Z10.30.171.1

and replacing with 5A1N;B2;K4;I10.30.171.31;J80

Re: Can't get switch to initate PnP - "Info: Startup config exists"

BOB NIEDERMAN — Thu, 19 Jan 2017 19:26:11 GMT

The switch never does a DHCP request, as noted above.

Re: Can't get switch to initate PnP - "Info: Startup config exists"

m.volodko — Mon, 23 Jan 2017 09:13:31 GMT

Hi,

Why your switch has a hostname?

If there is no config it should be "Switch".

Try to issue show startup config prior to reload command.

Misha

Re: Can't get switch to initate PnP - "Info: Startup config exists"

aradford — Tue, 24 Jan 2017 02:14:19 GMT

Is it possible to try 3.6.5 code please. There are a bunch of dhcp PnP issues with 3.6.3.

we have documented in the release notes, but it may not be obvious.

adam

Re: Can't get switch to initate PnP - "Info: Startup config exists"

aradford — Tue, 24 Jan 2017 08:00:09 GMT

I think CCP is for routers. It should not be on switches

Adam

Re: Can't get switch to initate PnP - "Info: Startup config exists"

timoffet — Mon, 20 Feb 2017 22:09:24 GMT

I am having the same issue on a 3650 running 3.6.6E. When I figure out what is going on, I will post the fix here.

Re: Can't get switch to initate PnP - "Info: Startup config exists"

xealo — Thu, 20 Dec 2018 22:55:32 GMT

Hi, I've been battling APIC-PNP with a couple of 3850's that I have. I can't even get a lone 3850 to connect to my PNP server. I've made sure that everything that needs to be is erased on it is. When I run the pnp trace, it tells me that discovery failed for dhcp and dns. I know that things are working as I was able to successfully provision a 9300 just a few weeks back. Nothing on the server or our network has changed. I've even gone to the extent of updating to the latest MD of 3.6.9. That didn't work. So then I updated to Denali. That didn't work either. So as a last resort, I updated the switch to MD 16.6.4. Still nothing. I'm at a loss here. Any help would be greatly appreciate!

ETA: So it looks like it just don't plain work on the 3850's. I tried the same process on the 9300 and it was able to connect to the PNP server without a hitch via DNS discovery. After doing some digging, the only solution is to have a bootstrap telling the management interface that it needs an IP via DHCP and also applies a PNP profile with the server IP address. This can only be delivered via the PnP app and not via USB to the switch. Quite the letdown.

Re: Can't get switch to initate PnP - "Info: Startup config exists"

casanavep — Fri, 31 May 2019 16:48:52 GMT

I have seen this issue exist in multiple flavors of IOS-XE, to include the current recommended. This impact PnP in both APIC-EM and DNA-C. Here's what appears to be a workaround:

Wipe and reload the switch, ensuring you answer no if asked if you would like to save the startup config after issuing the reload command.
After reboot, simply paste the following script and watch the terminal for it to kickoff the PnP process. This may take a couple minutes:

conf t

int vlan 1

no shut

ip address dhcp

! wait for interface VLAN 1 to come up and the switch to pull an IP address, then continue

pnp profile test

transport http ipv4 {IP of APIC-EM or DNA-C} port 80

end

Good luck!

Pete

It may take a few minutes, but nothing else should be needed.