topic Debug PnP process in Controllers

Debug PnP process

ymeloch — Fri, 01 Mar 2019 12:40:24 GMT

Hey Team,

I practiced today the APIC-EM PNP process on a WS-C3650-24TS (16.5.1a) device.

My goal was simple, deploy new configuration on that specific device.

I used DHCP (option 43) to share the controller information and properly configured the APIC-EM with the configuration file and the Project/Device information.

the final status of the provisioning task was Error

However, post the provisioning, I do see that the configuration was deployed and I can successfully ssh the device from a Linux box.

What am i missing?

Thanks much!

APIC-EM History Info

Received response from pnp agent for message correlatorId: CiscoPnP-1.0-6-263-FFA635D628-5 but with error code : ZTD_CMD_ERROR Response String: ERROR:PnP Service Error 1410:Invalid config found

service log:

Device Console Output:

switch: boot flash:cat3k_caa-universalk9.16.05.01a.SPA.bin

Reading full image into memory.......................................................................................................................................................................................................................................................................................................................................................................done

Bundle Image

--------------------------------------

Kernel Address : 0x5342f674

Kernel Size : 0x365eb6/3563190

Initramfs Address : 0x5379552a

Initramfs Size : 0x13bb4ec/20690156

Compression Format: mzip

Bootable image at @ ram:0x5342f674

Bootable image segment 0 address range [0x81100000, 0x81bbfdc0] is in range [0x80180000, 0x90000000].

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

File "flash:cat3k_caa-universalk9.16.05.01a.SPA.bin" uncompressed and installed, entry point: 0x816b7aa0

Loading Linux kernel with entry point 0x816b7aa0 ...

Bootloader: Done loading app on core_mask: 0xf

### Launching Linux Kernel (flags = 0x5)

SELinux: Could not open policy file <= /etc/selinux/targeted/polsmpi: instance 0 lsmpi_area at 0x80000000f9800000 (phys 0x0000000008694000)

lsmpi: instance 2 lsmpi_area at 0x80000000fa000000 (phys 0x00000000086b0000)

lsmpi: instance 3 lsmpi_area at 0x80000000fa800000 (phys 0x00000000086cc000)

lsmpi: instance 4 lsmpi_area at 0x80000000fb000000 (phys 0x00000000086e8000)

lsmpi version 0.2.2

Inst 0 Get kvaddr 0x80011c00f0000000

If this is not emulator, STOP, check ERS for new asic revid

Inst 0 Get kvaddr 0x80011c00f0000000

oobnd:

/auto/mcpbuilds21/release/16.05.01a/BLD-V16_05_01A_FC2/binos/drivers/kernel/obj-mips64_cge7-edison/doppler/oobnd/src/oobnd.c:oobnd_module_init: silent roll checkpoint

/auto/mcpbuilds21/release/16.05.01a/BLD-V16_05_01A_FC2/binos/drivers/kernel/obj-mips64_cge7-edison/doppler/oobnd/src/oobhal.c:oobhal_init_module: silent roll checkpoint

Both links down, not waiting for other switches

Switch number is 1

Restricted Rights Legend

Use, duplication, or disclosure by the Government is

subject to restrictions as set forth in subparagraph

Rights clause at FAR sec. 52.227-19 and subparagraph

Software clause at DFARS sec. 252.227-7013.

cisco Systems, Inc.

170 West Tasman Drive

San Jose, California 95134-1706

Cisco IOS Software [Everest], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.5.1a, RELEASE SOFTWARE (fc2)

Technical Support: http://www.cisco.com/techsupport

Compiled Tue 30-May-17 00:59 by mcpre

licensed under the GNU General Public License ("GPL") Version 2.0. The

software code licensed under GPL Version 2.0 is free software that comes

with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such

GPL code under the terms of GPL Version 2.0. For more details, see the

documentation or "License Notice" file accompanying the IOS-XE software,

or the applicable URL provided on the flyer accompanying the IOS-XE

software.

% failed to initialize nvram

FIPS: Flash Key Check : Begin

FIPS: Flash Key Check : End, Not Found, FIPS Mode Not Enabled

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to

export@cisco.com.

cisco WS-C3650-24TS (MIPS) processor (revision K0) with 858302K/6147K bytes of memory.

Processor board ID FDO2012E02L

2048K bytes of non-volatile configuration memory.

4194304K bytes of physical memory.

250456K bytes of Crash Files at crashinfo:.

1609272K bytes of Flash at flash:.

0K bytes of WebUI ODM Files at webui:.

Base Ethernet MAC Address : 00:3a:7d:63:ba:80

Motherboard Assembly Number : 73-15898-05

Motherboard Serial Number : FDO20120L5H

Model Revision Number : K0

Motherboard Revision Number : B0

Model Number : WS-C3650-24TS

System Serial Number : FDO2012E02L

%INIT: waited 0 seconds for NVRAM to be available

SETUP: new interface Vlan1 placed in "shutdown" state

Press RETURN to get started!

*Aug 17 18:54:26.490: %SMART_LIC-6-AGENT_READY: Smart Agent for Licensing is initialized

*Aug 17 18:54:27.077: %IOSXE_RP_NV-3-NV_ACCESS_FAIL: Initial read of NVRAM contents failed

*Aug 17 18:54:33.254: %NBAR-6-CACHE_SYNC_INFO: Cache synchronization. Initialized.

*Aug 17 18:54:39.578: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan

*Aug 17 18:54:40.455: %LINK-3-UPDOWN: Interface Lsmpi18/0/3, changed state to up

*Aug 17 18:54:40.456: %LINK-3-UPDOWN: Interface EOBC18/0/1, changed state to up

*Aug 17 18:54:40.457: %LINEPROTO-5-UPDOWN: Line protocol on Interface LI-Null0, changed state to up

*Aug 17 18:54:40.459: %LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed state to down

*Aug 17 18:54:40.459: %LINK-3-UPDOWN: Interface LIIN18/0/2, changed state to up

*Aug 17 18:54:40.859: %HMANRP-6-HMAN_IOS_CHANNEL_INFO: HMAN-IOS channel event for switch 1: EMP_RELAY: Channel UP!

*Aug 17 18:54:33.956: %STACKMGR-1-STACK_LINK_CHANGE: Switch 1 R0/0: stack_mgr: Stack port 1 on switch 1 is nocable

*Aug 17 18:54:33.956: %STACKMGR-1-STACK_LINK_CHANGE: Switch 1 R0/0: stack_mgr: Stack port 2 on switch 1 is down

*Aug 17 18:54:33.956: %STACKMGR-1-STACK_LINK_CHANGE: Switch 1 R0/0: stack_mgr: Stack port 2 on switch 1 is nocable

*Aug 17 18:54:33.956: %STACKMGR-6-SWITCH_ADDED: Switch 1 R0/0: stack_mgr: Switch 1 has been added to the stack.

*Aug 17 18:54:33.956: %STACKMGR-6-ACTIVE_ELECTED: Switch 1 R0/0: stack_mgr: Switch 1 has been elected ACTIVE.

*Aug 17 18:54:41.476: %LINEPROTO-5-UPDOWN: Line protocol on Interface Lsmpi18/0/3, changed state to up

*Aug 17 18:54:41.476: %LINEPROTO-5-UPDOWN: Line protocol on Interface EOBC18/0/1, changed state to up

*Aug 17 18:54:41.477: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0, changed state to down

*Aug 17 18:54:41.477: %LINEPROTO-5-UPDOWN: Line protocol on Interface LIIN18/0/2, changed state to up

*Aug 17 18:54:43.821: %IOSXE_MGMTVRF-6-CREATE_SUCCESS_INFO: Management vrf Mgmt-vrf created with ID 1, ipv4 table-id 0x1, ipv6 table-id 0x1E000001

*Aug 17 18:54:46.236: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to down

*Aug 17 18:55:05.245: %LINK-5-CHANGED: Interface Vlan1, changed state to administratively down

*Aug 17 18:55:05.467: %LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed state to up

*Aug 17 18:55:06.334: %SYS-5-RESTART: System restarted --

Cisco IOS Software [Everest], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.5.1a, RELEASE SOFTWARE (fc2)

Technical Support: http://www.cisco.com/techsupport

Compiled Tue 30-May-17 00:59 by mcpre

*Aug 17 18:55:06.903: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0, changed state to up

*Aug 17 18:55:07.018: AUTOINSTALL: Obtain siaddr 8.20.0.207 (as config server)

*Aug 17 18:55:08.953: %SYS-6-BOOTTIME: Time taken to reboot after reload = 438 seconds

*Aug 17 18:55:09.074: %PNP-6-HTTP_CONNECTING: PnP Discovery trying to connect to PnP server http://8.200.1.113:80/pnp/HELLO

*Aug 17 18:55:10.094: %PNP-6-HTTP_CONNECTED: PnP Discovery connected to PnP server http://8.200.1.113:80/pnp/HELLO

*Aug 17 18:55:16.123: %PNP-6-PROFILE_CONFIG: PnP Discovery profile pnp-zero-touch configured% Generating 2048 bit RSA keys, keys will be non-exportable...

*Aug 17 18:55:28.000: %SYS-6-CLOCKUPDATE: System clock has been updated from 18:55:26 UTC Thu Aug 17 2017 to 18:55:28 UTC Thu Aug 17 2017, configured from console by vty0.

Aug 17 18:55:29.784: %PKI-4-NOCONFIGAUTOSAVE: Configuration was modified. Issue "write memory" to save new IOS PKI configuration

Aug 17 18:55:34.048: %PNP-6-PNP_DISCOVERY_DONE: PnP Discovery done successfully

Aug 17 18:55:36.744: %AN-6-AN_ABORTED_BY_CONSOLE_INPUT: Autonomic disabled due to User intervention on console. configure 'autonomic' to enable it.

[OK] (elapsed time was 10 seconds)

Aug 17 18:55:38.453: %SSH-5-ENABLED: SSH 1.99 has been enabled

Aug 17 18:55:38.745: %PKI-4-NOCONFIGAUTOSAVE: Configuration was modified. Issue "write memory" to save new IOS PKI configuration

%Error opening tftp://8.20.0.207/network-confg (Timed out) -> That is the DHCP server

%Error opening tftp://8.20.0.207/cisconet.cfg (Timed out) -> That is the DHCP server

Aug 17 18:56:34.977: % Multiple self signed certificates in config

certificate for trust point TP-self-signed-4181268746 ignored

Aug 17 18:56:46.746: %DHCP-6-ADDRESS_ASSIGN: Interface GigabitEthernet0/0 assigned DHCP address 8.20.0.198, mask 255.255.255.0, hostname 3650-dhcp

%Error opening tftp://8.20.0.207/3650-dhcp-confg (Timed out) -> That is the DHCP server

Aug 17 18:58:05.941: AUTOINSTALL: Tftp script execution not successful for Gi0/0.

Aug 17 18:58:57.406: % Multiple self signed certificates in config

certificate for trust point TP-self-signed-4181268746 ignored

Aug 17 18:59:04.309: %DHCP-6-ADDRESS_ASSIGN: Interface GigabitEthernet0/0 assigned DHCP address 8.20.0.198, mask 255.255.255.0, hostname 3650-dhcp

3650-dhcp>

Device PNP output:

------------------ show pnp tasks ---------------------

Certificate-Install Task - Last Run ID:2, ST:7201, Result:Success, LT:48053, ET:1789 ms

Src:[http://8.200.1.113:80/ca/pem], Dst:[pnplabel]

Device-Auth Task - Never Run

Device-Info Task - Never Run

Image-Install Task - Never Run

SMU Task - Never Run

Config-Upgrade Task - Last Run ID:4, ST:5202, Result:Failed, LT:259482, ET:5489 ms

Src:[https://8.200.1.113:443/api/v1/file/onetimedownload/78fdd5df-7cba-4c79-bf79-cb98a0b12cba], Dst:[running]

Error Code:1410, Msg:[Invalid config found]

CLI-Config Task - Never Run

Licensing Task - Never Run

File-Transfer Task - Never Run

Redirection Task - Never Run

CLI-Exec Task - Last Run ID:1, ST:5401, Result:Success, LT:35934, ET:5 ms

Src:[cli-exec request], Dst:[running-exec]

Script Task - Never Run

Attachments:

1. APIC-EM PNP log file

2. The configuration used

Re: Debug PnP process

aradford — Sat, 19 Aug 2017 02:07:53 GMT

Hi,

a couple of questions.

1) Do you have option 150 set on the DHCP scope? That is why you are getting the config-server message. You can ignore those, as PnP will be preferred.

2) Is that really the config file on the server? It has some errors in it. For example the following string:

--More--

%Error opening tftp://8.20.0.207/3650-dhcp-confg (ip route vrf Mgmt-vrf 0.0.0.0 0.0.0.0 8.200.0.1"

You have lots of config in that file that can be ignored. I normally think of the PnP config file as the difference between the default config and the desired state.

I would start with a really simple example that just sets the hostname/credentials as the simplest test.

Be careful when editing files on a windows device that you avoid "special characters"

Adam

Re: Debug PnP process

aradford — Sun, 20 Aug 2017 18:41:41 GMT

BTW, what version of the PnP app are you using?

There is an interesting interaction between PnP and 16.x code with invalid configurations.

Can you try PnP APP 1.5.1.35 please?

Adam

Re: Debug PnP process

ymeloch — Mon, 21 Aug 2017 13:22:30 GMT

Hi Adam,

Thank for looking into this.

Answers:

1. I don't see option 150 in the DHCP scope. Three options are marked:

006 DNS Server
015 DNS Domain Name
043 Vendor Specific Info

I guess that I can remove 006 and 015?

2. You're right, found two errors in the config file (lines 271 & 272)

3. I surely agree with the idea of a simple config for a starter yet I would think that "complex" config should work as well.

4. I surely agree (2) with the editing file recommendation. I'm usually working w/ either Notepad++ (Win env.) or Linux vi. If you've any editor recommendation, please let me know (i recently saw several customer config issues due to hidden characters etc.)

5. I'm working with version 1.5.0.1368

Before upgrading, I made another attempt after fixing the configuration (removed lines 271 & 272) and removed the DNS entries from the DHCP. I got the same behavior (Job failed in APIC-EM w/ error messages but the device seemed to be provisioned as I can access it from my server and it contain all the configuration)

Console output -

*Aug 21 12:38:53.145: %SMART_LIC-6-AGENT_READY: Smart Agent for Licensing is initialized

*Aug 21 12:38:53.692: %IOSXE_RP_NV-3-NV_ACCESS_FAIL: Initial read of NVRAM contents failed

*Aug 21 12:38:59.910: %NBAR-6-CACHE_SYNC_INFO: Cache synchronization. Initialized.

*Aug 21 12:39:08.683: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan

*Aug 21 12:39:09.602: %LINK-3-UPDOWN: Interface Lsmpi18/0/3, changed state to up

*Aug 21 12:39:09.603: %LINK-3-UPDOWN: Interface EOBC18/0/1, changed state to up

*Aug 21 12:39:09.604: %LINEPROTO-5-UPDOWN: Line protocol on Interface LI-Null0, changed state to up

*Aug 21 12:39:09.606: %LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed state to down

*Aug 21 12:39:09.606: %LINK-3-UPDOWN: Interface LIIN18/0/2, changed state to up

*Aug 21 12:39:10.071: %HMANRP-6-HMAN_IOS_CHANNEL_INFO: HMAN-IOS channel event for switch 1: EMP_RELAY: Channel UP!

*Aug 21 12:39:00.841: %STACKMGR-1-STACK_LINK_CHANGE: Switch 1 R0/0: stack_mgr: Stack port 1 on switch 1 is nocable

*Aug 21 12:39:00.841: %STACKMGR-1-STACK_LINK_CHANGE: Switch 1 R0/0: stack_mgr: Stack port 2 on switch 1 is down

*Aug 21 12:39:00.841: %STACKMGR-1-STACK_LINK_CHANGE: Switch 1 R0/0: stack_mgr: Stack port 2 on switch 1 is nocable

*Aug 21 12:39:00.841: %STACKMGR-6-SWITCH_ADDED: Switch 1 R0/0: stack_mgr: Switch 1 has been added to the stack.

*Aug 21 12:39:00.841: %STACKMGR-6-ACTIVE_ELECTED: Switch 1 R0/0: stack_mgr: Switch 1 has been elected ACTIVE.

*Aug 21 12:39:10.713: %LINEPROTO-5-UPDOWN: Line protocol on Interface Lsmpi18/0/3, changed state to up

*Aug 21 12:39:10.713: %LINEPROTO-5-UPDOWN: Line protocol on Interface EOBC18/0/1, changed state to up

*Aug 21 12:39:10.713: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0, changed state to up

*Aug 21 12:39:10.713: %LINEPROTO-5-UPDOWN: Line protocol on Interface LIIN18/0/2, changed state to up

*Aug 21 12:39:13.405: %IOSXE_MGMTVRF-6-CREATE_SUCCESS_INFO: Management vrf Mgmt-vrf created with ID 1, ipv4 table-id 0x1, ipv6 table-id 0x1E000001

*Aug 21 12:39:15.883: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to down

*Aug 21 12:39:35.097: %LINK-5-CHANGED: Interface Vlan1, changed state to administratively down

*Aug 21 12:39:35.355: %LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed state to up

*Aug 21 12:39:35.890: %SYS-5-RESTART: System restarted --

Cisco IOS Software [Everest], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.5.1a, RELEASE SOFTWARE (fc2)

Technical Support: http://www.cisco.com/techsupport

Compiled Tue 30-May-17 00:59 by mcpre

*Aug 21 12:39:36.460: AUTOINSTALL: Obtain siaddr 8.20.0.207 (as config server)

*Aug 21 12:39:38.350: %SYS-6-BOOTTIME: Time taken to reboot after reload = 380 seconds

*Aug 21 12:39:38.559: %PNP-6-HTTP_CONNECTING: PnP Discovery trying to connect to PnP server http://8.200.1.113:80/pnp/HELLO

*Aug 21 12:39:39.576: %PNP-6-HTTP_CONNECTED: PnP Discovery connected to PnP server http://8.200.1.113:80/pnp/HELLO

*Aug 21 12:39:45.596: %PNP-6-PROFILE_CONFIG: PnP Discovery profile pnp-zero-touch configured% Generating 2048 bit RSA keys, keys will be non-exportable...

[OK] (elapsed time was 3 seconds)

*Aug 21 12:39:59.878: %SSH-5-ENABLED: SSH 1.99 has been enabled

*Aug 21 12:40:00.202: %PKI-4-NOCONFIGAUTOSAVE: Configuration was modified. Issue "write memory" to save new IOS PKI configuration

*Aug 21 12:40:19.000: %SYS-6-CLOCKUPDATE: System clock has been updated from 12:40:04 UTC Mon Aug 21 2017 to 12:40:19 UTC Mon Aug 21 2017, configured from console by vty0.

Aug 21 12:40:19.273: %PNP-6-PNP_DISCOVERY_DONE: PnP Discovery done successfully

Aug 21 12:40:19.695: %AN-6-AN_ABORTED_BY_CONSOLE_INPUT: Autonomic disabled due to User intervention on console. configure 'autonomic' to enable it.

Aug 21 12:40:20.236: %PKI-4-NOCONFIGAUTOSAVE: Configuration was modified. Issue "write memory" to save new IOS PKI configuration

%Error opening tftp://8.20.0.207/network-confg (Timed out)

%Error opening tftp://8.20.0.207/cisconet.cfg (Timed out)

%Error opening tftp://8.20.0.207/router-confg (Timed out)

%Error opening tftp://8.20.0.207/ciscortr.cfg (Timed out)

Aug 21 12:42:54.814: AUTOINSTALL: Tftp script execution not successful for Gi0/0.

Aug 21 12:43:03.359: % Multiple self signed certificates in config

certificate for trust point TP-self-signed-4181268746 ignored

Aug 21 12:43:14.760: %DHCP-6-ADDRESS_ASSIGN: Interface GigabitEthernet0/0 assigned DHCP address 8.20.0.198, mask 255.255.255.0, hostname 3650-dhcp

Aug 21 12:45:25.374: % Multiple self signed certificates in config

certificate for trust point TP-self-signed-4181268746 ignored

Aug 21 12:45:32.266: %DHCP-6-ADDRESS_ASSIGN: Interface GigabitEthernet0/0 assigned DHCP address 8.20.0.198, mask 255.255.255.0, hostname 3650-dhcp

APIC-EM History Info -

2017-08-21 08:45:35 (Eastern Daylight Time) Received response from pnp agent for message correlatorId: CiscoPnP-1.0-14-263-FFA7AA8038-12 but with error code : ZTD_CMD_ERROR Response String: ERROR:PnP Service Error 1410:Invalid config found

service log:

2017-08-21 08:44:17 (Eastern Daylight Time) Received response from pnp agent for message correlatorId: CiscoPnP-1.0-12-263-FFA7AA9830-10 but with error code : ZTD_CMD_ERROR Response String: ERROR:PnP Service Error 1410:Invalid config found

service log:

2017-08-21 08:43:25 (Eastern Daylight Time) Received response from pnp agent for message correlatorId: CiscoPnP-1.0-12-263-FFA7AA9830-10 but with error code : ZTD_CMD_ERROR Response String: ERROR:PnP Service Error 1410:Invalid config found

service log:

Going forward -

1. Upgrade to 1.5.1.35

2. Try with the same configuration.

3. If I'd see the same behavior, I'd simplify the configuration file.

Re: Debug PnP process

aradford — Mon, 21 Aug 2017 13:52:56 GMT

Thanks Yosef,

The reason is we have become stricter looking for config errors (actually too strict). You probably find 1.5.1.35 will work with this config.

In anycase, I would remove the certificate configs from the file, as that will cause issues due to strict checking. The device will have a self-signed cert as part of the boot process, and PnP will also assign a certificate.

You are also re-adding the "ip address dhcp" to the g0/0 interface. That will likely cause a new DHCP request, which is also not required.

What is IP address 8.20.0.207 is that the router for the 8.20.0.0/24 network? does it have "ip helper" on this interface?

I like vi, but that is a historic thing. sublime text is quite nice.

Adam

Re: Debug PnP process

ymeloch — Mon, 21 Aug 2017 15:16:17 GMT

Thanks, Adam.

Updates:

I updated to 1.5.1.35
Successfully run the 3650 provisioning with the same configuration (note the 10:31:11 printout... )

2017-08-21 10:31:18 (Eastern Daylight Time) Save startup Configuration was performed successfully

2017-08-21 10:31:11 (Eastern Daylight Time) Ignoring invalid config error and moving to PROVISIONED_CONFIG state

2017-08-21 10:28:37 (Eastern Daylight Time) Matched a pre-provisioned device in site pnp-3650

2017-08-21 10:27:31 (Eastern Daylight Time) Matched a pre-provisioned rule in site pnp-3650

2017-08-21 10:26:56 (Eastern Daylight Time) Received new work request from Agent while expecting work response. Retrying operation FILESYSTEM_INFO_REQUESTED

2017-08-21 10:26:56 (Eastern Daylight Time) Device authentication status has changed to Unauthenticated(SUDI feature not supported by device)

2017-08-21 10:26:34 (Eastern Daylight Time) Device first contact

2017-08-21 10:20:02 (Eastern Daylight Time) Matched a pre-provisioned rule in site pnp-3650

2017-08-21 10:19:29 (Eastern Daylight Time) Received new work request from Agent while expecting work response. Retrying operation FILESYSTEM_INFO_REQUESTED

2017-08-21 10:19:16 (Eastern Daylight Time) Device authentication status has changed to None

2017-08-21 10:19:15 (Eastern Daylight Time) Device first contact

Leason learned:

Work with the latest PNP release.
Make sure a proper configuration (e.g. hidden characters (although there's an APIC/PNP check for that), not relates conf ( -- more --- etc.), remove certificate (although APIC/PNP handled that successfully) etc.)
in DHCP - Uncheck options 006 & 015 (DNS Related)

The procedure I followed:

Load the configuration to APIC-EM PNP.
- Make sure that the configuration does not include special characters.
Add the device w/ the correct model and SN to APIC-EM PNP.
- Make sure that the device is in Pending state.
Configure DHCP w/ option 43 only
- Uncheck options 006 & 015 (DNS Related).
- Make sure to activate the scope.
On the device, run:
- no debug pnp all
- debug pnp all
  - Check that PNP messages appear. for example:
  - *Aug 21 12:28:35.749: %PNPA-DISCOVERY: 2 _pweh.info: System write erase done (PT=313020).
  - *Aug 21 12:28:35.750: %PNPA-DISCOVERY: Setting pnpa pnp_start_cfg_empty_flag var to TRUE
- wr er (confirm)
- reload (don't save conf)