APIC-EM PNP ZTP via DNS method fails, DHCP Option43 works

[dmr]

What might cause APIC-EM PNP ZTP via DNS discovery method to fail?

Even with pnpserver.<fqdn> and pnpntpserver.<fqdn> configured and reachable following DHCP assignment (for same fqdn as DHCP assignment),  APIC-EM PNP fails to complete device provisioning on a new target.

The same exact setup works perfectly if DHCP Option43 data is populated.  I am trying to get it to work via DNS per docs.

Testing with APIC-EM version 1.6.2.30018 and a WS-C3560CX-8TC-S running 15.2(6)E2

APIC-EM device stages go through:

• pending
• getting device info
• establishing secure channel
• getting device info -- GETS STUCK HERE, this is right after seeing on device console:  %PKI-4-NOCONFIGAUTOSAVE: Configuration was modified.  Issue "write memory" to save new IOS PKI configuration
• ERROR - Failed health check since device is stuck in non-terminal state FILESYSTEM_INFO_REQUESTED for more than threshold time: 0 hours, 16 minutes, 0 seconds

What am I missing ??

The full console output of the session is appended below (sanitized).

Thanks for any help/hints/pointers/advice ... --David

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2019.02.01 13:43:00 =~=~=~=~=~=~=~=~=~=~=~=

Switch#reload

System configuration has been modified. Save? [yes/no]: no
Proceed with reload? [confirm]

Feb  1 18:42:40.106: %SYS-5-RELOAD: Reload requested by console. Reload Reason: Reload command.
CPU rev: B
Image passed digital signature verification

Board rev: 2
Testing DataBus...
Testing AddressBus...
Testing Memory from 0x00000000 to 0x1fffffff...  
Using driver version 4 for media type 1

Xmodem file system is available.

Base ethernet MAC Address: xx:xx:xx:xx:xx:xx

The password-recovery mechanism is enabled.

APM USBUSB EHCI 1.00

USB EHCI 1.00

USB Console INIT

Initializing Flash...

mifs[5]: 10 files, 1 directories

mifs[5]: Total bytes     :    1806336

mifs[5]: Bytes used      :     687616

mifs[5]: Bytes available :    1118720

mifs[5]: mifs fsck took 1 seconds.

mifs[6]: 0 files, 1 directories

mifs[6]: Total bytes     :    3870720

mifs[6]: Bytes used      :       1024

mifs[6]: Bytes available :    3869696

mifs[6]: mifs fsck took 0 seconds.

mifs[7]: 5 files, 1 directories

mifs[7]: Total bytes     :     258048

mifs[7]: Bytes used      :       8192

mifs[7]: Bytes available :     249856

mifs[7]: mifs fsck took 1 seconds.

mifs[8]: 5 files, 1 directories

mifs[8]: Total bytes     :     258048

mifs[8]: Bytes used      :       8192

mifs[8]: Bytes available :     249856

mifs[8]: mifs fsck took 0 seconds.

mifs[9]: 8 files, 1 directories

mifs[9]: Total bytes     :  122185728

mifs[9]: Bytes used      :   46206976

mifs[9]: Bytes available :   75978752

mifs[9]: mifs fsck took 44 seconds.

...done Initializing Flash.

Loading "flash:c3560cx-universalk9-mz.152-6.E2.bin"...Verifying image flash:c3560cx-universalk9-mz.152-6.E2.bin............................................................................................................................................................................................................................................................................................................................................................Image passed digital signature verification

$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$

File "flash:c3560cx-universalk9-mz.152-6.E2.bin" uncompressed and installed, entry point: 0x3000

executing...


              Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

           cisco Systems, Inc.
           170 West Tasman Drive
           San Jose, California 95134-1706



Cisco IOS Software, C3560CX Software (C3560CX-UNIVERSALK9-M), Version 15.2(6)E2, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2018 by Cisco Systems, Inc.
Compiled Thu 13-Sep-18 04:00 by prod_rel_team
Initializing flashfs...
Using driver version 4 for media type 1
mifs[7]: 10 files, 1 directories
mifs[7]: Total bytes     : 1806336   
mifs[7]: Bytes used      : 687616    
mifs[7]: Bytes available : 1118720   
mifs[7]: mifs fsck took 0 seconds.
mifs[7]: Initialization complete.

mifs[8]: 0 files, 1 directories
mifs[8]: Total bytes     : 3870720   
mifs[8]: Bytes used      : 1024      
mifs[8]: Bytes available : 3869696   
mifs[8]: mifs fsck took 0 seconds.
mifs[8]: Initialization complete.

mifs[9]: 5 files, 1 directories
mifs[9]: Total bytes     : 258048    
mifs[9]: Bytes used      : 8192      
mifs[9]: Bytes available : 249856    
mifs[9]: mifs fsck took 0 seconds.
mifs[9]: Initialization complete.

mifs[10]: 5 files, 1 directories
mifs[10]: Total bytes     : 258048    
mifs[10]: Bytes used      : 8192      
mifs[10]: Bytes available : 249856    
mifs[10]: mifs fsck took 0 seconds.
mifs[10]: Initialization complete.

mifs[11]: 8 files, 1 directories
mifs[11]: Total bytes     : 122185728
mifs[11]: Bytes used      : 46206976  
mifs[11]: Bytes available : 75978752  
mifs[11]: mifs fsck took 1 seconds.
mifs[11]: Initialization complete.

...done Initializing flashfs.
Checking for Bootloader upgrade..
Boot Loader upgrade not needed(v)


FIPS: Flash Key Check : Begin
FIPS: Flash Key Check : End, Not Found, FIPS Mode Not Enabled

extracting front_end/front_end_ucode_info (43 bytes)
POST: MA BIST : Begin
POST: MA BIST : End, Status Passed

POST: TCAM BIST : Begin
POST: TCAM BIST : End, Status Passed

POST: ACT2 Authentication : Begin
POST: ACT2 Authentication : End, Status Passed
POST: PortASIC Port Loopback Tests : Begin
POST: PortASIC Port Loopback Tests : End, Status Passed

POST: PortASIC Macsec Loopback Tests : Begin
POST: PortASIC Macsec Loopback Tests : End, Status Passed

Waiting for Port download...Complete
Initializing Port Extension Feature Support...


This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C3560CX-8TC-S (APM86XXX) processor (revision G0) with 524288K bytes of memory.
Processor board ID xxxxxxxxxxx
Last reset from power-on
1 Virtual Ethernet interface
12 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address       : xx:xx:xx:xx:xx:xx
Motherboard assembly number     : 73-16472-05
Power supply part number        : 341-0208-03
Motherboard serial number       : xxxxxxxxxxx
Power supply serial number      : xxxxxxxxxxx
Model revision number           : G0
Motherboard revision number     : A0
Model number                    : WS-C3560CX-8TC-S
System serial number            : xxxxxxxxxxx
Top Assembly Part Number        : 68-5358-03
Top Assembly Revision Number    : E0
Version ID                      : V03
CLEI Code Number                : CMM1P10DRA
Hardware Board Revision Number  : 0x02


Switch Ports Model                     SW Version            SW Image                 
------ ----- -----                     ----------            ----------               
*    1 12    WS-C3560CX-8TC-S          15.2(6)E2             C3560CX-UNIVERSALK9-M    


Press RETURN to get started!


*Mar  1 00:00:28.972: Read env variable - LICENSE_BOOT_LEVEL = ipservices
*Jan  2 00:00:00.517: %IOS_LICENSE_IMAGE_APPLICATION-6-LICENSE_LEVEL: Module name = c3560cx Next reboot level = ipbase and License = No valid license found
Feb  1 18:46:06.764: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to down
Feb  1 18:46:11.716: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan
Feb  1 18:46:36.047: %SYS-5-RESTART: System restarted --
Cisco IOS Software, C3560CX Software (C3560CX-UNIVERSALK9-M), Version 15.2(6)E2, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2018 by Cisco Systems, Inc.
Compiled Thu 13-Sep-18 04:00 by prod_rel_team
Feb  1 18:46:40.035: %USB_CONSOLE-6-MEDIA_RJ45: Console media-type is RJ45.
Feb  1 18:46:54.558: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to up
Feb  1 18:46:55.557: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to up
Feb  1 18:46:55.575: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to up
Feb  1 18:47:56.151: AUTOINSTALL: Obtain siaddr xxx.xxx.xxx.xxx (as config server)
Feb  1 18:48:00.562: %PNP-6-HTTP_CONNECTING: PnP Discovery trying to connect to PnP server http://pnpserver.xxx.xx.xxx.xxx:80/pnp/HELLO
Feb  1 18:48:00.709: %PNP-6-HTTP_CONNECTED: PnP Discovery connected to PnP server http://pnpserver.xxx.xx.xxx.xxx:80/pnp/HELLO
Feb  1 18:48:01.719: %PNP-6-PROFILE_CONFIG: PnP Discovery profile pnp-zero-touch configured
Feb  1 18:48:46.650: %PNP-6-PNP_DISCOVERY_DONE: PnP Discovery done successfully
Feb  1 18:48:46.793: %SYS-6-CLOCKUPDATE: System clock has been updated from 18:48:44 UTC Fri Feb 1 2019 to 18:48:46 UTC Fri Feb 1 2019, configured from console by console.
Feb  1 18:48:47.191: %PKI-4-NOCONFIGAUTOSAVE: Configuration was modified.  Issue "write memory" to save new IOS PKI configuration

[dmr · Accepted solution]

Following up on my own post, hoping this helps someone. --David

My problem turned out to be a certificate issue.

With the DHCP Option43 PnP method, only the IP address needs to be in the APIC-EM's certificate, which it is by default with its self-signed cert. With the DNS PnP technique, the FQDN of the APIC-EM needs to also be part of the Subject
Alternative Name field in the APIC-EM's cert.

Bob Adams of Cisco TAC was *extremely* helpful in helping me resolve this.  He is awesome.

Here are some relevant notes from him:

Server ID check. This is something that was added into the IOS on Cisco devices around mid last year. Whenever the device is performing a PnP and gets a response from the server, when it switches to port 443, it checks the certificate on that server to make sure it matches what PnP thinks it should be. So, if you are using a PnP profile with the IP address, the certificate only needs to have the IP build into it. And if you are using the self-signed certificate that comes with APIC-EM, that is what it has. But when you switch to using DNS discovery, it is using the fully qualified name. So if that name does not exist in the SAN (Subject Alternative Name) field of the certificate, it won't pass the server ID check and will fail. The correction for this is to create a new cert for the server that includes both the IP address and DNS name in the SAN field.

How to generate a self-signed certificate for DNAC/APIC-EM
Login to the CLI of the APIC-EM server as the grapevine user.
1. Create a req.conf file with below content on your APIC-EM server, customize the State/City/Company/Division, IP.1, IP.2, IP.3, ... with the interface IP addresses and VIP, and optionally any additional DNS FQDN you would like to use.

[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no

[req_distinguished_name]
C = US
ST = VA
L = SomeCity
O = MyCompany
OU = MyDivision
CN = www.company.com<http://www.company.com/>

[v3_req]
keyUsage = keyEncipherment, keyCertSign
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names
basicConstraints=critical,CA:true

[alt_names]
IP.1 = 1.2.3.4
IP.2 = 5.6.7.8
IP.3 = 10.28.113.59
DNS.1 = kb.example.com
DNS.2 = helpdesk.example.org

2. Run (you will need to install openssl if it is not already there)
Without encryption:
openssl req -x509 -nodes -days 730 -newkey rsa:2048 -keyout key.pem -out cert.pem -config req.conf -extensions 'v3_req'

With encryption (this will prompt for a passphrase):
openssl req -x509 -days 730 -newkey rsa:2048 -keyout key.pem -out cert.pem -config req.conf -extensions 'v3_req'

3. Upload the key.pem and cert.pem to Certificates in APIC-EM , select PEM format when upload. Select encryption and enter the passphrase if one is used.

[2cellefsen]

Very useful post, I had exactly the same issue after we got a new batch of 500 2960-L's with updated IOS. The new switches that failed with just IP in the cert came from factory with version 15.2(6)E2b. Switches with 15.2(6)E1 work fine with just IP in the cert.

Thank you for a well written solution!

[ipagliani]

Ciao,

thanks for sharing.

I did exactly in that way but on router side the trustpoint create using the previous selfsig certificate (create during the installation) !!

I think that when router connect the APIC-EM download the own certificate in order to authenticate itself to the device.

Any idea ?

[dmr]

Following up on my own post, hoping this helps someone. --David

My problem turned out to be a certificate issue.

With the DHCP Option43 PnP method, only the IP address needs to be in the APIC-EM's certificate, which it is by default with its self-signed cert. With the DNS PnP technique, the FQDN of the APIC-EM needs to also be part of the Subject
Alternative Name field in the APIC-EM's cert.

Bob Adams of Cisco TAC was *extremely* helpful in helping me resolve this.  He is awesome.

Here are some relevant notes from him:

Server ID check. This is something that was added into the IOS on Cisco devices around mid last year. Whenever the device is performing a PnP and gets a response from the server, when it switches to port 443, it checks the certificate on that server to make sure it matches what PnP thinks it should be. So, if you are using a PnP profile with the IP address, the certificate only needs to have the IP build into it. And if you are using the self-signed certificate that comes with APIC-EM, that is what it has. But when you switch to using DNS discovery, it is using the fully qualified name. So if that name does not exist in the SAN (Subject Alternative Name) field of the certificate, it won't pass the server ID check and will fail. The correction for this is to create a new cert for the server that includes both the IP address and DNS name in the SAN field.

How to generate a self-signed certificate for DNAC/APIC-EM
Login to the CLI of the APIC-EM server as the grapevine user.
1. Create a req.conf file with below content on your APIC-EM server, customize the State/City/Company/Division, IP.1, IP.2, IP.3, ... with the interface IP addresses and VIP, and optionally any additional DNS FQDN you would like to use.

[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no

[req_distinguished_name]
C = US
ST = VA
L = SomeCity
O = MyCompany
OU = MyDivision
CN = www.company.com<http://www.company.com/>

[v3_req]
keyUsage = keyEncipherment, keyCertSign
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names
basicConstraints=critical,CA:true

[alt_names]
IP.1 = 1.2.3.4
IP.2 = 5.6.7.8
IP.3 = 10.28.113.59
DNS.1 = kb.example.com
DNS.2 = helpdesk.example.org

2. Run (you will need to install openssl if it is not already there)
Without encryption:
openssl req -x509 -nodes -days 730 -newkey rsa:2048 -keyout key.pem -out cert.pem -config req.conf -extensions 'v3_req'

With encryption (this will prompt for a passphrase):
openssl req -x509 -days 730 -newkey rsa:2048 -keyout key.pem -out cert.pem -config req.conf -extensions 'v3_req'

3. Upload the key.pem and cert.pem to Certificates in APIC-EM , select PEM format when upload. Select encryption and enter the passphrase if one is used