PnP config-upgrade fails (own PnP server implementation)

[paradoxxl]

Hi

I have started to create an own PnP server for the current semester project. The server is implemented in golang. As PnP is used by APIC-EM, I'll post the question in this section as someone might have experienced a similar issue.

I only cover the HTTP REST API with the following 3 methods as it is described in the protocol specifications (Cisco DevNet:):

1. GET     /pnp/HELLO
2. POST  /pnp/WORK-REQUEST
3. POST  /pnp/WORK-RESPONSE

With my server, I can successfully upgrade the system OS on a C3650 switch. But I have a problem wen I try to deploy the initial configuration. The behavior is the following:

1. The device calls /pnp/Hello on startup and receives the HTTP 200.
2. Afterwards, it calls /pnp/WORK-REQUEST with the body:

<?xml version="1.0" encoding="UTF-8"?>:<pnp xmlns="urn:cisco:pnp" version="1.0" udi="PID:WS-C3650-24TS,VID:V04,SN:.....">

<info xmlns="urn:cisco:pnp:work-info" correlator="CiscoPnP-1.0-1-3-3B34D1D0-2">

  <deviceId>

    <udi>PID:WS-C3650-24TS,VID:V04,SN:....</udi>

    <SUDI>PID:WS-C3650-24TS,SN:.....</SUDI>

    <hostname>Switch</hostname>

    <authRequired>false</authRequired>

    <viaProxy>false</viaProxy>

    <securityAdvise>Password in clear text in unsecured transport</securityAdvise>

  </deviceId>

</info>

</pnp>

     My response, where %v are the corresponging values from the request:

<?xml version="1.0" encoding="UTF-8"?>

<pnp xmlns="urn:cisco:pnp" version="1.0" udi="%v">

  <request correlator="%v" xmlns="urn:cisco:pnp:config-upgrade">

    <config details="all">

      <copy>

        <source>

          <location>http://192.168.100.10/pnp/test.cfg</location>

        </source>

      </copy>

    </config>

  </request>

</pnp>

     3. Then, de device gets the config from http://192.168.100.10/pnp/test.cfg which looks like this:

hostname asdf

!

enable password class

!

username cisco privilege 15 secret 8 $8$vshOqfCl41vryU$KsAkOVxWUZg0h3C98EURn5q0VxF0XE75S3GH5yixcQk

no aaa new-model

!

!

ip domain-name ip8.local

!

!

ip ssh version 2

!

interface Vlan1

ip address 192.168.100.33 255.255.255.0

no shutdown

!

!

no setup express

!

!

line con 0

stopbits 1

line aux 0

line vty 0 15

logging synchronous

login local

transport input ssh

!

ntp server pool.ntp.org

!

!

!

crypto key gen rsa mod 2048

!

end

     4. As the config is received, I can see the hostname change on the console. It calls /pnp/WORK-RESPONSE with the success message and receives a BYE-Message:

<?xml version="1.0" encoding="UTF-8"?>

<pnp xmlns="urn:cisco:pnp" version="1.0" udi="PID:WS-C3650-24TS,VID:V04,SN:....">

  <response correlator="CiscoPnP-1.0-1-3-3B75CD7C-2" success="1" xmlns="urn:cisco:pnp:config-upgrade">

    <serviceLog>Loading http://192.168.100.10/pnp/test.cfg </serviceLog>

  </response>

</pnp>

5. But then, the device reloads and ignores the configuration. I cannot see any error message while it starts, it just ignored the previous applied configuration.

Did anyone experience a similar problem with APIC-EM? I think the protocol implementation should be more or less ok as I could upgrade the switch from 03.07.04E to 03.07.05E using my implementation as well. Could there be a problem with the sample configuration I provided?

Thanks,

Dominik

[paradoxxl]

Further debugging:

I have tried to apply the config to run or start by using the 'applyTo' parameter in the WORK-REQUEST response. The default should be startup, but I tried both with the same result.

While the switch is applying the config and I access the console, the hostname changes, the passwords are being set and I have to use them in order to go into enable mode. When I run she show pnp tasks command, I get the following output:

------------------ show pnp tasks ---------------------

Certificate-Install Task - Never Run

Image-Install Task - Never Run

Config-Upgrade Task - Last Run ID:1, ST:5202, Result:Success, LT:240200, ET:26760 ms
        Src:[http://192.168.100.10/pnp/test.cfg], Dst:[running]

CLI-Config Task - Never Run

Licensing Task - Never Run

File-Transfer Task - Never Run

Redirection Task - Never Run

But afterwards, the switch does a reload and the whole thing starts again. Does anyone have an idea where I could start?

[paradoxxl · Accepted solution]

I investigated the problem now and I have found the problem: Incomplete or at lest out-dated protocol specs (the one I have mentioned in the first post).

Approach:

Fore debugging purposes, I installed APIC-EM on my lab environment and updated a certificate with a self generated one. Then I installed wireshark on the APIC-EM VM and configured the device in the PnP app. After the device reload, I recorded the traffic and decrypted the https traffic after the certificate installation part in wireshark using my known private key.

Findings:

I recorded 36 HTTP request/response pairs of communication between the device and the controller. Most of theme aren't interesting for my problem, but the config response in the /pnp/WORK-REQUEST has an additional tag <noReload> which wil probably fix the reloading. Unfortunately, I cannot decrypt the following WORK-RESPONSE Request after the device downloads and installs the config.

But in the next /pnp/WORK-REQUEST, the controller commands the device to write memory using a <execCLI> tag. This is quite contraty to the protocol specification as it says the config is applied to startup by default. Yet, it matches the actual behavior and should resolve my problem.

Question:

Is there an updated and complete specification of the PnP protocol and the xml schema publicly available? It would be more efficient reading the specs than debugging it

Once I have programmed the updated flow into my controller, I will post again whether it was successful or not.

Regards,

Domnik

[paradoxxl]

It works!

As I thought, using noReload and the following execCLI command did the job.

Regards,

Dominik

[javidr]

Hi Dominik,

Thanks for the tip. I got it to reload after adding the <reload> tags.

Do you mind sharing your golang script?

Thanks,

Javid

[SachaB77516]

Just in case anybody's still working with PnP, there is a complete schema spec here: https://developer.cisco.com/docs/network-plug-n-play/#!schemas/pnp-schema-files

[Tanasis Yankov]

The link is not working do you have this schema files ?