PnP onboarding fail - Command unsuccessful from the device

[RhysCrane2388]

Hi,

After an upgrade from DNA Center 2.3.5.4 to Catalyst Center 2.3.7.7, PnP fails for some swiches though works for others during onboarding. Strangely enough, this problem started a few days after the upgrade, pnp was working fine for 2 or 3 days after the upgrade before this happened.

It fails consistently in the workflow “Executing System Workflow to Initialize Device” with the following error. “NCOB04000: Command unsuccessful from the device. (PERMISSION_DENIED. authorization failed)”. Screenshot 1.

Within the workflow, CAPABILITY_INFO and DEVICE_INFO work fine, but the SHOW_PNP_STACK_INFO command is not successful. Screenshot 2. The command "show pnp stack-info" works fine when run from the console.

The example im looking at is a C9200L-48PL-4G running iosxe 17.12.4, though several others including a C9200L-24P-4G was also affected. It does not seem to be model related though as other 24 port switches of the same model have been onboarded successfully with the same image, profile, day0 template etc.

We have tried removing the device from pnp, resetting it with “pnp service reset”, removing the certs, startupconfig and keys manually with no success.

Ive even wiped the switch from the cli with "pnp service reset", deleted the switch from the pnp database and let it come into pnp unclaimed as "Switch" and this still happens, no site assignment, no day0 template.

The console logs show its setting up network settings in the switch, im guessing it has something to do with aaa or tacacs. Has anyone seen this? Why would onboarding set up network settings if its not even assigned to a site?

Cheers

Rhys

[marce1000]

          - FYI : https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvt63467

  M.

[Preston Chilcote]

@marce1000 That bug has been fixed for years.  Shouldn't be relevant here.

@RhysCrane2388 Are you pushing AAA commands in your template?  If so, can you provision those in a day-N template instead?

[RhysCrane2388]

No the day0 template doesn't have any aaa commands, and works with most switches, same model, same image from the factory. This problem is showing up in the onboarding process before the day0 template is even called. It also shows up even if the switch has not been claimed and just gets in touch with Cat Center PnP process.

I would expect a factory reset switch with no claim to just show up in PnP with the name "Switch" sitting there waiting for instructions. These switches get in touch and then fail before the claim process.

The reason i mention aaa is that for just these switches, somehow the aaa config is there (along with some other network settings looking stuff, logging servers, banner etc.), even after a "pnp service reset" or a manual removal of the startup and not saving the running config before a reload.

Cheers
Rhys

[Preston Chilcote]

That's very interesting.  You are right that a pnpa service reset should be enough to get things working.   If you don't have aaa in your onboarding template, then aaa shouldn't play any role in PnP.  (IIRC, Cat Center will not push aaa configs during pnp on its own).  There are other ways for configs to get on during bootup, such as tftpboot or a .cfg file in flash.  Take a close look at console logs to see if one of those is mentioned and "show pnp trace" to see if aaa is mentioned anywhere.

Might be fastest to open a TAC case though.

[RhysCrane2388]

Hi @marce1000 @Preston Chilcote 

Found the source of the aaa config, we found this in the console while it was working through PnP. "%SYS-5-CONFIG_I: Configured from tftp://255.255.255.255/network-confg by console".

So tftp from broadcast downloaded an older network-config file on a tftp server. It was the aaa config breaking it as described in the bug, just not coming from CatCenter.

I was under the impression while the PnP agent was running, it wouldn't run the other config methods, but they seem to be concurrent. I'm not sure what changed in our network, but i have to guess its just a separate change with that kind of timing.

Thanks for the help!

Cheers
Rhys