Solved: C1111 + APIC-EM PnP issue

steffenschumacher · ‎06-24-2020

So I had a working solution where I could deploy new DMVPN routers (C1111) using our APIC EM instance over the internet, using cloud redirect, where both a cert is deployed, IOS is upgraded and config is deployed.

Now I've found that this stopped working - and I'm not sure how this happened..

Cloud redirect still works, and the a pnp profile is deployed, and the cert from APIC-EM is deployed (as its also our PKI).

But then APIC EM gets stuck, reporting: ERROR_HEALTH_CHECK_TIMER_EXPIRED,

Failed health check since device is stuck in non-terminal state DEVICE_INFO_REQUESTED for more than threshold time: 0 hours, 16 minutes, 0 seconds

*Jun 24 12:05:20.872: %PKI-2-NON_AUTHORITATIVE_CLOCK: PKI functions can not be initialized until an authoritative time source, like NTP, can be obtained.

*Jun 24 12:05:22.399: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named TP-self-signed-2789104647 has been generated or imported by crypto-engine

*Jun 24 12:05:22.400: %SSH-5-ENABLED: SSH 1.99 has been enabled

*Jun 24 12:05:22.457: %PKI-4-NOCONFIGAUTOSAVE: Configuration was modified. Issue "write memory" to save new IOS PKI configuration

*Jun 24 12:05:22.527: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named TP-self-signed-2789104647.server has been generated or imported by crypto-engine

%Error opening tftp://255.255.255.255/network-confg (Timed out)

*Jun 24 12:05:48.621: %PNP-6-HTTP_CONNECTING: PnP Discovery trying to connect to PnP server https://52.203.231.173:443/pnp/HELLO

*Jun 24 12:05:49.121: %PNP-6-HTTP_CONNECTED: PnP Discovery connected to PnP server https://52.203.231.173:443/pnp/HELLO

*Jun 24 12:05:51.405: AUTOINSTALL: Tftp script execution not successful for Gi0/0/0.

*Jun 24 12:05:57.184: %PNP-6-PNP_DISCOVERY_DONE: PnP Discovery done successfully

*Jun 24 12:06:10.147: %AN-6-AN_ABORTED_BY_CONSOLE_INPUT: Autonomic disabled due to User intervention on console. configure 'autonomic' to enable it.

*Jun 24 12:06:12.000: %SYS-6-CLOCKUPDATE: System clock has been updated from 12:06:11 UTC Wed Jun 24 2020 to 12:06:12 UTC Wed Jun 24 2020, configured from console by vty0.

Jun 24 12:06:12.001: %PKI-6-AUTHORITATIVE_CLOCK: The system clock has been set.

Jun 24 12:06:13.120: %PKI-4-NOCONFIGAUTOSAVE: Configuration was modified. Issue "write memory" to save new IOS PKI configuration

Router#sh run | sec pnp pro

pnp profile pnp_redirection_profile

transport https host yyy.xxxxx.com port 443 remotecert primary-cert

APIC-EM version is 1.6.3.114 and the c1111 is 16.9.2 out of the box.

I've attached a log of the console, but I can't find any smoking guns..

steffenschumacher · ‎07-01-2020

So I figured this one out on my own..

When using cloud redirect(PnP Connect), it automatically copies the certificate of your PnP server, which in our case had just been renewed. I believe the reason behind this, is that it allows self-signed certificates to be validated by the cisco device when it is redirected to the PnP server.

In our case we use publicly signed certs, and it had just been renewed - and the cloud redirect thing doesn't automatically renew its copy, so the cisco device is fed an old cert which then does not match up any more..

View solution in original post

steffenschumacher · ‎07-01-2020

So I figured this one out on my own..

When using cloud redirect(PnP Connect), it automatically copies the certificate of your PnP server, which in our case had just been renewed. I believe the reason behind this, is that it allows self-signed certificates to be validated by the cisco device when it is redirected to the PnP server.

In our case we use publicly signed certs, and it had just been renewed - and the cloud redirect thing doesn't automatically renew its copy, so the cisco device is fed an old cert which then does not match up any more..