Updating management IP address of a switch during the PnP process

[Preston Chilcote]

Onboarding a switch with Plug and Play (PnP) requires that the onboarding switch initially get its IP address via DHCP.  After whichever method (i.e. DHCP option 43, DNS, PnP Connect) is successfully used to tell the switch the IP address of Cisco DNA, the onboarding device creates an HTTPS session to Cisco DNA.  Upon successful completion of a PnP Claim, Cisco DNA needs to add a management IP address to Inventory.  The IP address it selects is the source IP of the HTTPS session.  That commonly leads to the DHCP address showing up as the management IP in Inventory.  This can be fixed manually with a Edit Device action, however, it's helpful if everything is automatic.  It is very common for the onboarding configuration to replace the dynamic IP address with a static one, potentially on a different interface or vlan.  This commonly causes the PnP Claim process to timeout and fail since Cisco DNA can no longer reach the switch using the dynamic IP address.

With these facts in mind, an Onboarding configuration template can be created that allows Cisco DNA to both update the IP address and have Cisco DNA learn that new management IP address automatically.  The trick is make the onboarding device close the HTTPS connection (via interface shutdown).  The PnP agent will have configured a "pnp profile" for you, which will try to reestablish the HTTPS session to Cisco DNA every 5 seconds.  These templates will control the source of that new session, which Cisco DNA will use to infer the desired management IP address.

I tested these templates on a C9300-48U running IOS-XE version 17.9.3.  The upstream interface is configured as a trunk port with DTP enabled.

Example 1:  Using Vlan 1 for Onboarding and Management.

interface vlan 1
    shutdown
    ip address 192.168.1.2 255.255.255.0
    no shutdown

ip route 0.0.0.0 0.0.0.0 192.168.1.1

You can replace vlan 1 with whichever vlan you are using for pnp (as specificed by pnp startup-vlan command on the upstream switch.

Example 2: Using Vlan 2 for Onboarding and Vlan 3 for Management.  This requires "pnp startup-vlan 2" on the upstream switch.

interface vlan 2
    shutdown 
    !alternatively, you can configure "no interface vlan 2"

interface GigabitEthernet1/0/48
    no switchport trunk allowed vlan 2
    !when using pnp-startup vlan on a trunk port upstream, 
    !pnp-agent will configure "switchport trunk allowed vlan x"

interface vlan 3
    ip address 192.168.3.2 255.255.255.0

ip route 0.0.0.0 0.0.0.0 192.168.3.1

Example 3: Using Vlan 1 for Onboarding and Loopback 0 for Management.

interface vlan 1
  shutdown
  ip address 192.168.1.2 255.255.255.0
  no shutdown


interface Loopback 0
    ip address 10.1.1.1 255.255.255.0

ip route 0.0.0.0 0.0.0.0 192.168.1.1
ip http client source-interface Loopback0
! If we don't specify the source interface for http, the onboarding device will use
! vlan 1's new IP address to source http, and that is what Cisco DNA will infer is 
! the management IP address for Inventory.

I recommend keeping your Onboarding template as simple as the above.  The onboarding process will automatically add some config for you, including:

- CLI+SNMP credentials

- SSH config

- DNS

- Hostname

The rest of the production config should be present in a day-N template.  This is recommended so that a configuration change to any part of the config in the future can be done with a day-N provision, which doesn't require a repeat of the PnP process.

[catpires]

 

Just to add to these very useful tips&tricks for PnP:

 

      -When testing Example 1 and 2 I never flap the PnP VLAN SVI and it works just fine.
      -One note regarding "Example 3: Using Vlan 1 for Onboarding and Loopback 0 for Management." -  with the command "ip http client connection forceclose" there is no need to break reachability to the PnP VLAN SVI (by using "no interface vlan X" or shutdown). This forces closing the HTTPS connection.

[Torbjørn]

@Preston Chilcote, is there any way to update the management IP address for devices in inventory from templates without also having to manually updating the IP in inventory?

[Preston Chilcote]

@Torbjørn The only way I know of, besides manually updating the management IP in Inventory is to delete the device from inventory and discover it again.

[DanielLarsson63527]

Hi,

I mostly wrote the below comment to add some clarifications about the PNP-process in case anyone is trying to troubleshoot why things might not work.
It does work as long as PNP device and DNAC agreees about the onboarding...and all the databases are aligned! When they don't....below might be helpfull!

For anyone doing research around PNP this is a great resource and it covers some of the more annoying things with APIC-EM/DNAC and how it works. I want to add my opinion to anyone coming here looking for PNP/ZTP tips and how to onboard devices correctly.

Having worked with APIC-EM and then DNAC since it was "released" i can say that ZTP/PNP is a pain when you *need* to change the DHCP-address to something else automatically. The way APIC/APIC implementation in DNAC seems to be coded is...that it simply registers a "device id" that is tied to the DHCP-Discovery address and calling home from  different IP will most likely generate a mismatch somewhere in a database unless your uplink is L2/TRUNK with CDP. This is very DNAC release-dependent and what works in one release might fail in a different one.

It is very common that you want to onboard devices and route your Loopback0 as your management through whatever underlay you are using and have this as your management IP.

With the information provided for the above onboarding examples...In fact in most/many releases of DNAC above will not work and generate a "network connectivity error" even though all configuration as pushed to PNP-device if you are doing more advanced than above (especially L3 routed links PNP). For me ZTP is also configuring a lot of things and not just VLANs and a Loopback.

Funny thing is that APIC-EM/DNAC has the feature when you run a "Network Discovery" to just add to inventory with Loopback-interfaces...why this is not in production for Onboarding I have been wondering for years. Easily fixed in multiple ways but for example during "sync" after added to inventory when confiruation is parsed. One of the reasons this would be a fix for "inventory management ip address" is because of the under-the-hood working databases and processes of DNAC.

ip http client source-interface Loopback0 *does not update DNAC inventory ip address even though this has been in documentations for years* (during onboarding). This seems to work, for the most part but randomly also not, with switched/L2 PNP fairly stable. It has worked stable in maybe 1-2 releases of DNAC for L3 PNP. Don't know, since it's very undocumented, what exactly DNAC PNP would do differently when you switch to routed links (debugging logs indicates it treates them differently). It is true that it *will work under those cases where DNAC successfully opens a new HTTP/HTTPS session from new ip* and DNAC has a route back to both IP-addresses and all databases agree BEFORE device is added to inventory and AFTER site-config-task has completed.

ip http client connection forceclose ... does indeed make your device call APIC-EM/DNAC from the new ip-address but will not work if the "database DeviceID" is "null" for the new IP-address since it was not added in PNP-database (and PNP-agent does not create a new session for the same task) with that IP.  It is very common with enterprise-networks with ECMP, Loadbalancers, Proxy-servers and everything else that can be Inline between PNP Device <-> DNAC.

You can always troubleshoot this since it will generate an error similar to this one (you can verify in your FW that it tries to connect to DNAC Enterprise IP with new IP but onboarding fails):

| ERROR | pool-29-thread-1 | | c.c.e.i.impl.InventoryServiceImpl | syncDeviceWithEvent got request for ip <whatever_ip_you_had_during_dhcp_discovery> which does not exist. Ingnoring sync request as deviceId:null

You can also debug your onboarding and find logs similar to:

| DEBUG | SimpleAsyncTaskExecutor-1 | onboarding-service | c.c.g.a.i.GrapevineMessageListener | Finished processing message ZtdDeviceDomainConfigResponseMessage {context={public={AuditParentCorrelationContext=null, AuditSourceIp=<whatever_ip_you_had_during_dhcp_discovery> , RBACSecurityContext=<omitted>, accept-language=en_US, AuditCorrelationContext=6a2bb134-e60c-4f0d-884b-81cf519b501c}}, replyToChain=null, version=0, payload=Request [deviceId=6689c2ff413fc348ec3105e8, taskId=6a2bb134-e60c-4f0d-884b-81cf519b501c, configFormat=DEFAULT, siteConfig=

For PNP to actually complete and download site-specific configuration and be added to DNAC inventory, the pnp-device will need to call DNAC and request the above task and authenticate using the right tokens. This is where it generally fails if you try to switch DHCP-ip address.

And your config-archive logs will look like this:

| INFO | SimpleAsyncTaskExecutor-1 | | c.c.a.n.r.PnpDeleteNotificationHandler | Device Replacement in waiting state after PNP = null | NET-REFRESH[New:<serialnumber>] |
| INFO | SimpleAsyncTaskExecutor-2 | | c.c.a.n.r.PnpUnclaimedNotificationHandler | Checking DR Table if device with serial no <serialnumber> waiting for network refresh |

And your connection-manager-service logs will look like this:

| DEBUG | qtp293474277-44 | connection-manager-service | c.c.p.service.RestMessagingService | RestMessagingService.fetchCommand request body: {"protocol":"HTTPS","clientAddress":"<whatever_ip_you_had_during_dhcp_discovery> ","serialNumber":"<serialnumber>","sudiSerialNumber":"<serialnumber>,"platformId":"C9300-48U","correlatorId":"CiscoPnP-1.0-R35.201014-I46-P353-T1906807-44","macAddress":"","hostname":"Switch","authRequired":false,"lastProcessedCmdId":"65fbd7e3-e3e4-4191-8979-7b23643657ce","requestId":"953e15d47a55451eb6a1467a29242945","authStatus":"AUTHENTICATED","sessionId":"660d4ca3-2579-4d29-b3e6-4af6b32edd94","startupConfigPresent":false} | request-id=953e15d47a55451eb6a1467a29242945, sn=FCW2124L07H, address=<whatever_ip_you_had_during_dhcp_discovery>

| DEBUG | AsyncHttpClient-3-2 | connection-manager-service | c.c.p.service.RestMessagingService | Onboarding service returned response code 200 response message OK content {"BackoffCmd":{"cmdId":"cd3d3a18-f1d2-45d5-a647-e8a38102bf0c","serialNumber":"<serialnumber>","maxRetries":0,"ignore":false,"delay":59,"mode":"NONE"}} | sn=<serialnumber> address=<whatever_ip_you_had_during_dhcp_discovery> , request-id=953e15d47a55451eb6a1467a29242945

You can use APIs to look through all the work items and task-ids and you will see that there is a direct correlation to claiming a task and the DHCP-Discovered IP-address and the authentication for that task. Simply put, DNAC will not "successfully" onboard the workflow if it doesn't talk to the correct IP during onboarding and post-onboarding.

Anyway for anyone wondering what will always work in all cases, all scenarios..save yourself all your troubles and just do it this way if you want your ZTP/PNP to work over releases:
-Make sure APIC-EM/DNAC can talk with your PNP-DHCP-assigned address and keep it that way.  (PNP device will send a HTTP/HTTPS hello to DNAC and DNAC will respond during PNP and this IP is then bound to DNAC workflows, with site-tasks, config-tasks, onboarding-tasks etc)
-Then add all your VRFs, underlay configuration, Management-ips, VLANs ...anything you need...think ZTP, i personally just configure everything i need so it's really ZTP. Just remember to configure things in the right order (e.g ip addreses on links before routing protocols etc).
-You can move the DHCP-Discovery address around to whichever interface you want...but best option is to just use the IP you already were going to use anyway to save yourself trouble. (the trick is to use a DHCP-scope that assigns the address you need for your inventory and then just move this to Loopback0, it can be your P2P /30 links, your Mgmt Loopback0 address or whatever your prefer)
-What you *cannot do* is to use a different IP than DNAC wants you to have in all it's databases. (e.g discovery PNP is 1.1.1.1 and you remove this and go with 2.2.2.2 e.g with a http forceclose and update-source)

The above have worked in all releases since 1.x of DNAC and earlier in APIC-EM, having different IPs for PNP and various tricks for opening new HTTP/HTTPS sessions have always been a lot of trouble with bugs over the releases.

Or simply...
Edit the inventory manually to the Loopback-interface, this is supported in all topologies and all ZTP-deployments (including routed underlays and advanced configurations with subinterfaces and VRFs).

Onboarding L3-devices is a pain with APIC/DNAC and always has been.
Simply put, DNAC will just have to somehow learn the correct DHCPD ip address in one way or another during PNP.

Last but not least ... Word of Caution with L3 PNP:

DNAC has tons of bugs in all releases and what might work in one release might stop working in another. One single error in the configuration that generates an error message will make the onboarding fail and stop pushing the configuration where the error is....potentially making you unable to manage the device at all so build your management-network first thing and then everything else advanced or you will cut yourself off eventually. I suppose this is the reason cisco pushes you to do "very little" for PNP/Day0.

Configuration that works on a device with "copy-paste" from "factory defaults" will work for ZTP, everything else will fail.

Most documented examples assume you onboard L2-switches, and not e.g supported Routers or L3-switches in routed modes. It works with all sorts of advanced configurations and i personally *rarely* do any other "provisions" except for AAA/Site-specific settings after that. Just beware that DNAC is a monster with Docker Containers and databases all over...and i've had way more issues with Provisioning database errors than I have ever had with ZTP/PNP databases.