DNA Center LAN Automation Problem

[AminK]

Hello Community.

In my workplace we currently are on DNA Center v2.3.2.5 and have a running SD-Access fabric. In past we could easily use LAN automation feature to add new Edge switches in our building.

Recently our DNA Center certificate expired (Which was self-signed), So I generated a CSR with openssl in maglev and got a new certificate from our CA server and installed it on DNA.

I don't know it's because of that or anything else but last week we found out that LAN automation is not working anymore.

Here is what happens when I'm trying to do it:

1) DNA provision seed devices successfully, 2) I reload new switch with #pnpa service reset no-prompt, 3) I connect uplink to my intermediate switch, 4) Switch will boot up and gets ip from DHCP and contacts DNA Center, 5) it gets some info from it like CA chain, 6) then there will be this line and nothing else after that:

AUTOINSTALL: script execution not successful for Vl1.

I tested it several times with multiple switches and researched forums with no luck.

The only reasonable log that I found from switch pnp-summary was these lines:

mgmt_dhcp_enable:Enter 
mgmt_dhcp_enable:no vbond support 
RAC: Abort auto-install.
dhcp_autoinstall_abort: Abort auto-install

Also it worth mentioning that there is no firewall or ACL between DNA and new switch and after process failed, I tested my ping and telnet to DNA center and they were working correctly.

Thank you for your help.

[pieterh]

>>> So I generated a CSR with openssl in maglev and got a new certificate from our CA server and installed it on DNA. <<<
so this is not a self-signed certificte anymore !

did you follow the steps described in this document ? 
https://www.cisco.com/c/en/us/td/docs/cloud-systems-management/network-automation-and-management/dna-center/hardening_guide/b_cisco_catalyst_center_security_best_practices_guide.html 

first review the .csr's contains all the required ip-adresses and dns aliasses

Step 5

Verify the Certificate Signing Request content and ensure that the DNS names are populated correctly in the subjectAltName field.. openssl req -text -noout -verify -in server-cert.csr

check again if the are correcly populated in the retreived certificate from the CA.

[AminK]

I did follow cisco practices in filling out Subject alternative names. Including all FQDNs (3 for appliances and 1 for cluster) and also all IP addresses. And just to be on safe side, I also copied all previously Subject alternative names from old self signed certificate. Which was bunch of IPs like localhost.

[pieterh]

You might need to inform your SD-fabric that DNA is using a different certificate?

Something like re-registering to the fabric?

Google-AI suggests these steps:

After changing a certificate on a Cisco DNA Center or in an SD-Access fabric, you need to ensure the new certificate is properly propagated and validated across all devices involved. This involves several steps, including replacing the old certificate with the new one and verifying that the new certificate is trusted by all devices. Additionally, you might need to reboot or restart specific components to ensure the changes take effect. 

Specific Steps for Cisco DNA Center:

1. 1. Upload and Replace the New Certificate:

   • Navigate to "System > Settings > Trust & Privacy > System Certificates" in DNA Center.
   • Click "Replace Certificate" and choose the file containing the new certificate and its chain.
   • Save the changes.
2. 2. DNA Center will update the DNAC-CA Trustpoint on Managed Devices:

   • Once the system certificate is installed, DNA Center will update the "DNAC-CA" trustpoint on all managed network devices.
   • This ensures that the new certificate, and the internal PKI Root CA certificate, is trusted by the devices. 

Specific Steps for SD-Access:

1. 1. Validate Compatibility:

   • Verify that the new certificate and its associated hardware/software are compatible with the SD-Access fabric nodes by referring to the Cisco SD-Access Compatibility Matrix. 
2. 2. Replace the Certificate:

   • In DNA Center, manage the certificates using the "System > Settings > Certificates" path.
   • Follow the instructions for managing device certificates, trusted certificates, and certificate authority. 
3. 3. Forceful Replacement (If Needed):

   • If the new certificate is not immediately recognized, you might need to reboot the devices or restart specific services/applications. 

Verifying Certificate Changes:

1. 1. Validation:

   • Use the Validation tool in DNA Center to validate that the new certificates are properly installed and trusted by the network.
   • Check system event notifications for any issues related to certificate validity.
2. 2. Device-Specific Verification:

   • Log into individual devices managed by DNA Center and verify that the new certificate is present and trusted. 

Important Considerations:

• Rebooting/Restarting:

  For some certificates, restarting the Service Fabric nodes or rebooting the host machines might be necessary to force the immediate replacement of the certificate.
• Validation Rules:

  Be cautious when introducing a new certificate. Ensure it meets the validation rules of the SD-Access fabric, or it might break the cluster. 

[AminK]

I was thinking about this too. But after LAN Automation failed, I consoled to switch and run command of #show crypto pki certificate and my new certificate was there. Which means Switch got certificate and then it failed automation.

[Parthiv Shah]

Hi

Based on initial information, DnA Center running Version 2.3.2.5 is very old and end of support. Would request to upgrade to supported version - to latest 2.3.7.9 version and then open tac support case.

Thanks

[AminK]

I would be happy to do that. But there is another problem here. Cisco doesn't offer offline upgrade for DNAC. And currently DNAC suggests upgrading to 2.3.3.7. I don't know if it is going to work or not, and all Upgrade guide documents says path from 2.3.3.x not 2.3.2.x. For this reason I'm afraid to break my DNA Center and SDA Fabric. So that gets me to not upgrading it. And furthermore, LAN Automation was working before so I am thinking maybe it has something to do with switch, According to logs.

[pieterh]

I agree the version may not be the cause of your lan automation problem.

but about off-line upgrades..... are you sure ?
I myself have done an offline upgrade once, but do not recall what version
please take a look at this document
https://www.cisco.com/c/en/us/td/docs/cloud-systems-management/network-automation-and-management/dna-center/air_gap_deployment_guide/b_air_gap_deployment_guide/m_222x_or_223x_to_233x.html 

that containss the paragraphs:

Offline Update Workflow

An offline Catalyst Center update involves the following steps:

1. Raise a TAC request to get access to the image for the air gap/offline update.

2. Download the Catalyst Center binary image from a Cisco file server (requires access to the internet).

3. Verify the integrity of the downloaded image.

4. Transfer the downloaded image to the Catalyst Center cluster in the secure, air gap environment.

5. SSH to the Catalyst Center cluster and execute the binary.

6. Log in to the Catalyst Center GUI and perform a system update and an applications update.

Perform an Offline Update
This section applies only if you are upgrading from Cisco DNA Center 2.2.2.x or 2.2.3.x to 2.3.3.x.
If you are on a release earlier than 2.2.2.x, you must first upgrade to at least 2.2.2.x before completing the following steps.

so you are good with your running version v2.3.2.5 
be aware you need to make the upgradeto the final release  in multiple steps!
read carefully the possible upgrade paths.