Solved: Cisco Business Dashboard Certificate problems

Niels van Kampen · ‎10-13-2020

Hi all,

I've been working on a test setup with Business Dashboard and am running into some problems I can't seem to solve.

The setup:

Cisco Business Dashboard running in Azure on Ubuntu 16.04
Network Equipment
- RV340 router RV340-K9-G5
- 2x Business 250 Smart switches (1x POE, 1x non-POE) CBS250-8T-E-2G-EU and CBS250-8P-E-2G-EU
- CBW140AC-E access point

What I'm trying to archieve:

Redirect the devices to Business Dashboard using Plug and Play Connect (software.cisco.com)
Claim and provision the devices in Business Dashboard
Manage the devices from Business Dashboard

The problem:

I'm running into certificate problems.

When using a self-signed certificate created by Business Dashboard redirection from Plug and Play Connect works fine. The device shows up in Business Dashboard and I can provision it. But after provisioning the device remains offline. The error on the device itself noted the certificate of Business Dashboard was invalid. It seems that after provisioning the device loses the certificate pushed by Plug and Play Connect.

I then purchased a certicate signed by a public CA (Sectigo), as soon as I uploaded this certificate to Business Dashboard all devices were reporting online and everything was working fine.

To verify the entire process was working I reset alle devices back to factory defaults. I uploaded the Business Dashboard certificate to my Plug and Play Connect controller profile and booted up the first device.

The redirection from Plug and Play Connect now keeps failing on certificate errors.

I've tried uploading the following combinations of certificates to the Plug and Play Controller profile:

Only the server certificate (downloaded from Business Dashboard)
Only the server certificate (converted the crt received from the certificate broker to pem)
The full chain (root, intermediate 1, intermediate 2, server)
Partial chain 1 (root, intermediate 1, intermediate 2)
Partial chain 2 (intermediate 2, server)
Partial chain 3 (intermediate 1, intermediate 2, server)
Only the root certificate

I'm starting to lose hope, does anyone have any thoughts on how to fix this?

---
If my answer was useful please mark it as helpful.

Niels van Kampen · ‎10-15-2020

So I fixed part of the problem.

I took one of the switches and started trying more combinations of certificates to see if it would work. Eventually by uploading only 1st intermediate in the chain the devices were able to get redirected by plug and play connect and connect to the business dashboard.

Next problem, when only connecting the RV340 it won't come online in business dashboard...

---
If my answer was useful please mark it as helpful.

View solution in original post

Niels van Kampen · ‎10-15-2020

So I fixed part of the problem.

I took one of the switches and started trying more combinations of certificates to see if it would work. Eventually by uploading only 1st intermediate in the chain the devices were able to get redirected by plug and play connect and connect to the business dashboard.

Next problem, when only connecting the RV340 it won't come online in business dashboard...

---
If my answer was useful please mark it as helpful.

WesMC · ‎04-02-2021

Niels:

Can you offer an update and a little more color on your SSL work? For example: What keys and/or certificates did you upload to a router or switch vs the CBD itself? Did the CBD end up detecting certificates that it subsequently helped to deploy to additional devices?

I have a similar network to yours - RV340, SG350, three 140ACs and a Cisco Business Dashboard instance. Thus far, I have see several threads reporting frustration with SSL configurations - particularly on Cisco Business routers. The threads are often marked as "Solved", but the text is incomplete or at odds with a final of solved.

Here's my context:

I have a public Cert from Let's Encrypt facilitated by HostGator.

Here is the Let's Encrypt Certificate Hierarchy:

I have downloaded the self-signed ISRG Root X1 (isrgrootx1.pem) certificate.
I have also downloaded the the Let's Encrypt R3 (lets-encrypt-r3.pem) certificate, which is signed by the ISRG Root X1.
I was successful registering both of these two certificates directly with the RV340 "as downloaded" - no changes.

I opened the RV340's "Administration | Certificate" page.
I clicked the blue "Import Certificate..." button.
I browsed for one of the downloaded PEM files, entered a Certificate Name (e.g., isrg_root_x1, lets_encrypt_r3) and the certificates loaded without issue.
Once loaded, the certificates added a _CA suffix to the Certificate name (e.g., isrg_root_x1_CA, lets_encrypt_r3_CA).

The downloaded and installed certificate contain the following:

The file isrgrootx1.pem "C=US, O=Internet Security Research Group, CN=ISRG Root X1" is listed as the subject and the issuer (i.e., is a root).
The file lets-encrypt-r3.pem has subject "C=US, O=Let's Encrypt, CN=R3" and is issued by "C=US, O=Internet Security Research Group, CN=ISRG Root X1".

Now enter HostGator:

HostGator is pretty cool, but the documentation leaves a lot to be desired. HostGator automatically handles private key generation and obtains Let's Encrypt certificates - first for my initial domain and then as I introduce subdomains.

My thinking was to introduce the following:

x.com - My domain
corp.x.com - A public domain for obtaining public SSL certs, that would be used exclusively in a private network
abc.corp.x.com - An (internal only) site location (as a nested subdomain).
*.abc.corp.x.com - A wildcard entry for first-level objects at a site.

On account creation, the x.com private key (x.key) and certificate (x.crt) was generated.

I created subdomains corp.x.com and abc.corp.x.com in rapid succession. Hostgator's autogeneration facility gave me a single key (corp_x.key) and single certificate (corp_x.crt) that covered both. [Both corp.x.com and abc..corp.x.com are listed among the "Subject Alternate Names" in the resulting certificate.]

Finally, I created the subdomain * under abc.corp.x.com. This triggered Let's Encrypt autogeneration facility - yielding the private key (wc_abc_corp_x.key) and certificate (wc_abc_corp_x.crt).

At this point, I'm stuck:

I have not been successful installing any of the following directly on the RV340:

raw certificate as a .crt or .pem

x.crt
corp_x.crt
wc_abc_corp_x.crt

A pem file with both the private key and the certificate (e.g., cat x.key x.crt > x.pem)

x.pem
corp_x.pem
wc_abc_corp_x.pem

A pem file with the private key, certificate and lets-encrypt-r3.pem

x_intermed.pem
corp_x_intermed.pem
wc_abc_corp_x_intermed.pem

I have tried loading the above using the RV340 import certificate type "Local Certificate" as well as "CA Certificate".

The RV340 consistently rejects everything with the insanely unhelpful message "Certificate Import Failed". Cisco being Cisco, no details are offered as to the root cause of the failure.

I have not tried leveraging CBD for certificate installation lately. My first attempt bricked the CBD VM. Fortunately, I could roll back to a Time Machine backup.

If you have some combination that works for installing public certificates on an internal Cisco business network ... using any combination of the router, switch and/or CBD ... please let me know. [If you are using a public certificate, but had leverage a local Let's Encrypt instance an delegation, I'd be interested in that too.]

Thanks in advance for any details you can offer!

Wes

WesMC · ‎04-03-2021

I have been successful today using a PKCS12 file format.

It occurred to me that the RV340 might be rejecting a PEM file that included a private key and certificate - since the private key would not be protected in any way.

I decided to create a PKCS12 file containing just the private key and the certificate - no intermediate file (lets-encrypt-r3.pem) and no root file (isrgrootx1.pem). [As noted in my prior post, I was previously able to install the individual intermediate and root certificates on the RV340, individually. The intermediate and root certificates should be more stable than my Let's Encrypt wildcard certificate, which I suspect will need to be updated quarterly. With this in mind, I am trying to keep the pkcs12 file as minimal as possible.]

I used openssl to create the pkcs12 (PFX) file.

openssl pkcs12 -export -out wc-abc-corp-x-com.pfx \
    -inkey wc-abc-corp-x-com.key \
    -in wc-abc-corp-x-com.crt

Where:

wc-abc-corp-x-com.key - This is my private key for *.abc.corp.x.com
wc-abc-corp-x-com.crt - This is my public certificate for *.abc.corp.x.com
When prompted, I supplied a password to protect wc-abc-corp-x-com.pfx

In the RV340:

I opened the RV340's "Administration | Certificate" page.
I clicked the blue "Import Certificate..." button.
I selected "PKCS#12 encoded file" for the type.
I provided a simple certificate name "asterisk".
I browsed for and selected "wc-abc-corp-x-com.pfx".
I clicked the blue "Upload" button.
I provided the password protecting the file when prompted.

The PFX certificate loaded without issue. With the "Certificate Table" displayed, I selected my "asterisk" certificate and clicked on the blue "Select as My Primary Certificate..." button.

On the "System Configuration | System" page, I have:

Host Name: router
Domain Name: abc.corp.x.com

Without rebooting the router, I was able to open https://router.abc.corp.x.com/ in Chrome which displayed the "Connection is secured" icon.

Niels van Kampen · ‎04-07-2021

Sorry for the late reply.

Are you using PnP connect to redirect your devices to CBD or are you manually connecting the devices to CBD?

You shouldn't have to upload private keys to the RV to be able to connect to CBD. The router will try to verify the certificate of CBD so it must be able to verify the issuer of the certificate.

In the case of Lets Encrypt the chain looks like this:

Thru trial and error I found the Root CA has to be present on the device to be able to connect to CBD. The certificate is in PEM format.

The certificate below is the one we push to the devices from PnP Connect, so if you upload this to the device you should be able to connect.

You should only upload a certificate with a private key if you are terminating SSL sessions on the router (and maybe some other scenarios) but it shouldn't be necessary to connect the RV to CBD.

Let me know if this works for you.

---
If my answer was useful please mark it as helpful.

WesMC · ‎04-08-2021

In my case, I had the RV340, SG350 and 140ACs operational before I brought the CBD instance up. CBD discovered all of the above devices. [CBD found the IPs and looked for me to supply credentials, then reassigned devices the same <username, password> as CBD itself.] To date, nothing is enabled under PnP. So, I would say that CBD was pseudo-automatic.

Once CBD came online, it seemed to either set incorrectly or misread a few SG350 port configurations (wrt trunking and VLAN assignments). After a little reconciliation between CBD and the SG350, the two have existed in relative harmony.

There is some nagging frustration wrt the lack of cohesion across Cisco's business line(s). To name a few:

Missing SSH/CLI on the RV340 and 140ACs
A relatively ugly process for getting the 140ACs API IP to be DHCP assigned after an initial install with an auto-generated static IP.
The lack of SSL consistency across these devices - competing approaches on the RV340 and SG350 and absence of SSL management facility on the 140ACs.
Some disparity among fields across devices - e.g,. single vs multiple DNS slots.

Initially, I gave up on configuring the RV340 in a router-on-a-stick configuration. So, the VLANs have a footing on the RV340 (all VLANs required routes to the internet). When I gave up on ROAS, I let the RV340 handle DHCP tactically. I have since moved all DHCP back to the SG350. The VLAN DHCP pools assign an appropriate RV340 VLAN Interface for routing and the fixed address of the RV340 (on VLAN 1) for DNS. The RV340 can still provide multiple DNS per WAN and optionally two WANs - since the SG350 just refers DNS back to the router. [One of these days, I may revisit ROAS. The RV340 is pretty user-friendly at footing the VLANs and handling NAT; but, not so friendly for ROAS configuration.]

My final SSL arrangement looks like this:

RV340 - per my earlier post, the PKCS#12 file provided (as a package):

isrgrootx1.pem and lets-encrypt-r3.pem from the CA
wc-abc-corp-x-com.key (my private key for *.abc.corp.x.com)
wc-abc-corp-x-com.crt (my public certificate for *.abc.corp.x.com)

SG350

HostGator did not have a downloadable public key; so, I used openssl to generate one from the crt
$ openssl x509 -pubkey -noout -in wc-abc-corp-x-com.crt > wc-abc-corp-x-com.pubkey
The generated public key was in the third-party public key format; so, I had to modify this pubkey to an RSA key format.

Change BEGIN/END PUBLIC KEY to BEGIN RSA PUBLIC KEY-----
Remove the first 32 characters of the public key body

On the "Security | SSL Server | SSL Server Authent.." page, I:

Selected "Certificate ID" 1 or 2 and clicked "Import Certificate".
Cut and pasted wc-abc-corp-x-com.crt into the Certificate window.
Enabled Import RSA Key-Pair
Cut and pasted wc-abc-corp-x-com.pubkey into the Public Key window
Cut and pasted wc-abc-corp-x-com.key into the Private Key Plaintext window - i.e., let the SG350 handle encrypting the private text.

The CA certificates are not required on the SG350. I suspect that they are being resolved by the RV340.

140ACs

No solution (yet) for installing my CA-provided public key.

I can't say I look forward to navigating this mess on a quarterly basis. In the absence of a cohesive solution from Cisco, I may roll some home-grown code to fetch files from HostGator, manipulate them (yuck) and force feed the device GUIs (double yuck) where the devices lack a solid CLI.

Niels van Kampen · ‎04-08-2021

So you have CBD running in the local network and you're using the probe in CBD?

That's a different setup from what I have (CBD in Azure, probe running on the CBS switches). In my case the CBD certificate get's pushed to the devices by PnP connect and to ensure the certificates remain on the devices I include them in the config templates.

I've used Mustache "loops" in the configuration template for the RV340 to configure multiple vlans/subnets which I think could also work for the ROAS setup.

On the interface config to tag the vlans:

    <interface>
      <vlan-settings xmlns="http://cisco.com/ns/ciscosb/vlan">
        <untagged-vlan-id>1</untagged-vlan-id>
        <tagged-vlans>
        {{#networks}}
          <vlan-id>{{vlanid}}</vlan-id>
        {{/networks}}
        </tagged-vlans>         
      </vlan-settings>
    </interface>

And to create the SVI's:

{{#networks}}
    <interface>
      <name>VLAN{{vlanid}}</name>
      <description>VLAN{{vlanid}}</description>
      <type xmlns:ianaift="urn:ietf:params:xml:ns:yang:iana-if-type">ianaift:l2vlan</type>
      <enabled>true</enabled>
      <bonjour xmlns="http://cisco.com/ns/ciscosb/bonjour">
        <enable>true</enable>
      </bonjour>
      <ddns xmlns="http://cisco.com/ns/ciscosb/ddns">
        <username></username>
        <periodic-update-interval>0</periodic-update-interval>
        <enable>false</enable>
      </ddns>
      <dot1x xmlns="http://cisco.com/ns/ciscosb/lan-dot1x">
        <enable>true</enable>
        <authorized-mode>force</authorized-mode>
      </dot1x>
      <lldp xmlns="http://cisco.com/ns/ciscosb/lldp">
        <enable>true</enable>
        <transmit>true</transmit>
        <receive>true</receive>
      </lldp>
      <redundant-wan xmlns="http://cisco.com/ns/ciscosb/redundant-wan">
        <enable>false</enable>
        <net-tracker>
          <enable>true</enable>
          <min-hosts-reachable>1</min-hosts-reachable>
        </net-tracker>
        <precedence>1</precedence>
      </redundant-wan>
      <vlan-interface-settings xmlns="http://cisco.com/ns/ciscosb/vlan">
        <vlan-id>{{vlanid}}</vlan-id>
        <inter-vlan-routing>true</inter-vlan-routing>
        <enable-management>false</enable-management>
      </vlan-interface-settings>
      <ipv4 xmlns="urn:ietf:params:xml:ns:yang:ietf-ip">
        <enabled>true</enabled>
        <forwarding>false</forwarding>
        <address>
          <ip>{{int-address}}</ip>
          <prefix-length>{{mask}}</prefix-length>
        </address>
      </ipv4>
      <ipv6 xmlns="urn:ietf:params:xml:ns:yang:ietf-ip">
        <enabled>true</enabled>
        <forwarding>false</forwarding>
        <address>
          <ip>fec0::1</ip>
          <prefix-length>64</prefix-length>
        </address>
        <dup-addr-detect-transmits>1</dup-addr-detect-transmits>
        <autoconf>
          <create-global-addresses>true</create-global-addresses>
          <create-temporary-addresses>false</create-temporary-addresses>
          <temporary-valid-lifetime>604800</temporary-valid-lifetime>
          <temporary-preferred-lifetime>86400</temporary-preferred-lifetime>
        </autoconf>
        <setting xmlns="http://cisco.com/ns/ciscosb/wan-ip">
          <dhcp-pd-enable>false</dhcp-pd-enable>
        </setting>
        <ipv6-router-advertisements xmlns="urn:ietf:params:xml:ns:yang:ietf-ipv6-unicast-routing">
          <send-advertisements>false</send-advertisements>
          <max-rtr-adv-interval>30</max-rtr-adv-interval>
          <managed-flag>false</managed-flag>
          <other-config-flag>false</other-config-flag>
          <link-mtu>1500</link-mtu>
          <reachable-time>0</reachable-time>
          <retrans-timer>0</retrans-timer>
          <default-lifetime>3600</default-lifetime>
          <prefix-list>
            <prefix>
              <prefix-spec>fec0:{{vlanid}}::/64</prefix-spec>
              <valid-lifetime>7200</valid-lifetime>
              <on-link-flag>true</on-link-flag>
              <preferred-lifetime>3600</preferred-lifetime>
              <autonomous-flag>true</autonomous-flag>
            </prefix>
          </prefix-list>
          <router-preference xmlns="http://cisco.com/ns/ciscosb/routing">high</router-preference>
          <mode xmlns="http://cisco.com/ns/ciscosb/routing">unsolicited-multicast</mode>
        </ipv6-router-advertisements>
      </ipv6>
    </interface>
{{/networks}}

IPv6 functionality hasn't been tested and DHCP pools still have to be configured (which could also easily be done with Mustache).

And you would have configure the RV340 to use CBD as it's Pnp server to be able to use the configuration templates.

---
If my answer was useful please mark it as helpful.

WesMC · ‎04-08-2021

That's not a bad approach. I'm not fully convinced that the CBD's extract of the SG350 is sufficient to rebuild the switch from - less worried about the 140ACs as they are essentially slaves to the switch.

What's the best approach - something like this?

Save all device configs locally (i.e., in case I have to fall back).
Sync everything that can be sync'ed back to CBD.
Factory reset everything.
Have CBD discover and configure all devices via PnP.

I'll have to do some reading on PnP.