Certificate change on tools.cisco.com on October 5th 2018 causes multiple products to fail to regist...

Magnus Mortensen · ‎10-10-2018

Overview
FXOS based systems (Firepower 4100 and Firepower 9300 systems)
ASA Software on multiple platforms
Issues with FIPS enabled on the ASA platform

Overview

After a recent change to the certificate authority used to sign the certificate on tools.cisco.com, multiple system that rely on that server may fail to trust the certificate presented. This may manifest in many ways depending on the product or feature leveraging tools.cisco.com. Commonly, Smart Licensing registration or Smart Call Home may fail to connect and operate as expected. Refer to the products/services outline below for information on how to work around this certificate change.

The Certificate Authority that signed the certificate has changed to QuoVadis Root CA 2 and that CA certificate is available here:

FXOS based systems (Firepower 4100 and Firepower 9300 systems)

Affected functions/features: Smart Licensing

Cisco Bug ID: CSCvm81014

Symptoms: Smart licensing may fail to register (as seen in chassis manager) indicating that there was a failure when trying to authenticate the server. Specifically the failure reason will state: "Failed to authenticate server" The output of show license all may look like the following:

4100CHASSIS # show license all

Smart Licensing Status
======================

Smart Licensing is ENABLED

Registration:
  Status: REGISTERING - REGISTRATION IN PROGRESS
  Export-Controlled Functionality: Not Allowed
  Initial Registration: FAILED on Oct 09 18:03:27 2018 UTC
    Failure reason: Failed to authenticate server
  Next Registration Attempt: Oct 09 18:18:39 2018 UTC

Workaround: For the FXOS based platforms, adding the certificate to the chassis's trust store will allow the Smart Licensing Agent to validate the certificate from the cisco.com server. This is done via the CLI by SSH'ing to the chassis and changing into the security scope, adding a trustpoint, and then executing commit-buffer. Example:

4100CHASSIS #
4100CHASSIS # scope security
4100CHASSIS /security # create trustpoint QuoVadisRootCA2
4100CHASSIS /security/trustpoint* # set certchain
Enter lines one at a time. Enter ENDOFBUF to finish. Press ^C to abort.
Trustpoint Certificate Chain:
>

At this point. Paste certificate listed above, including all of the leading and ending hyphens. Once pasted in, enter ENDOFBUF and hit Enter. Save the change with the command commit-buffer.

>
>ENDOFBUF
4100CHASSIS /security/trustpoint* # commit-buffer
4100CHASSIS /security/trustpoint # end
4100CHASSIS #

When the Smart Licensing system re-attempts connection to the service, it should succeed.

ASA Software on multiple platforms

Affected functions/features: Smart Licensing, Smart Call Home

Cisco Bug ID: CSCvm80874

Symptoms: Smart Licensing may fail to register (as seen in chassis manager) indicating that there was a failure when trying to communicate with the service. In addition Smart Call Home functionality may also be impacted. The output of the command show license registration will indicate an error:

ASAv# show license registration
        Registration Status: Retry In Progress.
        Registration Start Time: Mar 22 13:25:46 2016 UTC
        Registration Status: Retry In Progress.
        Registration Start Time: Mar 22 13:25:46 2016 UTC
        Last Retry Start Time: Mar 22 13:26:32 2016 UTC.
        Next Scheduled Retry Time: Mar 22 13:45:31 2016 UTC.
        Number of Retries: 1.
        Last License Server response time: Mar 22 13:26:32 2016 UTC.
        Last License Server response message: Communication message send response error

Workaround: To allow the ASA to trust the new certificate, you can manually import it into the ASA's certificate trust store. For example:

ASA# config t
ASA(config)# crypto ca trustpoint QuoVadisRootCA2  
ASA(config-ca-trustpoint)# enrollment terminal
ASA(config-ca-trustpoint)# crl configure
ASA(config-ca-crl)# crypto ca authenticate QuoVadisRootCA2  
Enter the base 64 encoded CA certificate.
End with the word "quit" on a line by itself
<<PASTE IN THE CERTIFICATE FROM ABOVE, INCLUDING STARTING AND ENDING -'S >>
quit

INFO: Certificate has the following attributes:
Fingerprint:     5e397bdd f8baec82 e9ac62ba 0c54002b 
Do you accept this certificate? [yes/no]: yes

Trustpoint CA certificate accepted.

% Certificate successfully imported

On Version 9.5(2) and later, the ASAv platform has the trustpool configured to auto-import at 10:00 PM device local time:

ASAv# sh run crypto ca trustpool
crypto ca trustpool policy
 auto-import
ASAv# sh run all crypto ca trustpool
crypto ca trustpool policy
 revocation-check none
 crl cache-time 60
 crl enforcenextupdate
 auto-import
 auto-import url http://www.cisco.com/security/pki/trs/ios_core.p7b
 auto-import time 22:00:00

In addition, you can immediately update the local trust store with the following command line (Thanks to @Taisuke Nakamura):

ASA# crypto ca trustpool import url http://www.cisco.com/security/pki/trs/ios_core.p7b

Note: This command is also available on ASA version 9.5(1) and earlier, which does not support auto-import feature. This command is not supported on multiple context mode.

The below is example output of latest trustpool import before new smart license registration.

ASA# crypto ca trustpool import url http://www.cisco.com/security/pki/trs/ios_core.p7b
Root file signature verified.
Trustpool import:
   attempted:  10
   installed:  10
   duplicates: 0
   expired:    0
   failed:     0
ASA#

Issues with FIPS enabled on the ASA platform

If FIPS is enabled on the ASA, the certificate listed above may be rejected for not conforming to the signature cryptographic requirements. The QuoVadis Certificate has Signature Algorithm: sha1WithRSAEncryption. You will be shown an error upon import similar to the following:

Do you accept this certificate? [yes/no]: yes

Trustpoint CA certificate is not FIPS compliant.
% Error in saving certificate: status = FAIL

As a workaround, you may import the intermediate certificate for tools.cisco.com which is the HydrantID SSL ICA G2 certificate. This certificate has Signature Algorithm: sha256WithRSAEncryption

Certificate chain for tools.cisco.com:

> openssl s_client -connect tools.cisco.com:443

Certificate chain
 0 s:/C=US/ST=CA/L=San Jose/O=Cisco Systems, Inc./CN=tools.cisco.com
   i:/C=US/O=HydrantID (Avalanche Cloud Corporation)/CN=HydrantID SSL ICA G2
 1 s:/C=US/O=HydrantID (Avalanche Cloud Corporation)/CN=HydrantID SSL ICA G2
   i:/C=BM/O=QuoVadis Limited/CN=QuoVadis Root CA 2
 2 s:/C=BM/O=QuoVadis Limited/CN=QuoVadis Root CA 2
   i:/C=BM/O=QuoVadis Limited/CN=QuoVadis Root CA 2

The HydrantID SSL ICA G2 certificate is available here and can be imported in a similar manner:

Taisuke Nakamura · ‎10-26-2018

When you use ASA, which is affected version, you can also use the following command for latest trustpool import immediately. This command is also available on ASA version 9.5(1) or earlier, which does not support auto-import feature.

crypto ca trustpool import url http://www.cisco.com/security/pki/trs/ios_core.p7b

The below is example output of latest trustpool import before new smart license registration.

ciscoasa(config)# show version | in Version
Cisco Adaptive Security Appliance Software Version 9.4(4)24
Device Manager Version 7.5(2)61
ciscoasa(config)#
ciscoasa(config)# crypto ca trustpool import url http://www.cisco.com/security/pki/trs/ios_core.p7b
Root file signature verified.
Trustpool import:
   attempted:  10
   installed:  10
   duplicates: 0
   expired:    0
   failed:     0
ciscoasa(config)#
ciscoasa(config)# license smart register idtoken OThiYTZjNjYtMDZjMy00MDQ...<snip>
ciscoasa(config)#
ciscoasa(config)# INFO: ASAv platform license state is Licensed.
ciscoasa(config)#

Magnus Mortensen · ‎10-26-2018

@Taisuke Nakamura Great point and easy workaround. I have added that to the document! Thanks!

Troja007 · ‎12-12-2018

Hi all,

how about AMP Private Cloud?

Cheers

andreaswall · ‎12-18-2018

Can this type of fix be implemented on ASR9k devices aswell?

And how would one do that?

Thanks in advance.