cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements

Ask the Expert- SD-WAN

Certificate change on tools.cisco.com on October 5th 2018 causes multiple products to fail to register/license

5529
Views
20
Helpful
4
Comments

Overview

After a recent change to the certificate authority used to sign the certificate on tools.cisco.com, multiple system that rely on that server may fail to trust the certificate presented. This may manifest in many ways depending on the product or feature leveraging tools.cisco.com. Commonly, Smart Licensing registration or Smart Call Home may fail to connect and operate as expected. Refer to the products/services outline below for information on how to work around this certificate change.

 

The Certificate Authority that signed the certificate has changed to QuoVadis Root CA 2 and that CA certificate is available here:

 

-----BEGIN CERTIFICATE-----
MIIFtzCCA5+gAwIBAgICBQkwDQYJKoZIhvcNAQEFBQAwRTELMAkGA1UEBhMCQk0x
GTAXBgNVBAoTEFF1b1ZhZGlzIExpbWl0ZWQxGzAZBgNVBAMTElF1b1ZhZGlzIFJv
b3QgQ0EgMjAeFw0wNjExMjQxODI3MDBaFw0zMTExMjQxODIzMzNaMEUxCzAJBgNV
BAYTAkJNMRkwFwYDVQQKExBRdW9WYWRpcyBMaW1pdGVkMRswGQYDVQQDExJRdW9W
YWRpcyBSb290IENBIDIwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCa
GMpLlA0ALa8DKYrwD4HIrkwZhR0In6spRIXzL4GtMh6QRr+jhiYaHv5+HBg6XJxg
Fyo6dIMzMH1hVBHL7avg5tKifvVrbxi3Cgst/ek+7wrGsxDp3MJGF/hd/aTa/55J
WpzmM+Yklvc/ulsrHHo1wtZn/qtmUIttKGAr79dgw8eTvI02kfN/+NsRE8Scd3bB
rrcCaoF6qUWD4gXmuVbBlDePSHFjIuwXZQeVikvfj8ZaCuWw419eaxGrDPmF60Tp
+ARz8un+XJiM9XOva7R+zdRcAitMOeGylZUtQofX1bOQQ7dsE/He3fbE+Ik/0XX1
ksOR1YqI0JDs3G3eicJlcZaLDQP9nL9bFqyS2+r+eXyt66/3FsvbzSUr5R/7mp/i
Ucw6UwxI5g69ybR2BlLmEROFcmMDBOAENisgGQLodKcftslWZvB1JdxnwQ5hYIiz
PtGo/KPaHbDRsSNU30R2be1B2MGyIrZTHN81Hdyhdyox5C315eXbyOD/5YDXC2Og
/zOhD7osFRXql7PSorW+8oyWHhqPHWykYTe5hnMz15eWniN9gqRMgeKh0bpnX5UH
oycR7hYQe7xFSkyyBNKr79X9DFHOUGoIMfmR2gyPZFwDwzqLID9ujWc9Otb+fVuI
yV77zGHcizN300QyNQliBJIWENieJ0f7OyHj+OsdWwIDAQABo4GwMIGtMA8GA1Ud
EwEB/wQFMAMBAf8wCwYDVR0PBAQDAgEGMB0GA1UdDgQWBBQahGK8SEwzJQTU7tD2
A8QZRtGUazBuBgNVHSMEZzBlgBQahGK8SEwzJQTU7tD2A8QZRtGUa6FJpEcwRTEL
MAkGA1UEBhMCQk0xGTAXBgNVBAoTEFF1b1ZhZGlzIExpbWl0ZWQxGzAZBgNVBAMT
ElF1b1ZhZGlzIFJvb3QgQ0EgMoICBQkwDQYJKoZIhvcNAQEFBQADggIBAD4KFk2f
BluornFdLwUvZ+YTRYPENvbzwCYMDbVHZF34tHLJRqUDGCdViXh9duqWNIAXINzn
g/iN/Ae42l9NLmeyhP3ZRPx3UIHmfLTJDQtyU/h2BwdBR5YM++CCJpNVjP4iH2Bl
fF/nJrP3MpCYUNQ3cVX2kiF495V5+vgtJodmVjB3pjd4M1IQWK4/YY7yarHvGH5K
WWPKjaJW1acvvFYfzznB4vsKqBUsfU16Y8Zsl0Q80m/DShcK+JDSV6IZUaUtl0Ha
B0+pUNqQjZRG4T7wlP0QADj1O+hA4bRuVhogzG9Yje0uRY/W6ZM/57Es3zrWIozc
hLsib9D45MY56QSIPMO661V6bYCZJPVsAfv4l7CUW+v90m/xd2gNNWQjrLhVoQPR
TUIZ3Ph1WVaj+ahJefivDrkRoHy3au000LYmYjgahwz46P0u05B/B5EqHdZ+XIWD
mbA4CD/pXvk1B+TJYm5Xf6dQlfe6yJvmjqIBxdZmv3lh8zwc4bmCXF2gw+nYSL0Z
ohEUGW6yhhtoPkg3Goi3XZZenMfvJ2II4pEZXNLxId26F0KCl3GBUzGpn/Z9Yr9y
4aOTHcyKJloJONDO1w2AFrR4pTqHTI2KpdVGl/IsELm8VCLAAVBpQ570su9t+Oza
8eOx79+Rj1QqCyXBJhnEUhAFZdWCEOrCMc0u
-----END CERTIFICATE-----

 

FXOS based systems (Firepower 4100 and Firepower 9300 systems)

Affected functions/features: Smart Licensing 

Cisco Bug ID: CSCvm81014

Symptoms: Smart licensing may fail to register (as seen in chassis manager) indicating that there was a failure when trying to authenticate the server. Specifically the failure reason will state: "Failed to authenticate server" The output of show license all may look like the following:

4100CHASSIS # show license all

Smart Licensing Status
======================

Smart Licensing is ENABLED

Registration:
  Status: REGISTERING - REGISTRATION IN PROGRESS
  Export-Controlled Functionality: Not Allowed
  Initial Registration: FAILED on Oct 09 18:03:27 2018 UTC
    Failure reason: Failed to authenticate server
  Next Registration Attempt: Oct 09 18:18:39 2018 UTC

 

Workaround: For the FXOS based platforms, adding the certificate to the chassis's trust store will allow the Smart Licensing Agent to validate the certificate from the cisco.com server. This is done via the CLI by SSH'ing to the chassis and changing into the security scope, adding a trustpoint, and then executing commit-buffer. Example:

4100CHASSIS #
4100CHASSIS # scope security
4100CHASSIS /security # create trustpoint QuoVadisRootCA2
4100CHASSIS /security/trustpoint* # set certchain
Enter lines one at a time. Enter ENDOFBUF to finish. Press ^C to abort.
Trustpoint Certificate Chain:
>
At this point. Paste certificate listed above, including all of the leading and ending hyphens. Once pasted in, enter ENDOFBUF and hit Enter. Save the change with the command commit-buffer.
>
>ENDOFBUF
4100CHASSIS /security/trustpoint* # commit-buffer
4100CHASSIS /security/trustpoint # end
4100CHASSIS #

When the Smart Licensing system re-attempts connection to the service, it should succeed.

 

 

ASA Software on multiple platforms

Affected functions/features: Smart Licensing, Smart Call Home

Cisco Bug ID: CSCvm80874

Symptoms: Smart Licensing may fail to register (as seen in chassis manager) indicating that there was a failure when trying to communicate with the service. In addition Smart Call Home functionality may also be impacted. The output of the command show license registration will indicate an error: 

ASAv# show license registration
        Registration Status: Retry In Progress.
        Registration Start Time: Mar 22 13:25:46 2016 UTC
        Registration Status: Retry In Progress.
        Registration Start Time: Mar 22 13:25:46 2016 UTC
        Last Retry Start Time: Mar 22 13:26:32 2016 UTC.
        Next Scheduled Retry Time: Mar 22 13:45:31 2016 UTC.
        Number of Retries: 1.
        Last License Server response time: Mar 22 13:26:32 2016 UTC.
        Last License Server response message: Communication message send response error

 

 

Workaround: To allow the ASA to trust the new certificate, you can manually import it into the ASA's certificate trust store. For example: 

ASA# config t
ASA(config)# crypto ca trustpoint QuoVadisRootCA2  
ASA(config-ca-trustpoint)# enrollment terminal
ASA(config-ca-trustpoint)# crl configure
ASA(config-ca-crl)# crypto ca authenticate QuoVadisRootCA2  
Enter the base 64 encoded CA certificate.
End with the word "quit" on a line by itself
<<PASTE IN THE CERTIFICATE FROM ABOVE, INCLUDING STARTING AND ENDING -'S >>
quit

INFO: Certificate has the following attributes:
Fingerprint:     5e397bdd f8baec82 e9ac62ba 0c54002b 
Do you accept this certificate? [yes/no]: yes

Trustpoint CA certificate accepted.

% Certificate successfully imported

 

On Version 9.5(2) and later, the ASAv platform has the trustpool configured to auto-import at 10:00 PM device local time:

ASAv# sh run crypto ca trustpool
crypto ca trustpool policy
auto-import
ASAv# sh run all crypto ca trustpool
crypto ca trustpool policy
revocation-check none
crl cache-time 60
crl enforcenextupdate
auto-import
auto-import url http://www.cisco.com/security/pki/trs/ios_core.p7b
auto-import time 22:00:00

  

In addition, you can immediately update the local trust store with the following command line (Thanks to @Taisuke Nakamura):

ASA# crypto ca trustpool import url http://www.cisco.com/security/pki/trs/ios_core.p7b

 Note: This command is also available on ASA version 9.5(1) and earlier, which does not support auto-import feature. This command is not supported on multiple context mode.

 

The below is example output of latest trustpool import before new smart license registration.

ASA# crypto ca trustpool import url http://www.cisco.com/security/pki/trs/ios_core.p7b
Root file signature verified.
Trustpool import:
   attempted:  10
   installed:  10
   duplicates: 0
   expired:    0
   failed:     0
ASA#
    

Issues with FIPS enabled on the ASA platform

If FIPS is enabled on the ASA, the certificate listed above may be rejected for not conforming to the signature cryptographic requirements. The QuoVadis Certificate has Signature Algorithm: sha1WithRSAEncryption. You will be shown an error upon import similar to the following:

 

Do you accept this certificate? [yes/no]: yes

Trustpoint CA certificate is not FIPS compliant.
% Error in saving certificate: status = FAIL

 

As a workaround, you may import the intermediate certificate for tools.cisco.com which is the HydrantID SSL ICA G2 certificate. This certificate has Signature Algorithm: sha256WithRSAEncryption

 

Certificate chain for tools.cisco.com:

> openssl s_client -connect tools.cisco.com:443

Certificate chain
 0 s:/C=US/ST=CA/L=San Jose/O=Cisco Systems, Inc./CN=tools.cisco.com
   i:/C=US/O=HydrantID (Avalanche Cloud Corporation)/CN=HydrantID SSL ICA G2
 1 s:/C=US/O=HydrantID (Avalanche Cloud Corporation)/CN=HydrantID SSL ICA G2
   i:/C=BM/O=QuoVadis Limited/CN=QuoVadis Root CA 2
 2 s:/C=BM/O=QuoVadis Limited/CN=QuoVadis Root CA 2
   i:/C=BM/O=QuoVadis Limited/CN=QuoVadis Root CA 2

The HydrantID SSL ICA G2 certificate is available here and can be imported in a similar manner:

 

-----BEGIN CERTIFICATE-----
MIIGxDCCBKygAwIBAgIUdRcWd4PQQ361VsNXlG5FY7jr06wwDQYJKoZIhvcNAQEL
BQAwRTELMAkGA1UEBhMCQk0xGTAXBgNVBAoTEFF1b1ZhZGlzIExpbWl0ZWQxGzAZ
BgNVBAMTElF1b1ZhZGlzIFJvb3QgQ0EgMjAeFw0xMzEyMTcxNDI1MTBaFw0yMzEy
MTcxNDI1MTBaMF4xCzAJBgNVBAYTAlVTMTAwLgYDVQQKEydIeWRyYW50SUQgKEF2
YWxhbmNoZSBDbG91ZCBDb3Jwb3JhdGlvbikxHTAbBgNVBAMTFEh5ZHJhbnRJRCBT
U0wgSUNBIEcyMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA9p1ZOA9+
H+tgdln+STF7bdOxvnOERYyjo8ZbKumzigNePSwbQYVWuso76GI843yjaX2rhn0+
Jt0NVJM41jVctf9qwacVduR7CEi0qJgpAUJyZUuB9IpFWF1Kz14O3Leh6URuRZ43
RzHaRmNtzkxttGBuOtAg+ilOuwiGAo9VQLgdONlqQFcrbp97/fO8ZIqiPrbhLxCZ
fXkYi3mktZVRFKXG62FHAuH1sLDXCKba3avDcUR7ykG4ZXcmp6kl14UKa8JHOHPE
NYyr0R6oHELOGZMox1nQcFwuYMX9sJdAUU/9SQVXyA6u6YtxlpZiC8qhXM1IE00T
Q9+q5ppffSUDMC4V/5If5A6snKVP78M8qd/RMVswcjMUMEnov+wykwCbDLD+IReM
A57XX+HojN+8XFTL9Jwge3z3ZlMwL7E54W3cI7f6cxO5DVwoKxkdk2jRIg37oqSl
SU3z/bA9UXjHcTl/6BoLho2p9rWm6oljANPeQuLHyGJ3hc19N8nDo2IATp70klGP
kd1qhIgrdkki7gBpanMOK98hKMpdQgs+NY4DkaMJqfrHzWR/CYkdyUCivFaepaFS
K78+jVu1oCMOFOnucPXL2fQa3VQn+69+7mA324frjwZj9NzrHjd0a5UP7waPpd9W
2jZoj4b+g+l+XU1SQ+9DWiuZtvfDW++k0BMCAwEAAaOCAZEwggGNMBIGA1UdEwEB
/wQIMAYBAf8CAQAweAYDVR0gBHEwbzAIBgZngQwBAgEwCAYGZ4EMAQICMA4GDCsG
AQQBvlgAAmQBAjBJBgwrBgEEAb5YAAOHBAAwOTA3BggrBgEFBQcCARYraHR0cDov
L3d3dy5oeWRyYW50aWQuY29tL3N1cHBvcnQvcmVwb3NpdG9yeTByBggrBgEFBQcB
AQRmMGQwKgYIKwYBBQUHMAGGHmh0dHA6Ly9vY3NwLnF1b3ZhZGlzZ2xvYmFsLmNv
bTA2BggrBgEFBQcwAoYqaHR0cDovL3RydXN0LnF1b3ZhZGlzZ2xvYmFsLmNvbS9x
dnJjYTIuY3J0MA4GA1UdDwEB/wQEAwIBBjAfBgNVHSMEGDAWgBQahGK8SEwzJQTU
7tD2A8QZRtGUazA5BgNVHR8EMjAwMC6gLKAqhihodHRwOi8vY3JsLnF1b3ZhZGlz
Z2xvYmFsLmNvbS9xdnJjYTIuY3JsMB0GA1UdDgQWBBSYarYtLr+nqp/299YJr9WL
V/mKtzANBgkqhkiG9w0BAQsFAAOCAgEAlraik8EDDUkpAnIOajO9/r4dpj/Zry76
6SH1oYPo7eTGzpDanPMeGMuSmwdjUkFUPALuWwkaDERfz9xdyFL3N8CRg9mQhdtT
3aWQUv/iyXULXT87EgL3b8zzf8fhTS7r654m9WM2W7pFqfimx9qAlFe9XcVlZrUu
9hph+/MfWMrUju+VPL5U7hZvUpg66mS3BaN15rsXv2+Vw6kQsQC/82iJLHvtYVL/
LwbNio18CsinDeyRE0J9wlYDqzcg5rhD0rtX4JEmBzq8yBRvHIB/023o/vIO5oxh
83Hic/2Xgwksf1DKS3/z5nTzhsUIpCpwkN6nHp6gmA8JBXoUlKQz4eYHJCq/ZyC+
BuY2vHpNx6101J5dmy7ps7J7d6mZXzguP3DQN84hjtfwJPqdf+/9RgLriXeFTqwe
snxbk2FsPhwxhiNOH98GSZVvG02v10uHLVaf9B+puYpoUiEqgm1WG5mWW1PxHstu
Ew9jBMcJ6wjQc8He9rSUmrhBr0HyhckdC99RgEvpcZpV2XL4nPPrTI2ki/c9xQb9
kmhVGonSXy5aP+hDC+Ht+bxmc4wN5x+vB02hak8Hh8jIUStRxOsRfJozU0R9ysyP
EZAHFZ3Zivg2BaD4tOISO8/T2FDjG7PNUv0tgPAOKw2t94B+1evrSUhqJDU0Wf9c
9vkaKoPvX4w=
-----END CERTIFICATE-----

 

Comments
Cisco Employee

When you use ASA, which is affected version, you can also use the following command for latest trustpool import immediately. This command is also available on ASA version 9.5(1) or earlier, which does not support auto-import feature.

crypto ca trustpool import url http://www.cisco.com/security/pki/trs/ios_core.p7b

 

The below is example output of latest trustpool import before new smart license registration.

ciscoasa(config)# show version | in Version
Cisco Adaptive Security Appliance Software Version 9.4(4)24
Device Manager Version 7.5(2)61
ciscoasa(config)#
ciscoasa(config)# crypto ca trustpool import url http://www.cisco.com/security/pki/trs/ios_core.p7b
Root file signature verified.
Trustpool import:
   attempted:  10
   installed:  10
   duplicates: 0
   expired:    0
   failed:     0
ciscoasa(config)#
ciscoasa(config)# license smart register idtoken OThiYTZjNjYtMDZjMy00MDQ...<snip>
ciscoasa(config)#
ciscoasa(config)# INFO: ASAv platform license state is Licensed.
ciscoasa(config)#



 

Cisco Employee

@Taisuke Nakamura Great point and easy workaround. I have added that to the document! Thanks!

Cisco Employee

Hi all,

how about AMP Private Cloud?

Cheers

Beginner

Can this type of fix be implemented on ASR9k devices aswell?

And how would one do that?

 

Thanks in advance.