10-10-2018 07:30 AM - edited 02-21-2020 10:02 PM
After a recent change to the certificate authority used to sign the certificate on tools.cisco.com, multiple system that rely on that server may fail to trust the certificate presented. This may manifest in many ways depending on the product or feature leveraging tools.cisco.com. Commonly, Smart Licensing registration or Smart Call Home may fail to connect and operate as expected. Refer to the products/services outline below for information on how to work around this certificate change.
The Certificate Authority that signed the certificate has changed to QuoVadis Root CA 2 and that CA certificate is available here:
-----BEGIN CERTIFICATE----- MIIFtzCCA5+gAwIBAgICBQkwDQYJKoZIhvcNAQEFBQAwRTELMAkGA1UEBhMCQk0x GTAXBgNVBAoTEFF1b1ZhZGlzIExpbWl0ZWQxGzAZBgNVBAMTElF1b1ZhZGlzIFJv b3QgQ0EgMjAeFw0wNjExMjQxODI3MDBaFw0zMTExMjQxODIzMzNaMEUxCzAJBgNV BAYTAkJNMRkwFwYDVQQKExBRdW9WYWRpcyBMaW1pdGVkMRswGQYDVQQDExJRdW9W YWRpcyBSb290IENBIDIwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCa GMpLlA0ALa8DKYrwD4HIrkwZhR0In6spRIXzL4GtMh6QRr+jhiYaHv5+HBg6XJxg Fyo6dIMzMH1hVBHL7avg5tKifvVrbxi3Cgst/ek+7wrGsxDp3MJGF/hd/aTa/55J WpzmM+Yklvc/ulsrHHo1wtZn/qtmUIttKGAr79dgw8eTvI02kfN/+NsRE8Scd3bB rrcCaoF6qUWD4gXmuVbBlDePSHFjIuwXZQeVikvfj8ZaCuWw419eaxGrDPmF60Tp +ARz8un+XJiM9XOva7R+zdRcAitMOeGylZUtQofX1bOQQ7dsE/He3fbE+Ik/0XX1 ksOR1YqI0JDs3G3eicJlcZaLDQP9nL9bFqyS2+r+eXyt66/3FsvbzSUr5R/7mp/i Ucw6UwxI5g69ybR2BlLmEROFcmMDBOAENisgGQLodKcftslWZvB1JdxnwQ5hYIiz PtGo/KPaHbDRsSNU30R2be1B2MGyIrZTHN81Hdyhdyox5C315eXbyOD/5YDXC2Og /zOhD7osFRXql7PSorW+8oyWHhqPHWykYTe5hnMz15eWniN9gqRMgeKh0bpnX5UH oycR7hYQe7xFSkyyBNKr79X9DFHOUGoIMfmR2gyPZFwDwzqLID9ujWc9Otb+fVuI yV77zGHcizN300QyNQliBJIWENieJ0f7OyHj+OsdWwIDAQABo4GwMIGtMA8GA1Ud EwEB/wQFMAMBAf8wCwYDVR0PBAQDAgEGMB0GA1UdDgQWBBQahGK8SEwzJQTU7tD2 A8QZRtGUazBuBgNVHSMEZzBlgBQahGK8SEwzJQTU7tD2A8QZRtGUa6FJpEcwRTEL MAkGA1UEBhMCQk0xGTAXBgNVBAoTEFF1b1ZhZGlzIExpbWl0ZWQxGzAZBgNVBAMT ElF1b1ZhZGlzIFJvb3QgQ0EgMoICBQkwDQYJKoZIhvcNAQEFBQADggIBAD4KFk2f BluornFdLwUvZ+YTRYPENvbzwCYMDbVHZF34tHLJRqUDGCdViXh9duqWNIAXINzn g/iN/Ae42l9NLmeyhP3ZRPx3UIHmfLTJDQtyU/h2BwdBR5YM++CCJpNVjP4iH2Bl fF/nJrP3MpCYUNQ3cVX2kiF495V5+vgtJodmVjB3pjd4M1IQWK4/YY7yarHvGH5K WWPKjaJW1acvvFYfzznB4vsKqBUsfU16Y8Zsl0Q80m/DShcK+JDSV6IZUaUtl0Ha B0+pUNqQjZRG4T7wlP0QADj1O+hA4bRuVhogzG9Yje0uRY/W6ZM/57Es3zrWIozc hLsib9D45MY56QSIPMO661V6bYCZJPVsAfv4l7CUW+v90m/xd2gNNWQjrLhVoQPR TUIZ3Ph1WVaj+ahJefivDrkRoHy3au000LYmYjgahwz46P0u05B/B5EqHdZ+XIWD mbA4CD/pXvk1B+TJYm5Xf6dQlfe6yJvmjqIBxdZmv3lh8zwc4bmCXF2gw+nYSL0Z ohEUGW6yhhtoPkg3Goi3XZZenMfvJ2II4pEZXNLxId26F0KCl3GBUzGpn/Z9Yr9y 4aOTHcyKJloJONDO1w2AFrR4pTqHTI2KpdVGl/IsELm8VCLAAVBpQ570su9t+Oza 8eOx79+Rj1QqCyXBJhnEUhAFZdWCEOrCMc0u -----END CERTIFICATE-----
Affected functions/features: Smart Licensing
Cisco Bug ID: CSCvm81014
Symptoms: Smart licensing may fail to register (as seen in chassis manager) indicating that there was a failure when trying to authenticate the server. Specifically the failure reason will state: "Failed to authenticate server" The output of show license all may look like the following:
4100CHASSIS # show license all Smart Licensing Status ====================== Smart Licensing is ENABLED Registration: Status: REGISTERING - REGISTRATION IN PROGRESS Export-Controlled Functionality: Not Allowed Initial Registration: FAILED on Oct 09 18:03:27 2018 UTC Failure reason: Failed to authenticate server Next Registration Attempt: Oct 09 18:18:39 2018 UTC
Workaround: For the FXOS based platforms, adding the certificate to the chassis's trust store will allow the Smart Licensing Agent to validate the certificate from the cisco.com server. This is done via the CLI by SSH'ing to the chassis and changing into the security scope, adding a trustpoint, and then executing commit-buffer. Example:
4100CHASSIS # 4100CHASSIS # scope security 4100CHASSIS /security # create trustpoint QuoVadisRootCA2 4100CHASSIS /security/trustpoint* # set certchain Enter lines one at a time. Enter ENDOFBUF to finish. Press ^C to abort. Trustpoint Certificate Chain: >At this point. Paste certificate listed above, including all of the leading and ending hyphens. Once pasted in, enter ENDOFBUF and hit Enter. Save the change with the command commit-buffer.
> >ENDOFBUF 4100CHASSIS /security/trustpoint* # commit-buffer 4100CHASSIS /security/trustpoint # end 4100CHASSIS #
When the Smart Licensing system re-attempts connection to the service, it should succeed.
Affected functions/features: Smart Licensing, Smart Call Home
Cisco Bug ID: CSCvm80874
Symptoms: Smart Licensing may fail to register (as seen in chassis manager) indicating that there was a failure when trying to communicate with the service. In addition Smart Call Home functionality may also be impacted. The output of the command show license registration will indicate an error:
ASAv# show license registration Registration Status: Retry In Progress. Registration Start Time: Mar 22 13:25:46 2016 UTC Registration Status: Retry In Progress. Registration Start Time: Mar 22 13:25:46 2016 UTC Last Retry Start Time: Mar 22 13:26:32 2016 UTC. Next Scheduled Retry Time: Mar 22 13:45:31 2016 UTC. Number of Retries: 1. Last License Server response time: Mar 22 13:26:32 2016 UTC. Last License Server response message: Communication message send response error
Workaround: To allow the ASA to trust the new certificate, you can manually import it into the ASA's certificate trust store. For example:
ASA# config t ASA(config)# crypto ca trustpoint QuoVadisRootCA2 ASA(config-ca-trustpoint)# enrollment terminal ASA(config-ca-trustpoint)# crl configure ASA(config-ca-crl)# crypto ca authenticate QuoVadisRootCA2 Enter the base 64 encoded CA certificate. End with the word "quit" on a line by itself <<PASTE IN THE CERTIFICATE FROM ABOVE, INCLUDING STARTING AND ENDING -'S >> quit INFO: Certificate has the following attributes: Fingerprint: 5e397bdd f8baec82 e9ac62ba 0c54002b Do you accept this certificate? [yes/no]: yes Trustpoint CA certificate accepted. % Certificate successfully imported
On Version 9.5(2) and later, the ASAv platform has the trustpool configured to auto-import at 10:00 PM device local time:
ASAv# sh run crypto ca trustpool
crypto ca trustpool policy
auto-import
ASAv# sh run all crypto ca trustpool
crypto ca trustpool policy
revocation-check none
crl cache-time 60
crl enforcenextupdate
auto-import
auto-import url http://www.cisco.com/security/pki/trs/ios_core.p7b
auto-import time 22:00:00
In addition, you can immediately update the local trust store with the following command line (Thanks to @Taisuke Nakamura):
ASA# crypto ca trustpool import url http://www.cisco.com/security/pki/trs/ios_core.p7b
Note: This command is also available on ASA version 9.5(1) and earlier, which does not support auto-import feature. This command is not supported on multiple context mode.
The below is example output of latest trustpool import before new smart license registration.
ASA# crypto ca trustpool import url http://www.cisco.com/security/pki/trs/ios_core.p7b Root file signature verified. Trustpool import: attempted: 10 installed: 10 duplicates: 0 expired: 0 failed: 0 ASA#
If FIPS is enabled on the ASA, the certificate listed above may be rejected for not conforming to the signature cryptographic requirements. The QuoVadis Certificate has Signature Algorithm: sha1WithRSAEncryption. You will be shown an error upon import similar to the following:
Do you accept this certificate? [yes/no]: yes Trustpoint CA certificate is not FIPS compliant. % Error in saving certificate: status = FAIL
As a workaround, you may import the intermediate certificate for tools.cisco.com which is the HydrantID SSL ICA G2 certificate. This certificate has Signature Algorithm: sha256WithRSAEncryption
Certificate chain for tools.cisco.com:
> openssl s_client -connect tools.cisco.com:443 Certificate chain 0 s:/C=US/ST=CA/L=San Jose/O=Cisco Systems, Inc./CN=tools.cisco.com i:/C=US/O=HydrantID (Avalanche Cloud Corporation)/CN=HydrantID SSL ICA G2 1 s:/C=US/O=HydrantID (Avalanche Cloud Corporation)/CN=HydrantID SSL ICA G2 i:/C=BM/O=QuoVadis Limited/CN=QuoVadis Root CA 2 2 s:/C=BM/O=QuoVadis Limited/CN=QuoVadis Root CA 2 i:/C=BM/O=QuoVadis Limited/CN=QuoVadis Root CA 2
The HydrantID SSL ICA G2 certificate is available here and can be imported in a similar manner:
-----BEGIN CERTIFICATE----- MIIGxDCCBKygAwIBAgIUdRcWd4PQQ361VsNXlG5FY7jr06wwDQYJKoZIhvcNAQEL BQAwRTELMAkGA1UEBhMCQk0xGTAXBgNVBAoTEFF1b1ZhZGlzIExpbWl0ZWQxGzAZ BgNVBAMTElF1b1ZhZGlzIFJvb3QgQ0EgMjAeFw0xMzEyMTcxNDI1MTBaFw0yMzEy MTcxNDI1MTBaMF4xCzAJBgNVBAYTAlVTMTAwLgYDVQQKEydIeWRyYW50SUQgKEF2 YWxhbmNoZSBDbG91ZCBDb3Jwb3JhdGlvbikxHTAbBgNVBAMTFEh5ZHJhbnRJRCBT U0wgSUNBIEcyMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA9p1ZOA9+ H+tgdln+STF7bdOxvnOERYyjo8ZbKumzigNePSwbQYVWuso76GI843yjaX2rhn0+ Jt0NVJM41jVctf9qwacVduR7CEi0qJgpAUJyZUuB9IpFWF1Kz14O3Leh6URuRZ43 RzHaRmNtzkxttGBuOtAg+ilOuwiGAo9VQLgdONlqQFcrbp97/fO8ZIqiPrbhLxCZ fXkYi3mktZVRFKXG62FHAuH1sLDXCKba3avDcUR7ykG4ZXcmp6kl14UKa8JHOHPE NYyr0R6oHELOGZMox1nQcFwuYMX9sJdAUU/9SQVXyA6u6YtxlpZiC8qhXM1IE00T Q9+q5ppffSUDMC4V/5If5A6snKVP78M8qd/RMVswcjMUMEnov+wykwCbDLD+IReM A57XX+HojN+8XFTL9Jwge3z3ZlMwL7E54W3cI7f6cxO5DVwoKxkdk2jRIg37oqSl SU3z/bA9UXjHcTl/6BoLho2p9rWm6oljANPeQuLHyGJ3hc19N8nDo2IATp70klGP kd1qhIgrdkki7gBpanMOK98hKMpdQgs+NY4DkaMJqfrHzWR/CYkdyUCivFaepaFS K78+jVu1oCMOFOnucPXL2fQa3VQn+69+7mA324frjwZj9NzrHjd0a5UP7waPpd9W 2jZoj4b+g+l+XU1SQ+9DWiuZtvfDW++k0BMCAwEAAaOCAZEwggGNMBIGA1UdEwEB /wQIMAYBAf8CAQAweAYDVR0gBHEwbzAIBgZngQwBAgEwCAYGZ4EMAQICMA4GDCsG AQQBvlgAAmQBAjBJBgwrBgEEAb5YAAOHBAAwOTA3BggrBgEFBQcCARYraHR0cDov L3d3dy5oeWRyYW50aWQuY29tL3N1cHBvcnQvcmVwb3NpdG9yeTByBggrBgEFBQcB AQRmMGQwKgYIKwYBBQUHMAGGHmh0dHA6Ly9vY3NwLnF1b3ZhZGlzZ2xvYmFsLmNv bTA2BggrBgEFBQcwAoYqaHR0cDovL3RydXN0LnF1b3ZhZGlzZ2xvYmFsLmNvbS9x dnJjYTIuY3J0MA4GA1UdDwEB/wQEAwIBBjAfBgNVHSMEGDAWgBQahGK8SEwzJQTU 7tD2A8QZRtGUazA5BgNVHR8EMjAwMC6gLKAqhihodHRwOi8vY3JsLnF1b3ZhZGlz Z2xvYmFsLmNvbS9xdnJjYTIuY3JsMB0GA1UdDgQWBBSYarYtLr+nqp/299YJr9WL V/mKtzANBgkqhkiG9w0BAQsFAAOCAgEAlraik8EDDUkpAnIOajO9/r4dpj/Zry76 6SH1oYPo7eTGzpDanPMeGMuSmwdjUkFUPALuWwkaDERfz9xdyFL3N8CRg9mQhdtT 3aWQUv/iyXULXT87EgL3b8zzf8fhTS7r654m9WM2W7pFqfimx9qAlFe9XcVlZrUu 9hph+/MfWMrUju+VPL5U7hZvUpg66mS3BaN15rsXv2+Vw6kQsQC/82iJLHvtYVL/ LwbNio18CsinDeyRE0J9wlYDqzcg5rhD0rtX4JEmBzq8yBRvHIB/023o/vIO5oxh 83Hic/2Xgwksf1DKS3/z5nTzhsUIpCpwkN6nHp6gmA8JBXoUlKQz4eYHJCq/ZyC+ BuY2vHpNx6101J5dmy7ps7J7d6mZXzguP3DQN84hjtfwJPqdf+/9RgLriXeFTqwe snxbk2FsPhwxhiNOH98GSZVvG02v10uHLVaf9B+puYpoUiEqgm1WG5mWW1PxHstu Ew9jBMcJ6wjQc8He9rSUmrhBr0HyhckdC99RgEvpcZpV2XL4nPPrTI2ki/c9xQb9 kmhVGonSXy5aP+hDC+Ht+bxmc4wN5x+vB02hak8Hh8jIUStRxOsRfJozU0R9ysyP EZAHFZ3Zivg2BaD4tOISO8/T2FDjG7PNUv0tgPAOKw2t94B+1evrSUhqJDU0Wf9c 9vkaKoPvX4w= -----END CERTIFICATE-----
When you use ASA, which is affected version, you can also use the following command for latest trustpool import immediately. This command is also available on ASA version 9.5(1) or earlier, which does not support auto-import feature.
crypto ca trustpool import url http://www.cisco.com/security/pki/trs/ios_core.p7b
The below is example output of latest trustpool import before new smart license registration.
ciscoasa(config)# show version | in Version Cisco Adaptive Security Appliance Software Version 9.4(4)24 Device Manager Version 7.5(2)61 ciscoasa(config)# ciscoasa(config)# crypto ca trustpool import url http://www.cisco.com/security/pki/trs/ios_core.p7b Root file signature verified. Trustpool import: attempted: 10 installed: 10 duplicates: 0 expired: 0 failed: 0 ciscoasa(config)# ciscoasa(config)# license smart register idtoken OThiYTZjNjYtMDZjMy00MDQ...<snip> ciscoasa(config)# ciscoasa(config)# INFO: ASAv platform license state is Licensed. ciscoasa(config)#
@Taisuke Nakamura Great point and easy workaround. I have added that to the document! Thanks!
Hi all,
how about AMP Private Cloud?
Cheers
Can this type of fix be implemented on ASR9k devices aswell?
And how would one do that?
Thanks in advance.
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: