Re: Call-Home SSL errors

ChrisGflys3D · ‎11-12-2019

Hi Everybody!

I'm new to the field and new to my current place of work and that place of work is new to everything related to smart licensing, smart call-home, etc. Nobody here has touched IOS-XR until a month ago also. I am almost CCNA RS level knowledge wise. I have been tasked with getting the new NCS5501's we are installing to pull their smart licensing information. I've read the documentation on the NCS5501 relating to this process, but I'm clearly missing some dependencies...

Currently I have a test router configured with only the purpose of succeeding the SCH and pulling its smart license. I have a single port configured, the MgmtEth0/RP0/CPU0/0. It gets a DHCP address from our network and NATs out to the greater internet. The router is configured with a single ipv4 af static route to go to our default gateway.

I've configured the following:

domain name-server 4.2.2.2
!
http client vrf default
http client source-interface ipv4 MgmtEth0/RP0/CPU0/0
fpd auto-upgrade enable
ntp
server 74.6.168.73
server 107.155.79.108
server 129.128.12.20
server 174.94.155.224
!
call-home
service active
profile CiscoTAC-1
active
anonymous-reporting-only
reporting smart-call-home-data
!
!
interface MgmtEth0/RP0/CPU0/0
ipv4 address dhcp

I've confirmed the following:

NTP syncs successfully.

I can reach tools.cisco.com with a ping as well as a port 80 and port 443 telnet.

Yet whenever SCH attempts to reach cisco it reports back with an SSL error. There must be a dependency I'm missing somewhere. Is there something I need to do CA wise? Do I need to define one? When you do commands like checking for CRLs or certificates it appears there are none in the router.

Specifically I get this:

RP/0/RP0/CPU0:Nov 12 18:47:32.667 UTC: http_client[166]: %SECURITY-XR_SSL-3-CERT_VERIFY_ERR_2_PARAM : SSL certificate verify error: Error in fetching the issuer certificate/CRL 'Crypto Engine' detected the 'warning' condition 'Key data missing'

A "call-home trace all" gives me the repeating information. Seems to me to be saying, it reaches out, gets something back, writes it to a temp file, realizes it can't decrypt it, then throws it away.

4
Nov 12 18:47:58.513 call_home/trace 0/RP0/CPU0 t5414 HTTP req data free
Nov 12 18:47:58.513 call_home/trace 0/RP0/CPU0 t5414 httpc_file_resp_read: read http reponse "/tmp/call_home_http_1626434436.resp"
Nov 12 18:47:58.513 call_home/error 0/RP0/CPU0 t5414 httpc_file_resp_read: "/tmp/call_home_http_1626434436.resp" size is zero
Nov 12 18:47:58.513 call_home/trace 0/RP0/CPU0 t5414 httpc_file_del: remove file "/tmp/call_home_http_1626434436.resp"
Nov 12 18:47:58.513 call_home/trace 0/RP0/CPU0 t5414 httpc_file_del: remove file "/tmp/call_home_http_1626434436.post"
Nov 12 18:47:58.513 call_home/trace 0/RP0/CPU0 t5414 call_home_http_resp_data() status = 24
Nov 12 18:47:58.513 call_home/trace 0/RP0/CPU0 t5414 call_home_set_httpc_resp_status: is entered. tid (15), status (324), err_string ()
Nov 12 18:47:58.513 call_home/error 0/RP0/CPU0 t5414 call_home_http_resp_data(), httpc response error, Error during SSL communication
Nov 12 18:47:58.513 call_home/trace 0/RP0/CPU0 t14985 call_home_remove_httpc_resp_node() is entered
Nov 12 18:48:08.513 call_home/trace 0/RP0/CPU0 t14985 send_one_http_msg() is entered, url = https://tools.cisco.com/its/service/oddce/services/DDCEService
Nov 12 18:48:08.513 call_home/trace 0/RP0/CPU0 t14985 call_home_httpc_post() url https://tools.cisco.com/its/service/oddce/services/DDCEService, sid 1, len 10466
Nov 12 18:48:10.615 call_home/trace 0/RP0/CPU0 t14985 call_home_httpc_post: HTTP send successfully
Nov 12 18:48:10.615 call_home/trace 0/RP0/CPU0 t14985 call_home_wait_for_httpc_resp() is entered
Nov 12 18:48:10.615 call_home/trace 0/RP0/CPU0 t14985 call_home_add_httpc_resp_node(): is entered
Nov 12 18:48:11.479 call_home/trace 0/RP0/CPU0 t5414 call_home_http_resp_data(): Received response for (nil) tid 16 sid 1 status 24

When I google this error I get pretty much no information, I must be missing something so basic it's not even worth mentioning. I can't seem to figure it out though... I don't have any Security background which probably isn't helping me solve this problem. Only RS.

Any help would be greatly appreciated. Thanks in advance!

Josh Harpst · ‎11-12-2019

Hi Chris,
Can you check with your security team to see if you are using SSL inspection/interception within the network and report back?
Thanks!

ChrisGflys3D · ‎11-12-2019

Thanks Joshua,

I'll check with them tomorrow but I don't think this is the case. We are able to load https webpages on our workstations no problem.

ChrisGflys3D · ‎11-13-2019

I have some more information Joshua.

I placed the router on the edge so it has a public IP and direct internet access instead of having to NAT through our Firewall, though that shouldn't have been the problem. The reachability tests of an ipv4 ping and telnet to tools.cisco.com come back good. NTP resynchronized. I'm still getting the same error in the console. "key data is missing."

I do have some debug information if that's helpful for diagnosing the problem. The debugs I've activated are "debug crypto engine all" and "debug ssl error" "ssl event" "ssl handshake" and "ssl traffic". So at least I know it wasn't SSL inspection related on it's way out to the internet.

Here is the output:

'warning' condition 'Key data missing'
RP/0/RP0/CPU0:Nov 13 19:25:55.895 UTC: http_client[166]: Inside cert_verify_chain_cb
RP/0/RP0/CPU0:Nov 13 19:25:55.895 UTC: http_client[166]: number of cert in chain is 3
RP/0/RP0/CPU0:Nov 13 19:25:55.895 UTC: http_client[166]: Inside find_issuer_cert
RP/0/RP0/CPU0:Nov 13 19:25:55.895 UTC: http_client[166]: Returning the issuer cert
RP/0/RP0/CPU0:Nov 13 19:25:55.895 UTC: http_client[166]: Inside find_issuer_cert
RP/0/RP0/CPU0:Nov 13 19:25:55.895 UTC: http_client[166]: Returning the issuer cert
RP/0/RP0/CPU0:Nov 13 19:25:55.895 UTC: http_client[166]: Inside find_issuer_cert
RP/0/RP0/CPU0:Nov 13 19:25:55.895 UTC: http_client[166]: Returning the NULL
RP/0/RP0/CPU0:Nov 13 19:25:55.895 UTC: http_client[166]: Inside SSL_convert_x509_name_in_str
RP/0/RP0/CPU0:Nov 13 19:25:55.895 UTC: http_client[166]: Success able to fetch the issuer/subject name
RP/0/RP0/CPU0:Nov 13 19:25:55.895 UTC: http_client[166]: **** Issuer name is CN=QuoVadis Root CA 2,O=QuoVadis Limited,C=BM
RP/0/RP0/CPU0:Nov 13 19:25:56.002 UTC: cepki[375]: cepki_svr_process_msg: msg type: 20
RP/0/RP0/CPU0:Nov 13 19:25:56.002 UTC: cepki[375]: cepki_process_trustpool_vrf sending vrfname default and vrf is is 60000000
RP/0/RP0/CPU0:Nov 13 19:25:56.009 UTC: http_client[166]: Cert number = 0
RP/0/RP0/CPU0:Nov 13 19:25:56.009 UTC: http_client[166]: Inside SSL_convert_x509_name_in_str
RP/0/RP0/CPU0:Nov 13 19:25:56.009 UTC: http_client[166]: Success able to fetch the issuer/subject name
RP/0/RP0/CPU0:Nov 13 19:25:56.009 UTC: http_client[166]: **** subject name is CN=tools.cisco.com,O=Cisco Systems\, Inc.,L=San Jose,ST=CA,C=US
RP/0/RP0/CPU0:Nov 13 19:25:56.009 UTC: http_client[166]: Inside SSL_convert_x509_name_in_str
RP/0/RP0/CPU0:Nov 13 19:25:56.009 UTC: http_client[166]: Success able to fetch the issuer/subject name
RP/0/RP0/CPU0:Nov 13 19:25:56.010 UTC: http_client[166]: **** Issuer name is CN=HydrantID SSL ICA G2,O=HydrantID (Avalanche Cloud Corporation),C=US
RP/0/RP0/CPU0:Nov 13 19:25:56.010 UTC: http_client[166]: Inside find_issuer_cert
RP/0/RP0/CPU0:Nov 13 19:25:56.010 UTC: http_client[166]: Returning the issuer cert
RP/0/RP0/CPU0:Nov 13 19:25:56.010 UTC: http_client[166]: Got the cert item at position 0
RP/0/RP0/CPU0:Nov 13 19:25:56.122 UTC: http_client[166]: c3m_gen_md5_hash ...
RP/0/RP0/CPU0:Nov 13 19:25:56.447 UTC: http_client[166]: The issuer_cert and the crl_obj count is 1
RP/0/RP0/CPU0:Nov 13 19:25:56.447 UTC: http_client[166]: Error in fetching the issuer cert and CRL.
RP/0/RP0/CPU0:Nov 13 19:25:56.447 UTC: http_client[166]: %SECURITY-XR_SSL-3-CERT_VERIFY_ERR_2_PARAM : SSL certificate verify error: Error in fetching the issuer certificate/CRL 'Crypto Engine' detected the 'warning' condition 'Key data missing'

Does this help much? Anything else I could check?

Thanks for the help! I'm sure I'm missing something silly here...

ChrisGflys3D · ‎12-24-2019

Just in case anybody was interested. It turns out this is related to a bug in XR 7.0.1 where the OS isn't sure what to do with the HTTPS CRL data and just throws it away. It's supposed to be fixed in 7.1.0 in February according to the TAC guy that was helping me.

neuner · ‎01-14-2020

Thanks for posting this. We've hit the same bug. How is smart-licensing made mandatory in this version and it's not even able to connect to the server?

Anyway, one workaround is to disable CRL checking (which I do not recommend!) by configuring

crypto ca trustpoint Trustpool
 crl optional  ! not recommended

I assume that this bug also affects using a local smart satelite?

ChrisGflys3D · ‎01-15-2020

That's a good question and I'm not sure. But I would assume that it contacts a smart satellite in the exact same way it would contact tools.cisco.com. Therefore having the same problem.

I don't have the knowledge to know for sure.

BrianDean · ‎12-19-2022

This worked for me....I've been having this same probably and TAC has been struggling with this for a while. Problem still seems to exist in 7.5.2.

neuner · ‎12-19-2022

This is still a problem in 7.7.2, so more than three years and as far as I know in all versions of XR that have "smart" licensing. I don't understand how a security relevant bug like this is simply ignored for years and years, across all versions and product lines.

I have had TAC cases for other "smart" licensing issues where the first step for debugging was to configure `crl optional`.

Also, I can't find much about this in the docs. Is the assumption that everyone using "smart" licensing will have a divine inspiration that `crl optional` is needed, or does every customer raise a TAC case and debug this?

rehernandez · ‎05-04-2023

Hello Guys, just landing on this thread due to the same issue. After searching a lot, I found this bug related and solved the issue.

https://bst.cisco.com/bugsearch/bug/CSCvx00476

FN related

https://www.cisco.com/c/en/us/support/docs/field-notices/722/fn72290.html

For solved the issue, run the follow command on global mode for update the certificates.

RP/0/RP0/CPU0:ios#crypto ca trustpool import url http://www.cisco.com/security/pki/trs/ios.p7b

Other options are provided in the bug documentation.

Regards

johnlloyd_13 · ‎08-13-2024

hi,

i'm running NCS IOX-XR 7.4 and ran into a similar issue.

i tried the work around (HTTP and HTTPS) but download failed.

is there any other work around/fix?

i also don't see an IOS-XR fix release either.

#crypto ca trustpool import url http://www.cisco.com/security/pki/trs/ios.p7b

Tue Aug 13 05:27:33.409 UTC

RP/0/RP0/CPU0:Aug 13 05:27:38.855 UTC: cepki[290]: %SECURITY-PKI-6-ERR_1_PARAM : Download failed, HTTP returned an error

#crypto ca trustpool import url https://www.cisco.com/security/pki/trs/ios.p7b
Tue Aug 13 05:27:47.934 UTC

Cannot download certificates from CA server -
incorrect URL HTTP/TFTP/LOCAL specified
RP/0/RP0/CPU0:Aug 13 05:27:48.368 UTC: pki_cmd[69046]: %SECURITY-PKI-6-ERR_1_PARAM : Failed to import ca certificate.

#ping vrf Mgmt-intf cisco.com
Tue Aug 13 05:28:27.043 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 72.163.4.185, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 131/131/131 ms

Gardar · ‎08-13-2024

Hello,

I had this problem myself on an ASR9903 platform, stumbled upon this thread looking for a solution.

I managed to solve this, with a workaround, that is probably not within "Best practice" but I hope this helps.

http client vrf vrf-name
http client secure-verify-peer disable
http client source-interface ipv4 MgmtEth0/RP0/CPU0/0

After I put in the secure-verify-peer disable everything connected sucessfully.

Gardar · ‎08-13-2024

Hello,

I had this problem myself on an ASR9903 platform, stumbled upon this thread looking for a solution.

I managed to solve this, with a workaround, that is probably not within "Best practice" but I hope this helps.

http client vrf <vrf-name>
http client secure-verify-peer disable
http client source-interface ipv4 MgmtEth0/RP0/CPU0/0

After I put in the secure-verify-peer disable everything connected sucessfully.