cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements

Welcome to the Smart Call Home Community!

Our online forum for Smart Call Home customers to share, learn, and collaborate on Smart Call Home related topics. We encourage you to ask questions of Cisco experts, start a discussion, or share ideas and insight.

Smart Call Home enabled devices perform proactive diagnostics on their own components to provide real-time alerts and remediation advice when an issue is detected. An embedded support feature available on a broad range of Cisco products, it is provided at no additional cost with an active Smart Net Total Care Service, SP Base, Unified Computing Support Service, or Mission Critical Support Service contract for the designated products.

This Community will provide you with an overview about Cisco Smart Call Home features and how these features are embedded in a wide range of Cisco products to help your network. Smart Call Home provides higher network availability and support service quality.

628
Views
5
Helpful
6
Replies
Highlighted
Beginner

Call-Home SSL errors

Hi Everybody!

 

I'm new to the field and new to my current place of work and that place of work is new to everything related to smart licensing, smart call-home, etc. Nobody here has touched IOS-XR until a month ago also. I am almost CCNA RS level knowledge wise. I have been tasked with getting the new NCS5501's we are installing to pull their smart licensing information. I've read the documentation on the NCS5501 relating to this process, but I'm clearly missing some dependencies...

 

Currently I have a test router configured with only the purpose of succeeding the SCH and pulling its smart license. I have a single port configured, the MgmtEth0/RP0/CPU0/0. It gets a DHCP address from our network and NATs out to the greater internet. The router is configured with a single ipv4 af static route to go to our default gateway. 

 

I've configured the following:

domain name-server 4.2.2.2
!
http client vrf default
http client source-interface ipv4 MgmtEth0/RP0/CPU0/0
fpd auto-upgrade enable
ntp
server 74.6.168.73
server 107.155.79.108
server 129.128.12.20
server 174.94.155.224
!
call-home
service active
profile CiscoTAC-1
active
anonymous-reporting-only
reporting smart-call-home-data
!
!
interface MgmtEth0/RP0/CPU0/0
ipv4 address dhcp

 

I've confirmed the following:

NTP syncs successfully.

I can reach tools.cisco.com with a ping as well as a port 80 and port 443 telnet.

 

Yet whenever SCH attempts to reach cisco it reports back with an SSL error. There must be a dependency I'm missing somewhere. Is there something I need to do CA wise? Do I need to define one? When you do commands like checking for CRLs or certificates it appears there are none in the router.

 

Specifically I get this:

RP/0/RP0/CPU0:Nov 12 18:47:32.667 UTC: http_client[166]: %SECURITY-XR_SSL-3-CERT_VERIFY_ERR_2_PARAM : SSL certificate verify error: Error in fetching the issuer certificate/CRL 'Crypto Engine' detected the 'warning' condition 'Key data missing'

 

A "call-home trace all" gives me the repeating information. Seems to me to be saying, it reaches out, gets something back, writes it to a temp file, realizes it can't decrypt it, then throws it away.

4
Nov 12 18:47:58.513 call_home/trace 0/RP0/CPU0 t5414 HTTP req data free
Nov 12 18:47:58.513 call_home/trace 0/RP0/CPU0 t5414 httpc_file_resp_read: read http reponse "/tmp/call_home_http_1626434436.resp"
Nov 12 18:47:58.513 call_home/error 0/RP0/CPU0 t5414 httpc_file_resp_read: "/tmp/call_home_http_1626434436.resp" size is zero
Nov 12 18:47:58.513 call_home/trace 0/RP0/CPU0 t5414 httpc_file_del: remove file "/tmp/call_home_http_1626434436.resp"
Nov 12 18:47:58.513 call_home/trace 0/RP0/CPU0 t5414 httpc_file_del: remove file "/tmp/call_home_http_1626434436.post"
Nov 12 18:47:58.513 call_home/trace 0/RP0/CPU0 t5414 call_home_http_resp_data() status = 24
Nov 12 18:47:58.513 call_home/trace 0/RP0/CPU0 t5414 call_home_set_httpc_resp_status: is entered. tid (15), status (324), err_string ()
Nov 12 18:47:58.513 call_home/error 0/RP0/CPU0 t5414 call_home_http_resp_data(), httpc response error, Error during SSL communication
Nov 12 18:47:58.513 call_home/trace 0/RP0/CPU0 t14985 call_home_remove_httpc_resp_node() is entered
Nov 12 18:48:08.513 call_home/trace 0/RP0/CPU0 t14985 send_one_http_msg() is entered, url = https://tools.cisco.com/its/service/oddce/services/DDCEService
Nov 12 18:48:08.513 call_home/trace 0/RP0/CPU0 t14985 call_home_httpc_post() url https://tools.cisco.com/its/service/oddce/services/DDCEService, sid 1, len 10466
Nov 12 18:48:10.615 call_home/trace 0/RP0/CPU0 t14985 call_home_httpc_post: HTTP send successfully
Nov 12 18:48:10.615 call_home/trace 0/RP0/CPU0 t14985 call_home_wait_for_httpc_resp() is entered
Nov 12 18:48:10.615 call_home/trace 0/RP0/CPU0 t14985 call_home_add_httpc_resp_node(): is entered
Nov 12 18:48:11.479 call_home/trace 0/RP0/CPU0 t5414 call_home_http_resp_data(): Received response for (nil) tid 16 sid 1 status 24

 

When I google this error I get pretty much no information, I must be missing something so basic it's not even worth mentioning. I can't seem to figure it out though... I don't have any Security background which probably isn't helping me solve this problem. Only RS.

Any help would be greatly appreciated. Thanks in advance!

6 REPLIES 6
Highlighted
Cisco Employee

Re: Call-Home SSL errors

Hi Chris,
Can you check with your security team to see if you are using SSL inspection/interception within the network and report back?
Thanks!
Highlighted
Beginner

Re: Call-Home SSL errors

Thanks Joshua,

 

I'll check with them tomorrow but I don't think this is the case. We are able to load https webpages on our workstations no problem. 

Highlighted
Beginner

Re: Call-Home SSL errors

I have some more information Joshua.

 

I placed the router on the edge so it has a public IP and direct internet access instead of having to NAT through our Firewall, though that shouldn't have been the problem. The reachability tests of an ipv4 ping and telnet to tools.cisco.com come back good. NTP resynchronized. I'm still getting the same error in the console. "key data is missing."

 

I do have some debug information if that's helpful for diagnosing the problem. The debugs I've activated are "debug crypto engine all" and "debug ssl error" "ssl event" "ssl handshake" and "ssl traffic". So at least I know it wasn't SSL inspection related on it's way out to the internet.

 

Here is the output:

'warning' condition 'Key data missing'
RP/0/RP0/CPU0:Nov 13 19:25:55.895 UTC: http_client[166]: Inside cert_verify_chain_cb
RP/0/RP0/CPU0:Nov 13 19:25:55.895 UTC: http_client[166]: number of cert in chain is 3
RP/0/RP0/CPU0:Nov 13 19:25:55.895 UTC: http_client[166]: Inside find_issuer_cert
RP/0/RP0/CPU0:Nov 13 19:25:55.895 UTC: http_client[166]: Returning the issuer cert
RP/0/RP0/CPU0:Nov 13 19:25:55.895 UTC: http_client[166]: Inside find_issuer_cert
RP/0/RP0/CPU0:Nov 13 19:25:55.895 UTC: http_client[166]: Returning the issuer cert
RP/0/RP0/CPU0:Nov 13 19:25:55.895 UTC: http_client[166]: Inside find_issuer_cert
RP/0/RP0/CPU0:Nov 13 19:25:55.895 UTC: http_client[166]: Returning the NULL
RP/0/RP0/CPU0:Nov 13 19:25:55.895 UTC: http_client[166]: Inside SSL_convert_x509_name_in_str
RP/0/RP0/CPU0:Nov 13 19:25:55.895 UTC: http_client[166]: Success able to fetch the issuer/subject name
RP/0/RP0/CPU0:Nov 13 19:25:55.895 UTC: http_client[166]: **** Issuer name is CN=QuoVadis Root CA 2,O=QuoVadis Limited,C=BM
RP/0/RP0/CPU0:Nov 13 19:25:56.002 UTC: cepki[375]: cepki_svr_process_msg: msg type: 20
RP/0/RP0/CPU0:Nov 13 19:25:56.002 UTC: cepki[375]: cepki_process_trustpool_vrf sending vrfname default and vrf is is 60000000
RP/0/RP0/CPU0:Nov 13 19:25:56.009 UTC: http_client[166]: Cert number = 0
RP/0/RP0/CPU0:Nov 13 19:25:56.009 UTC: http_client[166]: Inside SSL_convert_x509_name_in_str
RP/0/RP0/CPU0:Nov 13 19:25:56.009 UTC: http_client[166]: Success able to fetch the issuer/subject name
RP/0/RP0/CPU0:Nov 13 19:25:56.009 UTC: http_client[166]: **** subject name is CN=tools.cisco.com,O=Cisco Systems\, Inc.,L=San Jose,ST=CA,C=US
RP/0/RP0/CPU0:Nov 13 19:25:56.009 UTC: http_client[166]: Inside SSL_convert_x509_name_in_str
RP/0/RP0/CPU0:Nov 13 19:25:56.009 UTC: http_client[166]: Success able to fetch the issuer/subject name
RP/0/RP0/CPU0:Nov 13 19:25:56.010 UTC: http_client[166]: **** Issuer name is CN=HydrantID SSL ICA G2,O=HydrantID (Avalanche Cloud Corporation),C=US
RP/0/RP0/CPU0:Nov 13 19:25:56.010 UTC: http_client[166]: Inside find_issuer_cert
RP/0/RP0/CPU0:Nov 13 19:25:56.010 UTC: http_client[166]: Returning the issuer cert
RP/0/RP0/CPU0:Nov 13 19:25:56.010 UTC: http_client[166]: Got the cert item at position 0
RP/0/RP0/CPU0:Nov 13 19:25:56.122 UTC: http_client[166]: c3m_gen_md5_hash ...
RP/0/RP0/CPU0:Nov 13 19:25:56.447 UTC: http_client[166]: The issuer_cert and the crl_obj count is 1
RP/0/RP0/CPU0:Nov 13 19:25:56.447 UTC: http_client[166]: Error in fetching the issuer cert and CRL.
RP/0/RP0/CPU0:Nov 13 19:25:56.447 UTC: http_client[166]: %SECURITY-XR_SSL-3-CERT_VERIFY_ERR_2_PARAM : SSL certificate verify error: Error in fetching the issuer certificate/CRL 'Crypto Engine' detected the 'warning' condition 'Key data missing'

 

Does this help much? Anything else I could check?

 

Thanks for the help! I'm sure I'm missing something silly here...

Highlighted
Beginner

Re: Call-Home SSL errors

Just in case anybody was interested. It turns out this is related to a bug in XR 7.0.1 where the OS isn't sure what to do with the HTTPS CRL data and just throws it away. It's supposed to be fixed in 7.1.0 in February according to the TAC guy that was helping me.

Highlighted
Beginner

Re: Call-Home SSL errors

Thanks for posting this. We've hit the same bug. How is smart-licensing made mandatory in this version and it's not even able to connect to the server?

 

Anyway, one workaround is to disable CRL checking (which I do not recommend!) by configuring

 

crypto ca trustpoint Trustpool
 crl optional  ! not recommended

I assume that this bug also affects using a local smart satelite?

Highlighted
Beginner

Re: Call-Home SSL errors

That's a good question and I'm not sure. But I would assume that it contacts a smart satellite in the exact same way it would contact tools.cisco.com. Therefore having the same problem.

I don't have the knowledge to know for sure.
CreatePlease to create content
Content for Community-Ad
Cisco Community April 2020 Spotlight Award Winners