C5921 Smart licensing - Fail to send out Call Home HTTP message

[rudimocnik]

Hi

I have c5921 running 15.5(3)M code and I have issues with smart licensing. I created the token in the CSSM and issued the command: 

license smart register idtoken {tokenSTRING}

I get the following logs:

%PKI-4-NOCONFIGAUTOSAVE: Configuration was modified.  Issue "write memory" to save new IOS PKI configuration
%SMART_LIC-3-COMM_FAILED: Communications failure with Cisco licensing cloud: Fail to send out Call Home HTTP message.

The thing is I have vrfs configured on the interfaces and c5921 only has Internet access through vrf PUBLIC.

c5921_312_127128#ping vrf PUBLIC software.cisco.com
Translating "software.cisco.com"...domain server (193.2.1.66) [OK]

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 104.108.74.32, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 40/42/45 ms
c5921_312_127128#

So the connectivity and dns resolution works. In addition I tried if firewall is in the way and the connection seems to be open.

c5921_312_127128#telnet software.cisco.com 80 /vrf PUBLIC
Translating "software.cisco.com"...domain server (255.255.255.255)

Translating "software.cisco.com"...domain server (193.2.1.66) [OK]
Trying e2757.dscb.akamaiedge.net (104.108.74.32, 80)... Open

and https ... 

c5921_312_127128#telnet software.cisco.com 443 /vrf PUBLIC
Translating "software.cisco.com"...domain server (255.255.255.255)

Translating "software.cisco.com"...domain server (193.2.1.66) [OK]
Trying e2757.dscb.akamaiedge.net (104.108.74.32, 443)... Open

I follow a guide from a bug report and configured call-home agent to use HTTP. Still no luck. Here is the call-home config:

call-home
 ! If contact email address in call-home is configured as sch-smart-licensing@cisco.com
 ! the email address configured in Cisco Smart License Portal will be used as contact email address to send SCH notifications.
 contact-email-addr sch-smart-licensing@cisco.com
 vrf PUBLIC
 profile "CiscoTAC-1"
  active
  destination transport-method http
  no destination transport-method email
  destination address http http://tools.cisco.com/its/service/oddce/services/DDCEService 

It seems that c5921 cannot send messages using vrf. Any ideas?

[Mark Malone]

Hi
before enabling registration did you enable smart license enable , sure you did just in case

looks like there may be known bugs too regarding CSSM and VRFs

Bug Search
CSCvm59508
Help | Feedback Feedback
VRF hostname resolution error is causing smart license register failure via HTTPS
CSCvm59508
Description
Symptom:
Smart license registration was not successful after "license smart register idtoken" was issued.
------------------------------------------------------------
smartlicserver[219]: %LICENSE-SMART_LIC-3-AGENT_REG_FAILED : Smart Agent for Licensing Registration with Cisco licensing cloud failed: Fail to send out Call Home HTTP message
smartlicserver[219]: %LICENSE-SMART_LIC-3-COMM_FAILED : Communications failure with Cisco licensing cloud: Fail to send out Call Home HTTP message
------------------------------------------------------------

Conditions:
This issue can be seen when using a smart license via HTTPS in a configured VRF.
This issue is not seen when HTTPS is used in a default VRF or HTTP is used.

------------------------------------------------------------
http client vrf 

call-home
vrf 
service active
contact smart-licensing
profile CiscoTAC-1
active
destination transport-method http
!
!

crypto ca trustpoint Trustpool
vrf 
!
------------------------------------------------------------

Workaround:
Consider to apply one of the below workarounds:

1) Disable Certificate Revocation List checking
------------------------------------------------------------
crypto ca trustpoint Trustpool
crl optional
!
------------------------------------------------------------

2) Using HTTP
Configure call-home so that HTTP is used instead of HTTPS.
------------------------------------------------------------
call-home
profile CiscoTAC-1
destination address http http://tools.cisco.com/its/service/oddce/services/DDCEService
!
!
------------------------------------------------------------
(*) HTTP is used by default when no k9sec package is installed.

[rudimocnik]

Thanks for your quick reply but I already found the solution posted below.

Yeah I looked into those bugs as well. Surprisingly the only thing I had to do was configure http source interface and it worked even on

default call-home config

. Note that I can only reach software.cisco.com via Ethernet0/0.10 that has vrf forwarding PUBLIC configured. 

[Mark Malone]

Thanks for posting the fix

[rudimocnik · Accepted solution]

Huh I finally cracked it. This worked even on

default call-home config

without vrf configured (using https).

The command I was missing is: 

enable
conf t
ip http client source-interface Ethernet 0/0.10

call-home config:

call-home
 ! If contact email address in call-home is configured as sch-smart-licensing@cisco.com
 ! the email address configured in Cisco Smart License Portal will be used as contact email address to send SCH notifications.
 contact-email-addr sch-smart-licensing@cisco.com
 profile "CiscoTAC-1"
  active
  destination transport-method http
  no destination transport-method email

[Gabriel Torrecilla]

Hi Guys!! 
I have  the same problem, but with a NCS5504 with IOS XR. (i've created the token)
I reach internet only from VRF INTERNET and a loopback 1.
Following the example of this topic, I've tried playing with the source interface and VRF in call-home and http-client, but the result is the same.

I think that the config should be someting like this

call-home
 vrf INTERNET
 service active
 contact smart-licensing
 source-interface Loopback1 
 profile CiscoTAC-1
  active
  destination transport-method http
 !  
!

http client vrf INTERNET
http client source-interface ipv4 Loopback1
!
!

So, internet access it working ok...

RP/0/RP0/CPU0:Core1#ping tools.cisco.com source loopback 1 vrf INTERNET
Tue Aug 27 15:28:51.254 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 173.37.145.8, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 200/200/204 ms
RP/0/RP0/CPU0:Core1#
!

!

!

RP/0/RP0/CPU0:Core1#telnet vrf INTERNET tools.cisco.com 443 source-interface loopback 1
Trying tools.cisco.com(2001:420:1201:5::a)...
Use specified source interface(Loopback1).
Global address not present, using link local addressas source address
Not able to get link local addressCan't use Loopback1 as source interface for IPv6.
Trying tools.cisco.com(173.37.145.8)...
Use specified source interface(Loopback1).
Use 186.189.64.1 as local address.
Connected to tools.cisco.com.
Escape sequence is '^^q'.

The error persits with diffent configs (with or without VRF / source interface)

RP/0/RP0/CPU0:Core1#RP/0/RP0/CPU0:Aug 27 15:26:36.378 UTC: call_home[346]: SMART-LICENSE-ERROR: smart_license_req_http_send[164], Failed to send request to all URLs.
RP/0/RP0/CPU0:Aug 27 15:26:36.378 UTC: call_home[346]: SMART-LICENSE-TRACE: call_home_smart_license_stats_update[703], Update smart license stats entry, subtype REGISTRATION,stats_type 2

Also i've checked all this items (which seems to be useful)

Next Steps:

- Verify that the Cisco device can ping tools.cisco.com or the nslookup translated IP

- Attempt to telnet from the Cisco device to tools.cisco.com on TCP port 443 (port used by HTTPS)

- Verify that the HTTPs client source interface is correct

- Verify that the URL/IP in the call home profile is set correctly on the Cisco device via

show call-home profile all

- Verify the ip route is pointing to the correct next hop

- Ensure TCP port 443 is not being blocked on the Cisco device, the path to Smart Call Home Server, or the Cisco Smart Software Manager satellite

- Ensure that the correct Virtual Routing and Forwarding (VRF) instance is configured if applicable

from this link
https://www.cisco.com/c/en/us/support/docs/switches/catalyst-9500-series-switches/214484-cisco-smart-licensing-troubleshooting.html#anc17
All test were ok... but still it doesn't work.

maybe there's a similar bug for IOS XR...

[TrivialPants]

Hi @Gabriel Torrecilla I was curious if you ever found the solution to this with your NCS router? We have followed all the same steps for similar devices, but are having trouble getting an NCS 540 to register too.

[maninder.negi@intl.verizon.com]

Hi,

Can you please post the DNS configuration you did?

Cisco has asked to configure below, but i would like to know for point number 7 from where I will get the IP address details?

1. enable
2. configure terminal
3. {ip | ipv6} name-server server-address 1 [server-address 2] [server-address 3] [server-address 4] [server-address 5] [server-address 6]
4. ip name-server vrf Mgmt-vrf server-address 1 [server-address 2] [server-address 3] [server-address 4] [server-address 5] [server-address 6]
5. ip domain lookup source-interface interface-type interface-number
6. ip domain name example.com

7. ip host tools.cisco.com ip-address

8. interface vlan_id
9. ntp server ip-address [version number] [key key-id] [prefer]
10. switchport access vlan vlan_id
11. ip route ip-address ip-mask subnet mask
12. license smart transport callhome
13. ip http client source-interface interface-type interface-number
14. exit
15. copy running-config startup-config

[dpowell4]

This totally worked! Thanks!

[JESSICA Walsh]

Thank you.

[jai_chandra2001]

Thanks guys. this is my working configuration for ASR1001 with mgmt-vrf .

###############################################################

service call-home
call-home
! If contact email address in call-home is configured as sch-smart-licensing@cisco.com
! the email address configured in Cisco Smart License Portal will be used as contact email address to send SCH notifications.
contact-email-addr sch-smart-licensing@cisco.com
vrf Mgmt-intf
no http secure server-identity-check
profile "CiscoTAC-1"
active
destination transport-method http
no destination transport-method email
license smart transport callhome




ip domain lookup vrf Mgmt-intf source-interface GigabitEthernet0




interface GigabitEthernet0
vrf forwarding Mgmt-intf
ip address <IP> 255.255.255.0

ip route vrf Mgmt-intf 0.0.0.0 0.0.0.0 <GW>




ip http client source-interface GigabitEthernet0




once done, use "license smart register idtoken <token>

###############################################################