Solved: Re: Smart licensing error in Cisco 1121 router

Rajesh11735 · ‎06-29-2023

Hello Guys,

I had installed security license for Cisco 1100 ISR few months ago and now I am getting the below error. I have posted my running configuration and few more outputs. Any help would be much appreciated.

Failure reason: Fail to send out Call Home HTTP message. (or)
Failure reason: SACL http unknown host


sh version (truncated o/p)

--------------------------------------------------------------------------------
Suite Suite Current Type Suite Next reboot
--------------------------------------------------------------------------------
FoundationSuiteK9 None Smart License None
securityk9
appxk9

Technology Package License Information:

-----------------------------------------------------------------
Technology Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------
appxk9 None Smart License None
uck9 None Smart License None
securityk9 securityk9 Smart License securityk9
ipbase ipbasek9 Smart License ipbasek9

The current throughput level is unthrottled
Smart Licensing Status: REGISTERED/AUTH EXPIRED



Router#show call-home profile all

Profile Name: CiscoTAC-1
Profile status: ACTIVE
Profile mode: Full Reporting
Reporting Data: Smart Call Home, Smart Licensing
Preferred Message Format: xml
Message Size Limit: 3145728 Bytes
Transport Method: http
HTTP address: https://tools.cisco.com/its/service/oddce/services/DDCEService
Other address(es): default

Periodic configuration info message is scheduled every 15 day of the month at 15:41

Periodic inventory info message is scheduled every 15 day of the month at 15:26

Alert-group Severity
------------------------ ------------
crash debugging
environment minor
inventory normal

Syslog-Pattern Severity
------------------------ ------------
.* major

Router#telnet tools.cisco.com 443 /source-interface gigabitEthernet 0/0/0
% Bad IP address or host name
Router#telnet 173.37.145.8 443 /source-interface gigabitEthernet 0/0/0
Trying 173.37.145.8, 443 ... Open
^C
[Connection to 173.37.145.8 closed by foreign host]

Router##show run all | in destination address http
destination address http https://tools.cisco.com/its/service/oddce/services/DDCEService 



Router#show lice all
Smart Licensing Status
======================

Smart Licensing is ENABLED

Registration:
Status: REGISTERED
Smart Account: XXXXXXX
Virtual Account: AccXXX
Export-Controlled Functionality: ALLOWED
Initial Registration: SUCCEEDED on Nov 04 09:30:59 2022 summer
Last Renewal Attempt: FAILED on Jun 29 15:18:45 2023 summer
Failure reason: SACL http unknown host
Next Renewal Attempt: Jun 29 15:19:14 2023 summer
Registration Expires: Nov 04 09:22:36 2023 summer

License Authorization:
Status: AUTH EXPIRED on Jun 29 15:18:54 2023 summer
Last Communication Attempt: FAILED on Jun 29 15:18:54 2023 summer
Failure reason: SACL http unknown host
Next Communication Attempt: Jun 29 16:18:44 2023 summer
Communication Deadline: DEADLINE EXCEEDED

License Conversion:
Automatic Conversion Enabled: False
Status: Not started

Export Authorization Key:
Features Authorized:
<none>

Utility:
Status: DISABLED

Data Privacy:
Sending Hostname: yes
Callhome hostname privacy: DISABLED
Smart Licensing hostname privacy: DISABLED
Version privacy: DISABLED

Transport:
Type: Smart
URL: https://smartreceiver.cisco.com/licservice/license
Proxy:
Not Supported

Miscellaneus:
Custom Id: <empty>

License Usage
==============

(ISR_1100_4P_Security):
Description:
Count: 1
Version: 1.0
Status: AUTH EXPIRED
Export status: NOT RESTRICTED

Product Information
===================
UDI: PID:C1121-4P,SN:XXXXXX

Agent Version
=============
Smart Agent for Licensing: 4.11.5_rel/41

Reservation Info
================
License reservation: DISABLED

Router#ping 171.70.168.183
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 171.70.168.183, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
Router#ping tools.cisco.com
% Unrecognized host or address, or protocol not running.

Router#ping 173.37.145.8
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 173.37.145.8, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 40/52/64 ms

Running config

ip name-server 8.8.8.8
no ip domain lookup
ip domain name custvpn
crypto pki trustpoint TP-self-signed-141864XX
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-141864XX
revocation-check none
rsakeypair TP-self-signed-1418643960
!
crypto pki trustpoint SLA-TrustPoint
enrollment terminal
revocation-check none

license udi pid C1121-4P sn XXXXXXXX
license boot level securityk9
license smart url default
license smart transport smart

ip http server
ip http secure-server
ip http secure-trustpoint SLA-TrustPoint
ip http client source-interface GigabitEthernet0/0/0

call-home
! If contact email address in call-home is configured as sch-smart-licensing@cisco.com
! the email address configured in Cisco Smart License Portal will be used as contact email address to send SCH notifications.
contact-email-addr sch-smart-licensing@cisco.com
profile "CiscoTAC-1"
active
destination transport-method http

Best Regards,
Rajesh

Flavio Miranda · ‎06-29-2023

Hi @Rajesh11735

I believe you got it when you did this two tests:

Router#telnet tools.cisco.com 443 /source-interface gigabitEthernet 0/0/0
% Bad IP address or host name
Router#telnet 173.37.145.8 443 /source-interface gigabitEthernet 0/0/0
Trying 173.37.145.8, 443 ... Open
^C
[Connection to 173.37.145.8 closed by foreign host]

Which means, you are not able to resolve tools.cisco.com. You successed when you used the IP address, which means you do have connectivity.

You have the command

ip name-server 8.8.8.8

but you also have

no ip domain lookup

and you should change it to

 ip domain lookup

View solution in original post

Flavio Miranda · ‎06-29-2023

For the licesing you dont need http or https. You can shut it down if you dont use it for device management

The ip domain lookup is often disable because it can cause and odd behavior when you type a wrong command and the router try to resolve the command and block the cli until the lookup times out.

But in your case the domain lookup is necessary

But Http and https is not. It is only good for device management via web.

View solution in original post

Flavio Miranda · ‎06-29-2023

Hi @Rajesh11735

I believe you got it when you did this two tests:

Router#telnet tools.cisco.com 443 /source-interface gigabitEthernet 0/0/0
% Bad IP address or host name
Router#telnet 173.37.145.8 443 /source-interface gigabitEthernet 0/0/0
Trying 173.37.145.8, 443 ... Open
^C
[Connection to 173.37.145.8 closed by foreign host]

Which means, you are not able to resolve tools.cisco.com. You successed when you used the IP address, which means you do have connectivity.

You have the command

ip name-server 8.8.8.8

but you also have

no ip domain lookup

and you should change it to

 ip domain lookup

Rajesh11735 · ‎06-29-2023

Thanks a lot for the quick help, Flavio. It worked!

We used no ip domain lookup as we don't need any DNS translations and the router just acts as DMVPN router.

Do we have to keep all these below commands ON, as the customer's security audit report recommends us to turn off http and https in router, else it flags them as High vulnerability. Any suggestions would be helpful.

ip http server
ip http secure-server
ip http secure-trustpoint SLA-TrustPoint
ip http client source-interface GigabitEthernet0/0/0

Flavio Miranda · ‎06-29-2023

For the licesing you dont need http or https. You can shut it down if you dont use it for device management

The ip domain lookup is often disable because it can cause and odd behavior when you type a wrong command and the router try to resolve the command and block the cli until the lookup times out.

But in your case the domain lookup is necessary

But Http and https is not. It is only good for device management via web.

MSD1001 · ‎06-30-2023

Flavio to our rescue again Thanks a lot sir !!

MSD1001 · ‎06-30-2023

Flavio, are we getting this dilog box because we are using a self signed certificate ?

Flavio Miranda · ‎06-30-2023

It seems it is.

Rajesh11735 · ‎06-30-2023

Thanks again! I have disabled those commands and licensing renewal works as expected.

Rajesh11735 · ‎07-01-2023

Flavio,

After enabling domain lookup, the routers started facing high cpu utilization. I have pasted few info here. Kindly have a look in this scenario.


sh ver
Cisco IOS XE Software, Version 17.02.02
Cisco IOS Software [Amsterdam], ISR Software (ARMV8EL_LINUX_IOSD-UNIVERSALK9-M), Version 17.2.2, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2020 by Cisco Systems, Inc.
Cisco IOS-XE software, Copyright (c) 2005-2020 by cisco Systems, Inc.

Router uptime is 3 hours, 26 minutes
Uptime for this control processor is 3 hours, 28 minutes
System returned to ROM by Reload Command at 06:15:08 summer Sat Jul 1 2023
System restarted at 06:18:35 summer Sat Jul 1 2023
System image file is "bootflash:c1100-universalk9.17.02.02.SPA.bin"
Last reload reason: Reload Command

cisco C1121-4P (1RU) processor with 1428661K/6147K bytes of memory.
Processor board ID XXXXXX
Router operating mode: Autonomous
1 Virtual Ethernet interface
6 Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
2945023K bytes of flash memory at bootflash:.

Configuration register is 0x2102

111111111144444111111111111111 11111
100
90
80
70
60
50
40
30
20
10
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per second (last 60 seconds)





411111111111111222211211121121221122111111112222222222222222
100
90
80
70
60
50
40
30
20
10
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%




99989999999999999999999999989999999999999998989
323523322222332324333333397979996799999180979989999598489998969798995999
100 *** ********** * ********** * ************* *
90 *********************************************
80 *********************************************
70 *********************************************
60 *********************************************
50 *********************************************
40 *********************************************
30 *********************************************
20 *********************************************
10 * #############################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%

*Jul 1 05:35:09: %PLATFORM-4-ELEMENT_WARNING: R0/0: smand: RP/0: Used Memory value 93% exceeds warning level 88%. Top memory allocators are: Process: keyman_rp_0. Tracekey: 1#91d9d4470b4beefceec6951822ffe414 Callsite ID: 3765957633 (diff_call: 175830). Process: cli_agent_rp_0. Tracekey: 1#ad5570b63160833ef7d4b402af09fe56 Callsite ID: 3490374658 (diff_call: 1967). Process: cpp_cp_svr_fp_0. Tracekey: 1#0a073a796857f2331f01cab359adceef Callsite ID: 3629618176 (diff_call: 625).

Regards,
Rajesh

Flavio Miranda · ‎07-01-2023

Hi @Rajesh11735

Let me understand something that I may did not. As per the config you just pasted before, your router´s licensing is installed on the router, right? You do not use smart licensing?

appxk9 None Smart License None
uck9 None Smart License None
securityk9 securityk9 Smart License securityk9
ipbase ipbasek9 Smart License ipbasek9

If that is the case, you dont need to worry about the logs

"Failure reason: Fail to send out Call Home HTTP message. (or)
Failure reason: SACL http unknown host"

The logs is related to smart licensing. And what you can do instead, is disable call home. If that is the case, try to disable call home

ISR#configure terminal
ISR(config)#no service call-home

Now, if you use smart licensing then we need to understand this high CPU problem.

Rajesh11735 · ‎07-01-2023

Thanks for checking, Flavio.

We do use smart licensing, but when I monitored the router activity for the last 2 days, I saw

1) either licensing renewal attempt fails (or) the license authorization at any given point. this wont be a problem until the license authorization is done, i feel.

2) CPU memory taking a huge hit and one of the routers had this log as well. 

Jun 30 10:06:53: %SYS-3-CPUHOG: Task is running for (2936)msecs, more than (2000)msecs (2/2),process = SAConversionPoll.
-Traceback= 1#9ee110f3eb0340ce68a15fb957640199 :400000+48D5B84 iosd_crb_crankshaft_unix:7F7370E000+668B4 linux-vdso:7F8D84B000+4F0 :400000+B3431FC :400000+38001FC :400000+B3386DC :400000+4F17C0 :400000+4F16D8 :400000+506878 :400000+5ECE94 :400000+5EBCAC :400000+5C5C7C :400000+5C6DB8 :400000+5E7998 :400000+63F688 :400000+3AAF570
Jun 30 10:07:01: %SYS-3-CPUHOG: Task is running for (2672)msecs, more than (2000)msecs (25/25),process = SAConversionPoll.
-Traceback= 1#9ee110f3eb0340ce68a15fb957640199 :400000+48D5B84 iosd_crb_crankshaft_unix:7F7370E000+668B4 linux-vdso:7F8D84B000+4F0 iosd_crb_crankshaft_unix:7F7370E000+66CF8 :400000+48AF96C :400000+380CFB0 :400000+37F79F8 :400000+37FC4DC :400000+B338670 :400000+4F25E0 :400000+713AC8 :400000+5F73E4 :400000+66744C :400000+68CBC8 :400000+632280 :400000+690D94
Jun 30 10:07:01: %SYS-3-CPUHOG: Task is running for (3544)msecs, more than (2000)msecs (25/25),process = SAConversionPoll.
-Traceback= 1#9ee110f3eb0340ce68a15fb957640199 :400000+48D5B84 iosd_crb_crankshaft_unix:7F7370E000+668B4 linux-vdso:7F8D84B000+4F0 :400000+62832C :400000+621BFC :400000+623084 :400000+63225C :400000+690D94 :400000+6911C8 :400000+5502C4 :400000+5ECF00 :400000+5EBCAC :400000+5C5C7C :400000+5C6DB8 :400000+5E7998 :400000+63F688

=========

Below are some outputs for your reference:



no license feature hseck9
license udi pid C1121-4P sn XXXXXX
license boot level securityk9
license smart url default
license smart transport smart

no ip http server
no ip http secure-server
ip http secure-trustpoint SLA-TrustPoint
ip http client source-interface GigabitEthernet0/0/0

call-home
! If contact email address in call-home is configured as sch-smart-licensing@cisco.com
! the email address configured in Cisco Smart License Portal will be used as contact email address to send SCH notifications.
contact-email-addr sch-smart-licensing@cisco.com
profile "CiscoTAC-1"
active
destination transport-method http

Router#sh licen all
Smart Licensing Status
======================

Smart Licensing is ENABLED

Registration:
Status: REGISTERED
Smart Account: XXXXXXXXX
Virtual Account: ACCXXXX
Export-Controlled Functionality: ALLOWED
Initial Registration: SUCCEEDED on Aug 16 17:03:45 2022 summer
Last Renewal Attempt: FAILED on Jul 01 06:13:57 2023 summer
Failure reason: Fail to send out Call Home HTTP message.
Next Renewal Attempt: Jul 16 17:50:54 2023 summer
Registration Expires: Aug 16 16:53:43 2023 summer

License Authorization:
Status: AUTHORIZED on Jul 01 10:19:34 2023 summer
Last Communication Attempt: SUCCEEDED on Jul 01 10:19:34 2023 summer
Next Communication Attempt: Jul 31 10:19:34 2023 summer
Communication Deadline: Aug 16 16:53:43 2023 summer

License Conversion:
Automatic Conversion Enabled: False
Status: Not started

Export Authorization Key:
Features Authorized:
<none>

Utility:
Status: DISABLED

Data Privacy:
Sending Hostname: yes
Callhome hostname privacy: DISABLED
Smart Licensing hostname privacy: DISABLED
Version privacy: DISABLED

Transport:
Type: Smart
URL: https://smartreceiver.cisco.com/licservice/license
Proxy:
Not Supported

Miscellaneus:
Custom Id: <empty>

License Usage
==============

Cisco 1100 Series with 4 LAN Ports , Security License (ISR_1100_4P_Security):
Description: Cisco 1100 Series with 4 LAN Ports , Security License
Count: 1
Version: 1.0
Status: AUTHORIZED
Export status: NOT RESTRICTED

Product Information
===================
UDI: PID:C1121-4P,SN:XXXXXXX

Agent Version
=============
Smart Agent for Licensing: 4.11.5_rel/41

Reservation Info
================
License reservation: DISABLED

Flavio Miranda · ‎07-01-2023

Hi

I saw some posts about this problem on the forum and outside. One of them fix the problem by enabling

ip cef

I am not sure about that but you can try.

Another solution and this one I found more interesting, would be deny DNS traffic inbound. I would try that also

Create an ACL

ip access-list extended 100

deny tcp any any eq 53

deny udp any any eq 53

Apply to the Internet interface

Rajesh11735 · ‎07-01-2023

Thanks Flavio!

Looks like CEF Is already enabled in the router and I have denied only inbound dns traffic. Hopefully, this works.

Router#sh ip cef sum
IPv4 CEF is enabled for distributed and running
VRF Default
42 prefixes (42/0 fwd/non-fwd)
Table id 0x0
Database epoch: 2 (42 entries at this epoch)

Flavio Miranda · ‎07-01-2023

Great. I believe the ACL make more sense. High CPU means the router is processing lots more information and this probably related to DNS queries coming from outside.

Hopefullly this fix the problem and you can get all sorted.

Rajesh11735 · ‎07-02-2023

Flavio,

I had my fingers crossed too, but unfortunately denying inbound DNS didnt work. We have 2 setups where in the first; I have applied ACL for inbound traffic. I see few hits in those rules, but the CPU has spiked twice to 100% in last 60 minutes. Had to turn off domain lookup to prevent any issues.

I observed below processes have clogged CPU resources big time.


SAGetRUMIds
SAUtilRepSave
SAMsgThread
keyman_rp_0
cli_agent_rp_0
cpp_cp_svr_fp_0
SAConversionPoll

interface GigabitEthernet0/0/0
description WAN interface
ip address 1.1.1.1 255.255.255.248
no ip redirects
ip nat outside
ip access-group OutsideToSelfACL in
negotiation auto
ip virtual-reassembly

Extended IP access list OutsideToSelfACL
10 deny tcp any any eq domain (5 matches)
20 deny udp any any eq domain (10 matches)
40 permit udp any host 1.1.1.1 eq isakmp (41 matches)
50 permit udp any host 1.1.1.1 eq non500-isakmp (5 matches)
60 permit ahp any host 1.1.1.1
70 permit esp any host 1.1.1.1 (39159 matches)
80 permit gre any host 1.1.1.1
90 permit tcp 2.2.2.0 0.0.0.255 any eq 22
100 permit tcp 2.2.2.0 0.0.255.255 any eq 22 (289 matches)

in the 2nd scenario, we have configured ZBF and I applied those entries to the Outside traffic to device (Self) ACL. We haven't applied ACL's to interfaces as we didnt see any issues till date. I have attached the config file for your reference.