topic Re: Cisco ISE 3.1 - Alarms: Smart Licensing Authorization Renewal Fail in Network Access Control

Cisco ISE 3.1 - Alarms: Smart Licensing Authorization Renewal Failure

milos_p — Fri, 24 Jun 2022 07:30:14 GMT

Hi guys,

I am running two node deployment with fresh ISE 3.1 patch3 installation.

I am getting every few days alarm like this:

Alarms: Smart Licensing Authorization Renewal Failure, with description when I open the alarm "Smart Licensing Authorization Renewal Failure: Details=Communication send error."

I have checked firewall, there is no block from ISE servers and communication is allowed in the exact timestamp of the alarm (I can see in the firewall logs).

Also I see in Licensing page, in "Last Authorization" column that there was successful authorization few hours after the alarm timestamp.

Anyone else experiencing similar behavior? Is it a bug and is there some workaround?

It's not affecting production in any way, just I am getting alarms approx. every second day for this matter.

Thanks a lot in advance!

Regards,

Milos

Re: Cisco ISE 3.1 - Alarms: Smart Licensing Authorization Renewal Fail

ahollifield — Fri, 24 Jun 2022 12:01:32 GMT

Yup, this is a common issue across multiple Cisco Smart Licensed products; not just ISE. It seems to be related to the backend Smart Licensing systems. Not sure if its unable to handle the load from customer products, some sort of DDoS protection, or what. These can be safely ignored as ISE does have a grace period.

Re: Cisco ISE 3.1 - Alarms: Smart Licensing Authorization Renewal Fail

milos_p — Fri, 24 Jun 2022 13:48:24 GMT

Hi,

I couldn't find any pattern in this, apart that it's happening every day or every second day, although as I said, not having any impact on production, as next authorization following the failed one is generally always being successful.

Regards,

Milos

Re: Cisco ISE 3.1 - Alarms: Smart Licensing Authorization Renewal Fail

ahollifield — Fri, 24 Jun 2022 14:06:13 GMT

Exactly the same behavior I am seeing across ISE, FMC, FDM, switches, etc. The issue has to be somewhere in the backend of the Smart Licensing system.

Re: Cisco ISE 3.1 - Alarms: Smart Licensing Authorization Renewal Fail

Marvin Rhoads — Fri, 24 Jun 2022 18:01:22 GMT

I see the same as @ahollifield mentioned - across multiple customers and products in different states. It is definitely a Cisco backend system issue.

Re: Cisco ISE 3.1 - Alarms: Smart Licensing Authorization Renewal Fail

milos_p — Fri, 24 Jun 2022 19:44:20 GMT

Hi guys,

Thanks a lot for the answers, you confirmed me that there is nothing wrong with my ISE deployment.

Regards,

Milos

Re: Cisco ISE 3.1 - Alarms: Smart Licensing Authorization Renewal Fail

milos_p — Tue, 28 Jun 2022 06:49:39 GMT

Hi to all,

Just to post an update as it looks like I have solved the issue.

So first question to everyone experiencing the same problem: Are you guys decrypting HTTPS traffic?

In my case, I am, and although ISE servers are matching bypass rule, it was last bypass rule after few URL bypass rules, meaning initial traffic is going to be decrypted in order to determine if traffic should be bypassed based on URL category.

When I configured hard bypass (by setting source IP addresses of ISE servers) before any kind of URL bypass rule, warning stopped and for the last 5 days not even one appeared.

Regards,

Milos

Re: Cisco ISE 3.1 - Alarms: Smart Licensing Authorization Renewal Fail

ahollifield — Tue, 28 Jun 2022 11:17:10 GMT

If you have your HTTPS decryption certificate trusted in ISE for "Cisco Services" I see no technical reason why you couldn't still keep decrypting the traffic. Not sure what the value of that would be though.

Re: Cisco ISE 3.1 - Alarms: Smart Licensing Authorization Renewal Fail

Marvin Rhoads — Tue, 28 Jun 2022 11:03:19 GMT

None of my customers who are experiencing this problem (> 10 of them) have SSL/TLS decryption (or proxy server or restriction of outbound traffic in any way) enabled.

Re: Cisco ISE 3.1 - Alarms: Smart Licensing Authorization Renewal Fail

milos_p — Tue, 28 Jun 2022 12:36:04 GMT

Hi,

Well, I really don't see any reason decrypting this traffic, so I want it to bypass decryption for ISE servers.

My problem was with bypass hitting some URL bypass rules, as I explained, and by putting IP bypass, it looks like it solved the issue for me.

Either way, I just wanted to post, as maybe someone will have same scenario as me.

If warning comes up again, I will update here as well, could be just a strange coincidence of configuring IP bypass and warnings not showing up for many days...

Regards,

Milos

Re: Cisco ISE 3.1 - Alarms: Smart Licensing Authorization Renewal Fail

Arne Bier — Tue, 28 Jun 2022 20:49:20 GMT

I have a TAC case open for this in ISE 3.0. If they come up with anything useful as a resolution I will let you know. I have never had a good experience with Smart Licensing, as much as I find the concept interesting, the implementation leaves me unimpressed.

Re: Cisco ISE 3.1 - Alarms: Smart Licensing Authorization Renewal Fail

Marcelo Morais — Wed, 29 Jun 2022 06:04:13 GMT

Hi @Arne Bier and @milos_p ,

please take a look at CSCwa79591 Smart Licensing Authorization Renewal Failure:Communication send error.

Note: unfortunately every time I click the link I am redirected to the following CSCwa72274 Intermittent "Communication Send error" while registering to smart licensing, hope that someone could change that, I would like to read the entire CSCwa79591 info : ) !!!

Hope this helps !!!

Re: Cisco ISE 3.1 - Alarms: Smart Licensing Authorization Renewal Fail

milos_p — Wed, 29 Jun 2022 07:58:39 GMT

Hi,

Just checked bug info, says "Bug CSCwa79591 is a duplicate of the bug displayed below" as it redirects to CSCwa72274 .

"Conditions: This issue is seen intermittently when backend systems are overloaded"

Workaround: System retries would automatically eventually get the messages thru, if not manual execution of authorization renewal or sync commands would send the data.

No fixes available.

BTW, bug CSCwa72274 is related to Cisco License Manager, at least this is what's written in details section.

Just to point again, as it is really funny, but with bypass that I implemented, I still didn't get any warning for last 5 days.

Re: Cisco ISE 3.1 - Alarms: Smart Licensing Authorization Renewal Fail

milos_p — Wed, 29 Jun 2022 07:59:20 GMT

Hi Arne,

Great, thanks a lot, keep us posted here what TAC said.

Re: Cisco ISE 3.1 - Alarms: Smart Licensing Authorization Renewal Fail

Marcelo Morais — Wed, 29 Jun 2022 14:44:16 GMT

Hi @milos_p ,

thanks, but I would like to read the entire info from the CSCwa79591 bug (please take a look to the picture that I provided, I'm not able to check the "Conditions") ... I think that CSCwa7951 provided a better description for the issue.

Regards

Re: Cisco ISE 3.1 - Alarms: Smart Licensing Authorization Renewal Fail

milos_p — Wed, 29 Jun 2022 15:47:53 GMT

Hi Marcelo,

When I click on the link you provided, it takes me right here:

Re: Cisco ISE 3.1 - Alarms: Smart Licensing Authorization Renewal Fail

Marcelo Morais — Wed, 29 Jun 2022 16:05:51 GMT

Hi @milos_p ,

I'm able to do this:

but if I click the link, then I am redirected to the other bug.

Regards

Re: Cisco ISE 3.1 - Alarms: Smart Licensing Authorization Renewal Fail

Arne Bier — Wed, 29 Jun 2022 20:15:45 GMT

I am using an on-prem CSSM that is air-gapped. Not that many devices talking Smart to it. Not sure why anything would be getting overloaded. It makes very little sense. Thee boxes have GigE interfaces, loads of RAM and CPU and the ability to perform retries - how overloaded does a system have to be? The CSSM CPU and MEM stats are super low when checking in the GUI.

Java bloat.

Re: Cisco ISE 3.1 - Alarms: Smart Licensing Authorization Renewal Fail

Marcelo Morais — Wed, 29 Jun 2022 21:20:36 GMT

Hi @Arne Bier ,

understood ... any Maximum Resource Limit Reached at Operations > Reports > Reports > Audit > Operations Audit?

Note: please take a look at: CSCwb29140 Threads getting exhuast post moving to latest patches were nss rpm is updated(Only 3.0p5&2.7p7,3.1P1., there is a Hot Patch for that.

Regards

Re: Cisco ISE 3.1 - Alarms: Smart Licensing Authorization Renewal Fail

milos_p — Mon, 04 Jul 2022 07:31:14 GMT

Hi guys,

Just to give you an update from my side.

After many days of not having warning, it appeared again, so it was just a strange coincidence of not showing and me putting IP bypass for so many days.