UCCE License Authorization and Registration Reverting Back to Old Data

[ejlackey]

Hello,

I have an issue with a UCCE 12.6(2) system reverting back to old registration and authorization data.

Using the Unified Contact Center Enterprise Management web portal running on the AW server, under Infrastructure>License Management, I can use a token I created on the Smart Account and successfully register and authorize the system. However, after a few minutes, it reverts back to the previous registration and authorization times and the system says "One or more product instances in the deployment are in the Enforcement state."

I've repeated the token and registration steps but it sill does the same thing. I have plenty of licenses and this product instance has registered before. The Smart Account does not show any errors for this system.

One of the recent things performed on this system is updating the certificates for some of the components and in so doing, backups were made of the cacerts files.

The SmartAgent log from tomcat mentions that it is failing to save to the keystore, so I'm wondering if I'm having some write permission issues on the system, or is there a different keystore for licensing? I tried restoring the cacerts backup that I made, but that did not resolve the issue. 

The other recent item is that the Smart Licenses were renewed. There were no changes though. Same number of Flex standard and premium seats.

Attached are screenshots and the SmartAgent log.

[Ramamoorthy Shanmugam]

Yes for UCCE deployment AW is responsible for managing the license and Keystore. if you change a few component certificates please follow the standard certificate exchange process again.

Note: I also had the same issue for one customer system was authorized and given some time likely 1 to 3 hours the status was changed automatically.  if not please restart the AW servers and check.

but your case saving to keystore failed. Please check the permission or re-do the certificate exchange process for updating the certificates for some of the components.

[ejlackey]

I have re-done the process for exchanging certificates.  No errors and all systems are communicating fine. Also restarted the AW.   I am monitoring wireshark and there are no errors in any of the SSL connections.  I re-authorized the license.  The screen updated with the correct registration and authorization information.  However, it reverted back to its previous Dates and Times after 10-15 minutes.

0000000500: cisco-a-awhds: Dec 21 2023 12:53:13.221 -0600: %_168-SMARTLICENSEOPERATIONS-6-com.cisco.ccbu.smartagentmanager.SmartAgentManager: Renew Authorization manually
0000000501: cisco-a-awhds: Dec 21 2023 12:53:13.221 -0600: %_168-SMARTLICENSEOPERATIONS-6-com.cisco.nesla.agent.impl.SmartAgentRequestHandler: Async Request Processor: enter renewAuth()
0000000502: cisco-a-awhds: Dec 21 2023 12:53:13.299 -0600: %_168-SMARTLICENSEOPERATIONS-6-com.cisco.nesla.agent.impl.SmartAgentRequestHandler: Async Request Processor: exit renewAuth()
0000000505: cisco-a-awhds: Dec 21 2023 12:53:13.971 -0600: %_Thread-23-3-com.cisco.nesla.agent.impl.manager.AgentKeyStoreManager: saving to keystore failed
0000000506: cisco-a-awhds: Dec 21 2023 12:53:13.971 -0600: %_Thread-23-3-com.cisco.nesla.agent.impl.manager.AgentKeyStoreManager: saving to keystore failed
0000000507: cisco-a-awhds: Dec 21 2023 12:53:14.002 -0600: %_Thread-23-3-com.cisco.nesla.agent.impl.manager.AgentKeyStoreManager: saving to keystore failed
0000000508: cisco-a-awhds: Dec 21 2023 12:53:14.002 -0600: %_Thread-23-3-com.cisco.nesla.agent.impl.manager.AgentKeyStoreManager: saving to keystore failed
0000000509: cisco-a-awhds: Dec 21 2023 12:53:14.002 -0600: %_Thread-23-6-com.cisco.ccbu.smartagentmanager.listener.GlobalNotificationListener: Received GlobalNotification: GlobalNotification [failReasonCode=Success, failMessage=Successful., enforceMode=InCompliance, allowRestricted=false, notificationType=NotifyEnforcementMode, agentID=null, tenantContextId=N.A.1703184740243]
0000000514: cisco-a-awhds: Dec 21 2023 12:53:14.002 -0600: %_Thread-23-6-com.cisco.ccbu.smartagentmanager.SmartAgentManager: Received entitlementList: [Entitlement [entitlementTag=regid.2019-06.com.cisco.CCE_FLEX_STD,12.5_352197e4-e16b-4e40-9da1-e817e9b3cbe7, displayName=null, entitlementVersion=12.51, enforceMode=Init, requestedDate=null, expiryDate=null, requestedCount=0, vendor=null, description=null, exportStatus=NOT_RESTRICTED, featureName=null, featureDescription=null, listener=com.cisco.ccbu.smartagentmanager.listener.EntitlementNotificationListener@1d84c95, subscription=null, intervals=null, isSlrInstalled=false, aggregatedSLRCount=0, slrTermInfoList=null], Entitlement [entitlementTag=regid.2019-06.com.cisco.CCE_FLEX_PREMIUM,12.5_23e13124-061b-40f1-ae20-2120fc363961, displayName=null, entitlementVersion=12.51, enforceMode=Init, requestedDate=null, expiryDate=null, requestedCount=0, vendor=null, description=null, exportStatus=NOT_RESTRICTED, featureName=null, featureDescription=null, listener=com.cisco.ccbu.smartagentmanager.listener.EntitlementNotificationListener@1d84c95, subscription=null, intervals=null, isSlrInstalled=false, aggregatedSLRCount=0, slrTermInfoList=null]]
0000000515: cisco-a-awhds: Dec 21 2023 12:53:14.002 -0600: %_Thread-23-6-com.cisco.ccbu.smartagentmanager.SmartAgentManager: Received entitlementList: [Entitlement [entitlementTag=regid.2019-06.com.cisco.CCE_FLEX_STD,12.5_352197e4-e16b-4e40-9da1-e817e9b3cbe7, displayName=null, entitlementVersion=12.51, enforceMode=Init, requestedDate=null, expiryDate=null, requestedCount=0, vendor=null, description=null, exportStatus=NOT_RESTRICTED, featureName=null, featureDescription=null, listener=com.cisco.ccbu.smartagentmanager.listener.EntitlementNotificationListener@1d84c95, subscription=null, intervals=null, isSlrInstalled=false, aggregatedSLRCount=0, slrTermInfoList=null], Entitlement [entitlementTag=regid.2019-06.com.cisco.CCE_FLEX_PREMIUM,12.5_23e13124-061b-40f1-ae20-2120fc363961, displayName=null, entitlementVersion=12.51, enforceMode=Init, requestedDate=null, expiryDate=null, requestedCount=0, vendor=null, description=null, exportStatus=NOT_RESTRICTED, featureName=null, featureDescription=null, listener=com.cisco.ccbu.smartagentmanager.listener.EntitlementNotificationListener@1d84c95, subscription=null, intervals=null, isSlrInstalled=false, aggregatedSLRCount=0, slrTermInfoList=null]]
0000000516: cisco-a-awhds: Dec 21 2023 12:53:14.002 -0600: %_Thread-23-6-com.cisco.ccbu.smartagentmanager.listener.GlobalNotificationListener: Received GlobalNotification: GlobalNotification [failReasonCode=Success, failMessage=Successful., enforceMode=NotApplicable, allowRestricted=false, notificationType=NotifyAuthRenewSuccess, agentID=null, tenantContextId=N.A.1703184740243]
0000000521: cisco-a-awhds: Dec 21 2023 12:53:14.002 -0600: %_Thread-23-6-com.cisco.ccbu.smartagentmanager.SmartAgentManager: Received entitlementList: [Entitlement [entitlementTag=regid.2019-06.com.cisco.CCE_FLEX_STD,12.5_352197e4-e16b-4e40-9da1-e817e9b3cbe7, displayName=null, entitlementVersion=12.51, enforceMode=Init, requestedDate=null, expiryDate=null, requestedCount=0, vendor=null, description=null, exportStatus=NOT_RESTRICTED, featureName=null, featureDescription=null, listener=com.cisco.ccbu.smartagentmanager.listener.EntitlementNotificationListener@1d84c95, subscription=null, intervals=null, isSlrInstalled=false, aggregatedSLRCount=0, slrTermInfoList=null], Entitlement [entitlementTag=regid.2019-06.com.cisco.CCE_FLEX_PREMIUM,12.5_23e13124-061b-40f1-ae20-2120fc363961, displayName=null, entitlementVersion=12.51, enforceMode=Init, requestedDate=null, expiryDate=null, requestedCount=0, vendor=null, description=null, exportStatus=NOT_RESTRICTED, featureName=null, featureDescription=null, listener=com.cisco.ccbu.smartagentmanager.listener.EntitlementNotificationListener@1d84c95, subscription=null, intervals=null, isSlrInstalled=false, aggregatedSLRCount=0, slrTermInfoList=null]]
0000000522: cisco-a-awhds: Dec 21 2023 12:53:14.002 -0600: %_Thread-23-6-com.cisco.ccbu.smartagentmanager.SmartAgentManager: Received entitlementList: [Entitlement [entitlementTag=regid.2019-06.com.cisco.CCE_FLEX_STD,12.5_352197e4-e16b-4e40-9da1-e817e9b3cbe7, displayName=null, entitlementVersion=12.51, enforceMode=Init, requestedDate=null, expiryDate=null, requestedCount=0, vendor=null, description=null, exportStatus=NOT_RESTRICTED, featureName=null, featureDescription=null, listener=com.cisco.ccbu.smartagentmanager.listener.EntitlementNotificationListener@1d84c95, subscription=null, intervals=null, isSlrInstalled=false, aggregatedSLRCount=0, slrTermInfoList=null], Entitlement [entitlementTag=regid.2019-06.com.cisco.CCE_FLEX_PREMIUM,12.5_23e13124-061b-40f1-ae20-2120fc363961, displayName=null, entitlementVersion=12.51, enforceMode=Init, requestedDate=null, expiryDate=null, requestedCount=0, vendor=null, description=null, exportStatus=NOT_RESTRICTED, featureName=null, featureDescription=null, listener=com.cisco.ccbu.smartagentmanager.listener.EntitlementNotificationListener@1d84c95, subscription=null, intervals=null, isSlrInstalled=false, aggregatedSLRCount=0, slrTermInfoList=null]]

[Ramamoorthy Shanmugam]

Please Ask Cisco TAC might be we are hitting a bug. Even though I verified all the ES for 12.6 not able to find any Bug ID for your scenario.

[mparra.fusionet]

I had the same problem @ejlackey , the issue ended up being that the PrincipalAW (where you launch the SPOG cceadmin page) couldn't connect to the Router, in the clgr logs of the AW it would say :

dis-uaw Trace: [client: Tomcat9(upcc)] Update of central controller: failed, message: Another user has changed the configuration data. Re-retrieve the data and try save again. If the problem persists, you need to reload your local database. You can do this using the Initialize Local Database tool., LastUpdateKey: XXXXXX

Going to the Logger the clgr service was not launching to connect to the router

05:39:03:798 la-clgr Trace: LastUpdateKey for A Configuration is 10944034590484.0
05:39:03:798 la-clgr Trace: LastUpdateKey for B Configuration is 11040251725802.0
05:39:03:798 la-clgr The sideA Logger cannot come online in duplexed mode because its database is out of date.
05:39:10:600 la-clgr Trace: NodeManagerHandler: Logger Performing State Transfer
05:39:20:605 la-clgr Trace: NodeManagerHandler: Logger Performing State Transfer
05:39:20:605 la-clgr Trace: Logger Performing State Transfer for 10004 Milliseconds
05:39:30:618 la-clgr Trace: NodeManagerHandler: Logger Performing State Transfer
05:39:30:618 la-clgr Trace: Logger Performing State Transfer for 20017 Milliseconds
05:39:33:805 la-clgr Requesting MDS termination due to error.
05:39:33:805 la-clgr Connection to MDS process closed.


In the router the clgr service was not connecting because the logger was stopping it every time it was starting whenever it launched to connect in the logger as you can see in the log below.

00:00:00:038 ra-mds Midnight: 4/16/2025.
00:00:23:265 ra-mds Client: clgr, stopping due to error.
00:00:23:280 ra-mds Client: clgr, state transfer operation aborted.
Last API Error [10053]: An established connection was aborted by the software in your host machine.
00:00:34:814 ra-mds Client: clgr, registered with handle: 36.
00:00:34:830 ra-mds Client: clgr, started.
00:01:06:826 ra-mds Client: clgr, stopping due to error.
00:01:06:826 ra-mds Client: clgr, state transfer operation aborted.
Last API Error [10053]: An established connection was aborted by the software in your host machine.
00:01:18:358 ra-mds Client: clgr, registered with handle: 36.
00:01:18:358 ra-mds Client: clgr, started.
00:01:50:287 ra-mds Client: clgr, stopping due to error.
00:01:50:303 ra-mds Client: clgr, state transfer operation aborted.
Last API Error [10053]: An established connection was aborted by the software in your host machine.
00:02:01:898 ra-mds Client: clgr, registered with handle: 36.
00:02:01:898 ra-mds Client: clgr, started.
00:02:33:814 ra-mds Client: clgr, stopping due to error.
00:02:33:814 ra-mds Client: clgr, state transfer operation aborted.
Last API Error [64]: The specified network name is no longer available.
00:02:45:348 ra-mds Client: clgr, registered with handle: 36.
00:02:45:348 ra-mds Client: clgr, started.
00:03:17:312 ra-mds Client: clgr, stopping due to error.
00:03:17:312 ra-mds Client: clgr, state transfer operation aborted.
Last API Error [10053]: An established connection was aborted by the software in your host machine.

Having triangulated the problem to a DB not synched (probably caused by a VM maintenance that wasn't done correctly or an extended network outage), I did the DB synched as documented below and the issue was solved (it took like 5 minutes for the SPOG to refresh by itself).


I have seen other issues where it is a certificate thing (this wasn't the case) the AW might not have the self signed certificate of the router or Logger, or if these VMs have CA signed certs their application certificates are expired, also I have seen where the root and intermediate certificates of the CA signed certs of the router and logger are expired and that of course breaks the application certificates causing this issue, you will probably see more alerts in the Tomcat logs if this is the case.