Cisco 9800 High CPU every 15 minutes caused by SAMsgThread process

[DATHOZ]

We have 6 pairs of 9800-80 on HA, and we noticed that on all pairs there is a process (SAMsgThread) that runs every 15 minutes that affects the 9800 controller CPU. That process SAMsgThread is the responsible for Smart Licensing operations. Depending on the time of the day the CPU hits 100% and it may affect client transactions depending on the qty of APs that are hosted in the controller.

The controller version now is running 17.6.4 and couple weeks back was on 17.6.2.

I have a ticket opened but they are not helping much. Has anybody experienced this issue?

Looking at the the command "show license eventlog" it displays the following every 15 min:

"2022-09-30 04:19:53.428 MST SAEVT_HA_MESSAGE messageType="SmartAgentHaMsgTSFileChange"
2022-09-30 04:34:53.428 MST SAEVT_HA_MESSAGE messageType="SmartAgentHaMsgTSFileChange"
2022-09-30 04:49:53.560 MST SAEVT_HA_MESSAGE messageType="SmartAgentHaMsgTSFileChange"
2022-09-30 05:04:53.486 MST SAEVT_HA_MESSAGE messageType="SmartAgentHaMsgTSFileChange"
2022-09-30 05:19:53.554 MST SAEVT_HA_MESSAGE messageType="SmartAgentHaMsgTSFileChange"
2022-09-30 05:34:53.561 MST SAEVT_HA_MESSAGE messageType="SmartAgentHaMsgTSFileChange"
2022-09-30 05:49:53.464 MST SAEVT_HA_MESSAGE messageType="SmartAgentHaMsgTSFileChange"
2022-09-30 06:04:53.214 MST SAEVT_HA_MESSAGE messageType="SmartAgentHaMsgTSFileChange"
2022-09-30 06:19:53.462 MST SAEVT_HA_MESSAGE messageType="SmartAgentHaMsgTSFileChange"

[marce1000]

               - FYI : https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwb05103

 M.

[Rich R]

The bug Marce referenced is actually addressed by https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwa85199 which in turn refers to another bug: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwa85525 
both of which should be fixed in 17.6.5 and should not affect 17.9 at all!  Since you're using 17.9.3 you should not be affected by any of these.  But the root cause of it all was smart licensing not getting acknowledgements from Cisco so:
1. Make sure your smart licensing is working and getting acknowledgements back from Cisco license servers.
2. Use the workaround described on CSCwa85199 to clear the problem.  If you didn't get the smart licensing working (as per 1) then the problem will come back.

PS: all this assumes it is also the SAMsgThread as per the original post.  If it's something else then it's a completely different problem which should be on a new thread.
Note that iosd is what hosts the platform independent IOS processes so you should use "show proc cpu sorted" to see those processes.