IOX - App unhealthy - Stderr: Error: link instance creation failed - link creation failed - Secure storage has already created a key: secure storage error

[cvailler]

Hello,

My router is managed in IOT Operation Center.
I have uploaded EI-Agent (Edge Intelligence) application.
The application is running, but the agent is not registering in EI-Cloud.

The application in IOX Local Device Manager is showing : app : unhealthy : 

Stderr: Error: link instance creation failed - link creation failed - Secure storage has already created a key: secure storage error

And in the log of the application : 

2020-09-29 16:18:04.702971] INFO(1000): Starting broker version, 1.5.0.9486a0df7
[2020-09-29 16:18:04.703580] INFO(1001): Broker pid, 3015
[2020-09-29 16:18:04.703853] INFO(3059): Signal handler registered
[2020-09-29 16:18:06.448742] FATAL(3017): Internal error, Secure storage has already created a key: secure storage error
[2020-09-29 16:18:06] Terminated broker with exit status ${ret}
[2020-09-29 16:18:06] Restarting broker in 10 seconds

 In system troubleshoot : 

eid: IR809_JMX2017X05G
pfm: IR809G-LTE-GA-K9
s/n: JMX2017X05G
images: Lnx: 1.11.0.5., IOx: 1.11.0:r/1.11.0.0:dfba2bf
boot: 2020-09-29 15:43:23
time: 2020-09-29 16:20:08
load: 16:20:08 up 36 min, 0 users, load average: 0.35, 0.42, 0.30
memory: ok, used: 120/926 (12%)
disk: ok, used: /:4504/474516 (0%), /software:863680/1362080 (63%)
process: ok, running: 5/5
networking: ok
logs: warning, errors: caf (32)
apps: ok, eiagent (R)


In the CAF log : 

2020-09-29 16:14:46,531 [utils:INFO] [MonitoringService] [utils.py:584 - getPlatformProductID()] Product ID is 'IR809G-LTE-GA-K9'
2020-09-29 16:14:46,624 [pdservices:INFO] [Worker-1] [autoconfigcli.py:458 - container_event_cb()] Autoconfig network_event_cb. Event [app_health_failed] ignored.
2020-09-29 16:15:50,099 [runtime.hosting:ERROR] [MonitoringService] [monitoring.py:471 - start_monitoring()] App health script return failure 1 out:[] error:Error: link instance creation failed - link creation failed - Secure storage has already created a key: secure storage error

2020-09-29 16:15:50,141 [utils:INFO] [MonitoringService] [utils.py:584 - getPlatformProductID()] Product ID is 'IR809G-LTE-GA-K9'
2020-09-29 16:15:52,827 [pdservices:INFO] [Worker-1] [autoconfigcli.py:458 - container_event_cb()] Autoconfig network_event_cb. Event [app_health_failed] ignored.
2020-09-29 16:16:09,620 [runtime.hosting:INFO] [CP Server Thread-14] [platformcapabilities.py:308 - check_disk_status()] Calling disk status script: /home/root/iox/caf/scripts/disk_status.sh
2020-09-29 16:16:09,678 [runtime.hosting:INFO] [CP Server Thread-14] [platformcapabilities.py:311 - check_disk_status()] Executed /home/root/iox/caf/scripts/disk_status.sh: returncode: 0, message:
2020-09-29 16:16:09,725 [pdservices:INFO] [CP Server Thread-14] [linux_capabilities.py:84 - list_all_syscaps_details()] default
2020-09-29 16:16:09,726 [pdservices:INFO] [CP Server Thread-14] [linux_capabilities.py:72 - list_systemwide_syscap()] System wide System Capabilities:
2020-09-29 16:16:09,727 [pdservices:INFO] [CP Server Thread-14] [linux_capabilities.py:73 - list_systemwide_syscap()] {'setfcap': {'status': False, 'description': 'set file capabilities'}, 'sys_resource': {'status': False, 'description': 'use reserved space on ext2 filesystem, override disk quota limits,etc'}, 'sys_ptrace': {'status': False, 'description': 'to trace arbitrary process using ptrace'}, 'mac_override': {'status': False, 'description': 'override Mandatory Access Control. (for Smack)'}, 'sys_boot': {'status': False, 'description': 'use reboot and kexec_load'}, 'audit_write': {'status': False, 'description': 'write records to Kernel auditing log'}, 'mknod': {'status': False, 'description': 'create special files using mknod'}, 'sys_rawio': {'status': False, 'description': 'perform IO,port operations,ioctl,open /dev/mem and /dev/kmem map files'}, 'sys_pacct': {'status': False, 'description': 'use acct to switch process accounting on or off'}, 'setpcap': {'status': False, 'description': 'grant or remove any capabilityin callers permitted capability set'}}
2020-09-29 16:16:09,728 [pdservices:INFO] [CP Server Thread-14] [linux_capabilities.py:68 - list_app_configurable_syscap()] App configurable System Capabilities:
2020-09-29 16:16:09,730 [pdservices:INFO] [CP Server Thread-14] [linux_capabilities.py:69 - list_app_configurable_syscap()] {'sys_nice': {'status': False, 'description': 'raise process nice value(with nice and setpriority sys calls)'}, 'audit_control': {'status': False, 'description': 'enable or disable kernel auditing'}}
2020-09-29 16:16:10,556 [utils:INFO] [CP Server Thread-9] [utils.py:584 - getPlatformProductID()] Product ID is 'IR809G-LTE-GA-K9'
2020-09-29 16:16:10,686 [utils:INFO] [CP Server Thread-9] [utils.py:584 - getPlatformProductID()] Product ID is 'IR809G-LTE-GA-K9'
2020-09-29 16:16:15,091 [runtime.hosting:INFO] [CP Server Thread-14] [platformcapabilities.py:308 - check_disk_status()] Calling disk status script: /home/root/iox/caf/scripts/disk_status.sh
2020-09-29 16:16:15,136 [runtime.hosting:INFO] [CP Server Thread-14] [platformcapabilities.py:311 - check_disk_status()] Executed /home/root/iox/caf/scripts/disk_status.sh: returncode: 0, message:
2020-09-29 16:16:15,185 [pdservices:INFO] [CP Server Thread-14] [linux_capabilities.py:84 - list_all_syscaps_details()] default
2020-09-29 16:16:15,186 [pdservices:INFO] [CP Server Thread-14] [linux_capabilities.py:72 - list_systemwide_syscap()] System wide System Capabilities:
2020-09-29 16:16:15,187 [pdservices:INFO] [CP Server Thread-14] [linux_capabilities.py:73 - list_systemwide_syscap()] {'setfcap': {'status': False, 'description': 'set file capabilities'}, 'sys_resource': {'status': False, 'description': 'use reserved space on ext2 filesystem, override disk quota limits,etc'}, 'sys_ptrace': {'status': False, 'description': 'to trace arbitrary process using ptrace'}, 'mac_override': {'status': False, 'description': 'override Mandatory Access Control. (for Smack)'}, 'sys_boot': {'status': False, 'description': 'use reboot and kexec_load'}, 'audit_write': {'status': False, 'description': 'write records to Kernel auditing log'}, 'mknod': {'status': False, 'description': 'create special files using mknod'}, 'sys_rawio': {'status': False, 'description': 'perform IO,port operations,ioctl,open /dev/mem and /dev/kmem map files'}, 'sys_pacct': {'status': False, 'description': 'use acct to switch process accounting on or off'}, 'setpcap': {'status': False, 'description': 'grant or remove any capabilityin callers permitted capability set'}}
2020-09-29 16:16:15,188 [pdservices:INFO] [CP Server Thread-14] [linux_capabilities.py:68 - list_app_configurable_syscap()] App configurable System Capabilities:
2020-09-29 16:16:15,189 [pdservices:INFO] [CP Server Thread-14] [linux_capabilities.py:69 - list_app_configurable_syscap()] {'sys_nice': {'status': False, 'description': 'raise process nice value(with nice and setpriority sys calls)'}, 'audit_control': {'status': False, 'description': 'enable or disable kernel auditing'}}
2020-09-29 16:16:53,560 [runtime.hosting:ERROR] [MonitoringService] [monitoring.py:471 - start_monitoring()] App health script return failure 1 out:[] error:Error: link instance creation failed - link creation failed - Secure storage has already created a key: secure storage error

2020-09-29 16:16:53,718 [utils:INFO] [MonitoringService] [utils.py:584 - getPlatformProductID()] Product ID is 'IR809G-LTE-GA-K9'
2020-09-29 16:16:56,123 [pdservices:INFO] [Worker-1] [autoconfigcli.py:458 - container_event_cb()] Autoconfig network_event_cb. Event [app_health_failed] ignored.
2020-09-29 16:17:58,392 [runtime.hosting:ERROR] [MonitoringService] [monitoring.py:471 - start_monitoring()] App health script return failure 1 out:[] error:Error: link instance creation failed - link creation failed - Secure storage has already created a key: secure storage error

2020-09-29 16:17:58,427 [utils:INFO] [MonitoringService] [utils.py:584 - getPlatformProductID()] Product ID is 'IR809G-LTE-GA-K9'
2020-09-29 16:17:59,227 [pdservices:INFO] [Worker-1] [autoconfigcli.py:458 - container_event_cb()] Autoconfig network_event_cb. Event [app_health_failed] ignored.
2020-09-29 16:19:01,682 [runtime.hosting:ERROR] [MonitoringService] [monitoring.py:471 - start_monitoring()] App health script return failure 1 out:[] error:Error: link instance creation failed - link creation failed - Secure storage has already created a key: secure storage error

2020-09-29 16:19:01,719 [utils:INFO] [MonitoringService] [utils.py:584 - getPlatformProductID()] Product ID is 'IR809G-LTE-GA-K9'
2020-09-29 16:19:02,323 [pdservices:INFO] [Worker-1] [autoconfigcli.py:458 - container_event_cb()] Autoconfig network_event_cb. Event [app_health_failed] ignored.
2020-09-29 16:20:07,446 [runtime.hosting:ERROR] [MonitoringService] [monitoring.py:471 - start_monitoring()] App health script return failure 1 out:[] error:Error: link instance creation failed - link creation failed - Secure storage has already created a key: secure storage error

2020-09-29 16:20:07,705 [utils:INFO] [MonitoringService] [utils.py:584 - getPlatformProductID()] Product ID is 'IR809G-LTE-GA-K9'
2020-09-29 16:20:08,491 [pdservices:INFO] [Worker-1] [autoconfigcli.py:458 - container_event_cb()] Autoconfig network_event_cb. Event [app_health_failed] ignored.