Re: RADkit autoboot on RHEL

Stuart-ITGL · ‎09-26-2023

I'm trying to get RADkit service 1.5.8 to autoboot on startup on RHEL 9. Followed the steps outlined in the documentation here - https://radkit.cisco.com/docs/pages/service_advanced.html - but it keeps exiting - see output below:

radkit.service - RADKit Service
     Loaded: loaded (/etc/systemd/system/radkit.service; enabled; preset: disabled)
     Active: activating (auto-restart) (Result: exit-code) since Tue 2023-09-26 12:07:02 BST; 2s ago
    Process: 5553 ExecStart=/home/radkit/.local/radkit/versions/1.5.8/bin/radkit-service run --headless (code=exited, status=203/EXEC)
   Main PID: 5553 (code=exited, status=203/EXEC)
        CPU: 6ms

Sep 26 12:07:02 RADkit-RHEL systemd[1]: radkit.service: Main process exited, code=exited, status=203/EXEC
Sep 26 12:07:02 RADkit-RHEL systemd[1]: radkit.service: Failed with result 'exit-code'.

If I manually start the service then that runs with no issues.

Where am I going wrong?

Timothy Stewart · ‎09-26-2023

Hello,

The status 203/EXEC usually means the executable could not be found:

https://www.freedesktop.org/software/systemd/man/systemd.exec.html#%24SERVICE_RESULT

I can reproduce this same error by creating a service with a invalid path to the executable, for example:

[root@cats-web system]# more radkit-test.service
[Unit]
Description=Radkit Service
After=network.target
Requires=network.target

[Service]
Type=simple
User=tstewart
Group=tstewart
Restart=on-failure
RestartSec=3
Environment="RADKIT_SERVICE_SUPERADMIN_PASSWORD_FILE=/home/tstewart/superadmin-password"
ExecStart=/home/tstewart/miniconda3/envs/radkit/bin/radkit-service1 run --headless. <-------does not exist

[Install]
WantedBy=multi-user.target

When I start the service, I get the same status=203/EXEC:

[root@cats-web system]# systemctl status radkit-test.service
● radkit-test.service - Radkit Service
   Loaded: loaded (/etc/systemd/system/radkit-test.service; disabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: exit-code) since Tue 2023-09-26 08:31:44 EDT; 2s ago
  Process: 9460 ExecStart=/home/tstewart/miniconda3/envs/radkit/bin/radkit-service1 run --headless (code=exited, status=203/EXEC)
 Main PID: 9460 (code=exited, status=203/EXEC)

Sep 26 08:31:44 cats-web systemd[1]: radkit-test.service: main process exited, code=exited, status=203/EXEC
Sep 26 08:31:44 cats-web systemd[1]: Unit radkit-test.service entered failed state.
Sep 26 08:31:44 cats-web systemd[1]: radkit-test.service failed.

Can you verify the path/binary that you are attempting to start?

Thanks

Tim

Stuart-ITGL · ‎09-26-2023

Hi Tim,

[radkit@RADkit-RHEL system]$ more radkit.service
[Unit]
Description=RADkit Service
After=network.target
Requires=network.target

[Service]
Type=simple
User=radkit
Restart=on-failure
RestartSec=3
LoadCredential=superadmin-password:/home/radkit/.local/bin/superadmin-password
ExecStart=/home/radkit/.local/bin/radkit-service run --headless

[Install]
WantedBy=multi-user.target

As you can see from the radkit.service conf file, the path to the executable is "/home/radkit/.local/bin/radkit-service"

If I go to that directory then I can see this:

[radkit@RADkit-RHEL bin]$ pwd
/home/radkit/.local/bin

[radkit@RADkit-RHEL bin]$ ls -lh
total 4.0K
lrwxrwxrwx. 1 radkit radkit 59 Sep 25 16:25 radkit-client -> /home/radkit/.local/radkit/versions/1.5.8/bin/radkit-client
lrwxrwxrwx. 1 radkit radkit 60 Sep 25 16:25 radkit-control -> /home/radkit/.local/radkit/versions/1.5.8/bin/radkit-control
lrwxrwxrwx. 1 radkit radkit 64 Sep 25 16:25 radkit-interactive -> /home/radkit/.local/radkit/versions/1.5.8/bin/radkit-interactive
lrwxrwxrwx. 1 radkit radkit 62 Sep 25 16:25 radkit-medic-gui -> /home/radkit/.local/radkit/versions/1.5.8/bin/radkit-medic-gui
lrwxrwxrwx. 1 radkit radkit 68 Sep 25 16:25 radkit-network-console -> /home/radkit/.local/radkit/versions/1.5.8/bin/radkit-network-console
lrwxrwxrwx. 1 radkit radkit 59 Sep 25 16:25 radkit-remote -> /home/radkit/.local/radkit/versions/1.5.8/bin/radkit-remote
lrwxrwxrwx. 1 radkit radkit 60 Sep 25 16:25 radkit-service -> /home/radkit/.local/radkit/versions/1.5.8/bin/radkit-service
lrwxrwxrwx. 1 radkit radkit 64 Sep 25 16:25 radkit-service-gui -> /home/radkit/.local/radkit/versions/1.5.8/bin/radkit-service-gui
-rw-------. 1 root   root   13 Sep 25 16:28 superadmin-password

The path matches the symbolic link that is created as part of the install/bootstrapping process. I've also tried adding in the full "proper" path that the symbolic link refers to as well with the same result

Timothy Stewart · ‎09-26-2023

Hello,

The journalctl messages will give you a better idea as to what is happening, and you can view it using the command "journalctl -u radkit.service" to see all of the messages for the newly created radkit.serivce. I am guessing your issue is actually with the LoadCredential systemd directive and it not being supported in your version of systemd. The LoadCrendential directive is supported in systemd v247 and above (systemctl --version will show you the version) and if it is not supported, then you will see a error message similar to this in the journalctl:

/etc/systemd/system/radkit.service:11: Unknown lvalue 'LoadCredential' in section 'Service'

If that is the case, then I would suggest setting the environment variable RADKIT_SERVICE_SUPERADMIN_PASSWORD_FILE in the service file by using this directive:

Environment="RADKIT_SERVICE_SUPERADMIN_PASSWORD_FILE=/home/radkit/.local/bin/superadmin-password"

Tim

Stuart-ITGL · ‎09-27-2023

Hi Tim.

Systemd version is 252 so should support the LoadCredential directive.

The journelctl looks like it has provided the answer though - Permission Denied

Sep 26 11:12:14 RADkit-RHEL systemd[1]: Started RADKit Service.
Sep 26 11:12:14 RADkit-RHEL systemd[2254]: radkit.service: Failed to locate executable /home/radkit/.local/bin/radkit-service: Permission denied
Sep 26 11:12:14 RADkit-RHEL systemd[2254]: radkit.service: Failed at step EXEC spawning /home/radkit/.local/bin/radkit-service: Permission denied
Sep 26 11:12:14 RADkit-RHEL systemd[1]: radkit.service: Main process exited, code=exited, status=203/EXEC
Sep 26 11:12:14 RADkit-RHEL systemd[1]: radkit.service: Failed with result 'exit-code'.
Sep 26 11:12:17 RADkit-RHEL systemd[1]: radkit.service: Scheduled restart job, restart counter is at 6.
Sep 26 11:12:17 RADkit-RHEL systemd[1]: Stopped RADKit Service.

Little confused as to why this is though - should I simply change ownership of radkit-service to root?

Timothy Stewart · ‎09-27-2023

Hello,

Thank you for letting us know about the permission denied error. Who is the owner of the superadmin-password file? Since you have the Radkit user defined in the service file, I would suggest changing the superadmin-password file ownership to the Radkit user which should resolve the permission denied error.

We will revisit / test the Systemd service instructions within the Radkit documentation and will update them to make this is a little easier to implement. Sorry for the troubles you are having.

Thanks
Tim

Stuart-ITGL · ‎09-27-2023

Hi Tim,

Owner of the superadmin-password file is root as described in the advanced service setup pages. Changing the ownership back to the "radkit" user hasn't resolved the permission denied error unfortunately.

I've tested this on the latest Ubuntu Server LTS version and that works absolutely fine. It's only with RHEL9 that I've had this issue.

Please feel free to drop me a DM on Webex if you want to get any extra information to help with this

gosekar · ‎03-21-2024

I had this problem too and found out this denial permission to be happening because of selinux.
Either (1) force selinux to be permissive or disable selinux(/etc/selinux/config) or

[gosekar@gosekar-server ~]$ getenforce 
Enforcing
[gosekar@gosekar-server ~]$ sudo setenforce 0 `
[gosekar@gosekar-server ~]$ sudo systemctl restart radkit; systemctl status radkit
● radkit.service - RADKit Service
     Loaded: loaded (/etc/systemd/system/radkit.service; enabled; preset: disabled)
     Active: active (running) since Thu 2024-03-21 16:46:09 IST; 16ms ago
   Main PID: 3140574 (radkit-service)
      Tasks: 1 (limit: 407615)
     Memory: 2.3M
        CPU: 13ms
     CGroup: /system.slice/radkit.service
             └─3140574 /home/gosekar/.local/radkit/versions/1.6.5/venv/bin/python -I /home/gosekar/.local/bin/radkit-service run --headless

(2) add a local selinux policy module to allow only this executable(log message guides to do this).

[gosekar@gosekar-server ~]$ sudo grep "SELinux is preventing" /var/log/messages
Mar 21 17:06:00 gosekar-server setroubleshoot[3142144]: SELinux is preventing /usr/lib/systemd/systemd from read access on the lnk_file radkit-service. For complete SELinux messages run: sealert -l be1c3a20-a1fc-4a35-a7ca-32dcc2b4c042
Mar 21 17:06:00 gosekar-server setroubleshoot[3142144]: SELinux is preventing /usr/lib/systemd/systemd from read access on the lnk_file radkit-service.#012#012***** Plugin catchall (100. confidence) suggests **************************#012#012If you believe that systemd should be allowed read access on the radkit-service lnk_file by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c '(-service)' --raw | audit2allow -M my-service#012# semodule -X 300 -i my-service.pp#012

[gosekar@gosekar-server ~]$ sealert -l be1c3a20-a1fc-4a35-a7ca-32dcc2b4c042
SELinux is preventing /usr/lib/systemd/systemd from read access on the lnk_file radkit-service.
***** Plugin catchall (100. confidence) suggests **************************
If you believe that systemd should be allowed read access on the radkit-service lnk_file by default. Then you should report this as a bug.
You can generate a local policy module to allow this access. Do allow this access for now by executing:
# ausearch -c '(-service)' --raw | audit2allow -M my-service
# semodule -X 300 -i my-service.pp

[gosekar@gosekar-server ~]$ sudo ausearch -c '(-service)' --raw | audit2allow -M my-service
******************** IMPORTANT ***********************
To make this policy package active, execute:
semodule -i my-service.pp
[gosekar@gosekar-server ~]$ sudo semodule -X 300 -i my-service.pp
[gosekar@gosekar-server ~]$ sudo systemctl status radkit.service 
● radkit.service - RADKit Service
    Loaded: loaded (/etc/systemd/system/radkit.service; enabled; preset: disabled)
    Active: active (running) since Thu 2024-03-21 17:10:32 IST; 2s ago
  Main PID: 3142314 (radkit-service)
     Tasks: 1 (limit: 407615)
    Memory: 9.3M
       CPU: 2.478s
    CGroup: /system.slice/radkit.service
            └─3142314 /home/gosekar/.local/radkit/versions/1.6.5/venv/bin/python -I /home/gosekar/.local/bin/radkit-service run --headless