Cannot start systemd service from NetworkManager's dispatcher script

Ask Fedora
,
1

I have a NetworkManager dispatcher script, which tries to start/stop systemd services. However, starting/stopping the service does not work from the dispatcher, while it works just fine (same command line) from the command line executed as root. Can anyone help me what’s going on?

The script `/etc/NetworkManager/dispatcher.d/toggle-syncthing-service.sh` 
#!/bin/bash

set -eou pipefail

EXPECTED_INTERFACE="wlo1"
EXPECTED_INTERFACE_NAME="[censored]"
EXPECTED_INTERFACE_SSID="[censored]"
USERNAME="christian"

INTERFACE=$1
NM_ACTION=$2

if [[ -z ${INTERFACE:=} ]]
then
	echo "No interface specified. Exiting."
fi

if [[ "${INTERFACE}" != "$EXPECTED_INTERFACE" ]]
then
	echo "Found connection interface ${INTERFACE:=[undefined]} instead of $EXPECTED_INTERFACE, exiting."
	exit
fi

if [[ "${CONNECTION_ID:=}" != "$EXPECTED_INTERFACE_NAME" ]]
then
	echo "Found connection named ${CONNECTION_ID:=} instead of $EXPECTED_INTERFACE_NAME, exiting."
	exit
fi

if [[ "${CONNECTION_UUID:=}" != "$EXPECTED_INTERFACE_SSID" ]]
then
	echo "Found connection SSID ${CONNECTION_UUID:=} instead of $EXPECTED_INTERFACE_SSID, exiting."
	exit
fi

case "$NM_ACTION" in
	up)
	echo "Starting syncthing as $USERNAME"
	systemctl --user -M $USERNAME@ start syncthing.service
	;;
	down)
	echo "Stopping syncthing…"
	systemctl --user -M $USERNAME@ stop syncthing.service
	;;
esac

Good case

When executing the commands directly as root,

systemctl --user -M christian@ start syncthing.service
systemctl --user -M christian@ start syncthing.service

they work as expected. No interesting logging.

Bad case

When executing the commands from NetworkManager dispatcher, the service does not start/stop. Instead, I get the following errors in syslog:
When starting:

nm-dispatcher[18473]: Starting syncthing as christian
nm-dispatcher[18475]: Failed to start transient service unit: Connection reset by peer
nm-dispatcher[18474]: Failed to start syncthing.service: Transport endpoint is not connected
nm-dispatcher[18474]: See user logs and 'systemctl --user status syncthing.service' for details.
nm-dispatcher[18441]: req:4 'up' [wlo1], "/etc/NetworkManager/dispatcher.d/toggle-syncthing-service.sh": complete: process failed with Script '/etc/NetworkManager/dispatcher.d/toggle-syncthing-service.sh' exited with status 1
NetworkManager[1003]: <warn>  [1792178417.4367] dispatcher: (43) /etc/NetworkManager/dispatcher.d/toggle-syncthing-service.sh failed (failed): Script '/etc/NetworkManager/dispatcher.d/toggle-syncthing-service.sh' exited with status 1

When stopping:

nm-dispatcher[18349]: Stopping syncthing…
nm-dispatcher[18351]: Failed to start transient service unit: Connection reset by peer
nm-dispatcher[18350]: Failed to stop syncthing.service: Transport endpoint is not connected
nm-dispatcher[18350]: See user logs and 'systemctl --user status syncthing.service' for details.
nm-dispatcher[18331]: req:2 'down' [wlo1], "/etc/NetworkManager/dispatcher.d/toggle-syncthing-service.sh": complete: process failed with Script '/etc/NetworkManager/dispatcher.d/toggle-syncthing-service.sh' exited with status 1
NetworkManager[1003]: <warn>  [1792178404.7480] dispatcher: (38) /etc/NetworkManager/dispatcher.d/toggle-syncthing-service.sh failed (failed): Script '/etc/NetworkManager/dispatcher.d/toggle-syncthing-service.sh' exited with status 1

In systemctl --user status syncthing.service, there is nothing. There is also nothing else in the syslogs, neither the system syslogs (as root) nor the user syslogs.

Is there something special about the NetworkManager dispatcher execution environment?
Is this a bug in systemd or NetworkManager?

Additional information

I extracted straces, but I cannot upload them here due to restrictions. Any suggested location where to upload them?

2

I’m guessing wildly…

Is there a user session running at the point you try to start syncthing in the script?
I can list the sessions with loginctl.

Maybe you need to enable linger on your user with loginctl?

3

Yes, there is a user session running. I thought I would only need to enable lingering in case I have problems with it while no session is running? But I didn’t think of it, so it might save me some future headache.

Anyways, enabling lingering doesn’t make a difference: Same problem with and without lingering.

4

(post deleted by author)

5

I think you need to start it as a system service like

systemctl start syncthing@christian.service

Or you could just enable the service

systemctl enable syncthing@christian.service
6

There were no denials, so I guess this is not it?

7

This makes syncthing start, as user christian. However, it seems like there is a difference between:

  1. Running systemctl --user start syncthing.service as christian
  2. Running systemctl start syncthing@christian.service as root

When running the first one, systemctl --user status syncthing.service as christian shows that it is running. When running the second one, the same command shows that it is not. And I’m quite confused about the difference of these two. Why is that? What’s the difference between the two?

8

The difference is that when running systemctl --user ... you need to talk to the systemd instance that is part of your user session. The NetworkManager doesn’t know how to do that.

To show the status you run systemctl status syncthing@christian.service.

You can check the running systemd instances

ps $(pidof systemd)

which could show something like

    PID TTY      STAT   TIME COMMAND
      1 ?        Ss     0:01 /usr/lib/systemd/systemd --switched-root --system --deserialize=59 rhgb testing
   1311 ?        Ss     0:00 /usr/lib/systemd/systemd --user

where process 1 handles system services and process 1311 handles user services which are started with systemctl --user .... Either instance doesn’t know about services started by the other.

9

I tested this for you and it works in permissive SELinux mode.
There are multiple denials logged with dontaudit off, see:
semanage-dontaudit: SELinux Policy Management dontaudit tool | Man Page | System Administration | policycoreutils-python-utils | ManKier

10

The script uses the -M user@ to talk to the systemd for the user.
NetworkManager is not involved.