Systemd unit mount dependency

Ask Fedora
,
1

Ever since upgrading to F35 it appears that the systemd unit DAG has changed startup order. I have a stratisd fs mount declared in my /etc/fstab which mounts the source required for another systemd user service. After upgrading to F35 I absolutely can NOT get the service to start on startup AFTER the file system mounts. I’ve tried setting systemd directives for After, Requires, and RequiresMountsFor in the service file. But none of these seems to affect the systemd DAG. Can anyone help diagnose what’s going on here?

pb.service is the user service I’m trying to start. On system reboot this is the status (notice the log timestamps are before the mnt-nas.mount timestamp startup times below):

$ systemctl --user status pb.service
○ pb.service - Telegram Bots
     Loaded: loaded (/home/erick/.config/systemd/user/pb.service; enabled; vendor preset: disabled)
     Active: inactive (dead) since Mon 2021-12-20 19:21:54 CET; 26s ago
    Process: 2392 ExecStart=/mnt/nas/workspace/src/chatbots/pb.py -P prod (code=exited, status=203/EXEC)
   Main PID: 2392 (code=exited, status=203/EXEC)
        CPU: 1ms

Dec 20 19:21:54 jupiter systemd[2308]: Started Telegram Bots.
Dec 20 19:21:54 jupiter systemd[2392]: pb.service: Failed to locate executable /mnt/nas/workspace/src/chatbots/pb.py: No such file or directory
Dec 20 19:21:54 jupiter systemd[2392]: pb.service: Failed at step EXEC spawning /mnt/nas/workspace/src/chatbots/pb.py: No such file or directory
Dec 20 19:21:54 jupiter systemd[2308]: pb.service: Main process exited, code=exited, status=203/EXEC
Dec 20 19:21:54 jupiter systemd[2308]: pb.service: Failed with result 'exit-code'.

Systemd unit file for pb.service:

$ cat /home/erick/.config/systemd/user/pb.service
[Unit]
Description=Telegram Bots
After=network.target mnt-nas.mount
Requires=mnt-nas.mount
RequiresMountsFor=/mnt/nas

[Service]
WorkingDirectory=/mnt/nas/workspace/src/chatbots
ExecStart=/mnt/nas/workspace/src/chatbots/pb.py -P prod

[Install]
WantedBy=default.target

Systemd dependencies for pb.service:

$ systemctl list-dependencies --user pb.service
pb.service
● ├─app.slice
● ├─mnt-nas.mount
● └─basic.target
●   ├─paths.target
●   ├─sockets.target
●   │ ├─dbus.socket
●   │ ├─pipewire-pulse.socket
●   │ ├─pipewire.socket
●   │ └─podman.socket
●   └─timers.target
●     └─grub-boot-success.timer

Systemd reverse dependencies for pb.service:

19:22:41 [erick@jupiter:~] 
$ systemctl list-dependencies --user pb.service --reverse
pb.service
● └─default.target

Systemd status for the /mnt/nas mount point:

$ systemctl status mnt-nas.mount
● mnt-nas.mount - /mnt/nas
     Loaded: loaded (/etc/fstab; generated)
     Active: active (mounted) since Mon 2021-12-20 19:22:00 CET; 3min 55s ago
      Where: /mnt/nas
       What: /dev/mapper/stratis-1-7e18ddcd99244c92b926100a7498630b-thin-fs-779e3ab97b244bc496283ae2f21e8cad
       Docs: man:fstab(5)
             man:systemd-fstab-generator(8)
      Tasks: 0 (limit: 154339)
     Memory: 20.0K
        CPU: 15ms
     CGroup: /system.slice/mnt-nas.mount

Dec 20 19:21:58 jupiter systemd[1]: Mounting /mnt/nas...
Dec 20 19:22:00 jupiter systemd[1]: Mounted /mnt/nas.

Systemd version & Fedora kernel info:

$ uname -a
Linux jupiter 5.15.6-200.fc35.x86_64 #1 SMP Wed Dec 1 13:41:10 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
$ systemctl --version
systemd 249 (v249.7-2.fc35)
+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified
2

How if change to Wants=? It’s more weaker and your service will still startup even though the services specified with After= fail (but if fstab failed to mount the disk, your system will error and can’t start up anyway).

Update:

You could also add Restart=. Here the documentation.

3

I didn’t think you could mix systemd system and user units that way. (I could be wrong??)
Maybe just do something simple like checking to see if the file exists and restarting …

[Unit]
Description=Telegram Bots

[Service]
WorkingDirectory=/mnt/nas/workspace/src/chatbots
ExecStartPre=/usr/bin/test -f /mnt/nas/workspace/src/chatbots/pb.py
ExecStart=/mnt/nas/workspace/src/chatbots/pb.py -P prod

RestartSec=10
Restart=on-failure

You may want to play with the timing/number of restarts allowed.

you could also convert it to a system unit running as a USER.

2 Likes