Systemd timer located in /usr/local fails to start on start-up

Project Discussion
1

I use a systemd timer to start an associated service; simply a rsync backup of certain (e.g. /home) directories. A copy of the timer content is below, and the file is located in the /usr/local/lib/systemd/system/ directory. FYI, the rsync systemd unit and script are both tested and work well.

# cat /usr/local/lib/systemd/system/rsync-backup.timer
[Unit]
Description=Run backup via rsync script each morning.

[Timer]
OnCalendar=*-*-* 3:00:00

[Install]
WantedBy=timers.target

My problem is that the above timer shows the following status after each boot, the key part being inactive (dead).

● rsync-backup.timer - Run backup via rsync script each morning.
     Loaded: loaded (/usr/local/lib/systemd/system/rsync-backup.timer; enabled; vendor preset: disabled)
     Active: inactive (dead)
    Trigger: n/a
   Triggers: ● rsync-backup.service

I can simply solve this by entering the following command and everything comes up as expected. So long as I do not restart before the next backup, the backup runs at the expected time and goes smoothly.

# systemctl enable --now rsync-backup.timer
# systemctl status rsync-backup.timer
● rsync-backup.timer - Run backup via rsync script each morning.
     Loaded: loaded (/usr/local/lib/systemd/system/rsync-backup.timer; enabled; vendor preset: disabled)
     Active: active (waiting) since Sat 2021-01-30 22:19:15 EST; 3s ago
    Trigger: Sun 2021-01-31 03:00:00 EST; 4h 40min left
   Triggers: ● rsync-backup.service

Not wanting to have to complete the above manual steps each boot, I have checked/tried the following:

  • confirmed the SELinux contexts are correct.
  • confirmed the permissions are correct.
  • attempted adding a systemd.preset file (changes vendor preset: from disabled to enabled however, does not solve the original problem).
  • confirmed that systemd checks the correct path via systemd-analyze unit-paths
  • confirmed that btrfs subvolumes are mounted prior to timer.targets is run (by using systemd-analyze plot).

Some additional details that I think might have an affect:

  • /usr/local (i.e. /var/usrlocal) is a separate subvolume (i.e. usrlocal) on a single btrfs volume (fedora).`

I am not sure if I am doing some incorrectly or if this is a quirk of a Silverblue installtion (in combination with some of the quirks of my setup). Would greatly appreciate any assistance and happy to take advice or corrections and answer any questions.

1 Like
2

Does it work if you put the unit file in /etc/systemd/system?

3

Hi @vgaetera,

I hadn’t tried it so great suggestion! Moved the systemd.timer only into the /etc/systemd/system folder, re-booted, and checked the status of both the timer and service unit files once booted.

$ systemctl status rsync-backup.timer 
● rsync-backup.timer - Run backup via rsync script each morning.
     Loaded: loaded (/etc/systemd/system/rsync-backup.timer; enabled; vendor preset: disabled)
     Active: inactive (dead)
    Trigger: n/a
   Triggers: ● rsync-backup.service

Jan 31 20:11:30 host systemd[1]: rsync-backup.timer: Refusing to start, unit rsync-backup.service to trigger not loaded.
Jan 31 20:11:30 host systemd[1]: Failed to start Run backup via rsync script each morning..

$ systemctl status rsync-backup.service
● rsync-backup.service - Backup via rsync to remote server.
     Loaded: loaded (/usr/local/lib/systemd/system/rsync-backup.service; enabled; vendor preset: disabled)
     Active: inactive (dead)
TriggeredBy: ● rsync-backup.timer

It returns an error that the service file (i.e. rsync-backup.service) is not loaded however, the service file status indicates that it is. To me, this indicates a timing error but eager to hear other conclusions/thoughts.

1 Like
4

Try to move both the timer and service units to /etc/systemd/system.
In addition run this:

sudo systemctl daemon-reload

Yep, it looks like a race condition, although this might be a Silverblue-specific issue.

1 Like
5

Yep, I can do this and both the timer and service work flawlessly … but I am attempting to keep any local configurations to /usr/local so that I can back them up. This avoids having to back up the entire /etc directory.

To further my experimentation, I decided to try testing timer and service files in a Silverblue VM. The only difference is that I did not create a separate partition for /usr/local (i.e. /var/usrlocal). This resulted in the same end-result; the timer file was loaded but inactive on reboot. This presumably eliminates the possibility that using a brfs subvolume for /var/usrlocal creates an issue.

I took this one step further and did the same but on a Fedora 33 VM (i.e. not SIlverblue), with the same timer and service located in /usr/local. Success! The timer loaded and was active on boot and the service ran correctly. In other words, I did not need to take the manual step of using systemctl start.

Therefore my conclusion is that this is an issue specifically related to Silverblue that falls out of the expected behavior of a non ostree-based distribution. This is based on man systemd.unit documentation relating to local configuration and above experimentation. I also hazard a guess it is likely related to the use of /var/usrlocal, as the path falls outside of what is stated within systemd documentation.

Question is, does this constitute a bug and (if so) where would be the best place to file one?

A possible work-around is to add another unit path to systemd configuration however, I have not attempted this and unsure what is possible.

1 Like
6

It appears to be here: Silverblue issue/bug reporting location - #5 by sanja
However, since the issue is supposed to be reported against systemd, it’s best to file a ticket on the systemd issue tracker as well and cross-link the reports.

1 Like
7

Thanks @vgaetera for you help and assistance! I raised an issue as suggested.

1 Like