Systemd service to close LUKS user home broken after F35 upgrade

Hello everyone,

I have a setup to unlock and mount a LUKS encrypted user home folder automatically when the user logs in and then unmount and lock the volume, when the user logs out.

This was working well on F34, but after I upgraded to F35, I could not login with the user anymore.

I used a combination of PAM configurations, scripts and systemd services based on this Arch Linux article.

The problem is related to my cryptsetup-dynamo.service systemd service, which closes the luks volume and was supposed to run after the dynamo user logs out and the home folder is unmounted.
However, on F35 it seems to execute the stop action during the login process and locks the volume shortly after it was opened and the mount process and the user login fails.

My workaround is to disable the cryptsetup-dynamo.service to avoid locking the LUKS device once it is openend, this way everything works fine except that the LUKS device remains
open even after the user logs out, which I wanted to close automatically as on F34.

Does anyone have any ideas what could be the problem on F35?

Thank you in advance for any hints.

Here is my /etc/systemd/system/home-dynamo.mount systemd file for mounting the unlocked home directory for the dynamo user

[Unit]
Requires=user@3000.service
Before=user@3000.service

[Mount]
Where=/home/dynamo
What=/dev/mapper/luks-dynamo
Type=btrfs
Options=subvol=dynamo,noatime,compress=zstd:1

[Install]
RequiredBy=user@3000.service

And here is /etc/systemd/system/cryptsetup-dynamo.service which should take care of closing the LUKS volume after the user logs out and the home folder is unounted.

[Unit]
DefaultDependencies=no
BindsTo=dev-vgsys-dynamo.device
After=dev-vgsys-dynamo.device
BindsTo=dev-mapper-dynamo.device
Requires=home-dynamo.mount
Before=home-dynamo.mount
Conflicts=umount.target
Before=umount.target

[Service]
Type=oneshot
RemainAfterExit=yes
TimeoutSec=0
ExecStop=/usr/sbin/cryptsetup close luks-dynamo

[Install]
RequiredBy=dev-mapper-luks\x2ddynamo.device

In the logs I see that the LUKS device was opened successfully, then I see that mounting the home directory started, but it suddenly fails and unmount runs, which is followed by the execution of the stop command for the cryptsetup-dynamo.service to lock the LUKS device, then it seems there is another try to mount the home directory, but the LUKS device is already closed and it fails. At least, I assume this is the order as the logs only have second resolution so not sure about the order of events as they appear to happen at the same time.

Sep 04 20:00:34 capybara systemd[1]: Condition check resulted in /dev/dm-3 being skipped.
Sep 04 20:00:34 capybara systemd[1]: Condition check resulted in /sys/devices/virtual/block/dm-3 being skipped.
Sep 04 20:00:34 capybara systemd[1]: Unnecessary job was removed for /dev/mapper/dynamo.
Sep 04 20:00:34 capybara systemd[1]: Created slice User Slice of UID 3000.
Sep 04 20:00:34 capybara systemd[1]: Finished cryptsetup-dynamo.service.
Sep 04 20:00:34 capybara systemd[1]: Mounting /home/dynamo...
Sep 04 20:00:34 capybara systemd[1]: Starting User Runtime Directory /run/user/3000...
Sep 04 20:00:34 capybara kernel: BTRFS info (device dm-3): use zstd compression, level 1
Sep 04 20:00:34 capybara kernel: BTRFS info (device dm-3): disk space caching is enabled
Sep 04 20:00:34 capybara kernel: BTRFS info (device dm-3): has skinny extents
Sep 04 20:00:34 capybara kernel: BTRFS error (device dm-3): open_ctree failed
Sep 04 20:00:34 capybara systemd[1]: home-dynamo.mount: Mount process exited, code=killed, status=15/TERM
Sep 04 20:00:34 capybara systemd[1]: home-dynamo.mount: Failed with result 'signal'.
Sep 04 20:00:34 capybara systemd[1]: Unmounted /home/dynamo.
Sep 04 20:00:34 capybara systemd[1]: Stopping cryptsetup-dynamo.service...
Sep 04 20:00:34 capybara systemd-logind[1756]: New session 9 of user dynamo.
Sep 04 20:00:34 capybara systemd[1]: Finished User Runtime Directory /run/user/3000.
Sep 04 20:00:34 capybara systemd-homed[1752]: block device /sys/devices/virtual/block/dm-3 has been removed.
Sep 04 20:00:34 capybara systemd-homed[1752]: block device /sys/devices/virtual/block/dm-3 has been removed.
Sep 04 20:00:34 capybara systemd[1]: cryptsetup-dynamo.service: Deactivated successfully.
Sep 04 20:00:34 capybara systemd[1]: Stopped cryptsetup-dynamo.service.
Sep 04 20:00:34 capybara systemd[1]: Mounting /home/dynamo...
Sep 04 20:00:34 capybara mount[14223]: mount: /home/dynamo: special device /dev/mapper/luks-dynamo does not exist.
Sep 04 20:00:34 capybara systemd[1]: home-dynamo.mount: Mount process exited, code=exited, status=32/n/a
Sep 04 20:00:34 capybara systemd[1]: home-dynamo.mount: Failed with result 'exit-code'.
Sep 04 20:00:34 capybara systemd[1]: Failed to mount /home/dynamo.

This is the part I am suspicious about, there is a BTRFS error and a mount failure and an unmount followed by stopping the cryptesetup-dynamo.service which then locks the LUKS device and does not allow the login process to continue.

For comparison here is a successful login with cryptsetup-dynamo.service disabled

Sep 04 21:09:30 capybara systemd[1]: Created slice User Slice of UID 3000.
Sep 04 21:09:30 capybara systemd[1]: Mounting /home/dynamo...
Sep 04 21:09:30 capybara systemd[1]: Starting User Runtime Directory /run/user/3000...
Sep 04 21:09:30 capybara kernel: BTRFS info (device dm-3): use zstd compression, level 1
Sep 04 21:09:30 capybara kernel: BTRFS info (device dm-3): disk space caching is enabled
Sep 04 21:09:30 capybara kernel: BTRFS info (device dm-3): has skinny extents
Sep 04 21:09:30 capybara systemd-logind[1756]: New session 17 of user dynamo.
Sep 04 21:09:30 capybara systemd[1]: Finished User Runtime Directory /run/user/3000.
Sep 04 21:09:30 capybara kernel: BTRFS info (device dm-3): enabling ssd optimizations
Sep 04 21:09:30 capybara systemd[1]: Mounted /home/dynamo.
Sep 04 21:09:30 capybara systemd[1]: Starting User Manager for UID 3000...

There is no BTRFS error, there is no mount failure and no unmount, everything works as expected.

But I noticed something else as well, see these lines in the failed login:

In the successful login there is no “Unnecessary job was removed” messaged followed by “Finished cryptsetup-dynamo.service”. Not sure if this is a red herring, but my current theory is that somehow between F34 and F35 the behaviour of systemd services changed slightly and for some reason my cryptsetup-dynamo.service is marked “unnecessary” and “removed” which triggers it’s “stop” action that closes LUKS causing the BTRFS error, the mount error and the following unmount and login failure. I have no idea how to prove or disprove this yet…

Finally, I learned how to add precise timestamps to journalctl and added logging to pam_crytpsetup.sh as well to see when is the LUKS device opened

Sep 04 21:36:16.914318487 pam_cryptsetup.sh: cryptsetup open /dev/vgsys/dynamo luks-dynamo
Sep 04 21:36:20.710829 capybara systemd[1]: Condition check resulted in /dev/disk/by-label/dynamo being skipped.
Sep 04 21:36:20.710912 capybara systemd[1]: Condition check resulted in /dev/dm-3 being skipped.
Sep 04 21:36:20.710973 capybara systemd[1]: Condition check resulted in /sys/devices/virtual/block/dm-3 being skipped.
Sep 04 21:36:20.711049 capybara systemd[1]: Unnecessary job was removed for /dev/mapper/dynamo.
Sep 04 21:36:20.714143 capybara systemd[1]: Created slice User Slice of UID 3000.
Sep 04 21:36:20.715787638 pam_cryptsetup.sh: cryptsetup open /dev/vgsys/dynamo luks-dynamo, exit code=0
Sep 04 21:36:20.716491 capybara systemd[1]: Starting cryptsetup-dynamo.service...
Sep 04 21:36:20.718868 capybara systemd[1]: Starting User Runtime Directory /run/user/3000...
Sep 04 21:36:20.721784 capybara cryptsetup-close-dynamo-start.sh[45630]: 21:36:20.721484226 cryptsetup-dynamo.service: started
Sep 04 21:36:20.722587 capybara systemd[1]: Finished cryptsetup-dynamo.service.
Sep 04 21:36:20.726978 capybara systemd[1]: Stopping cryptsetup-dynamo.service...
Sep 04 21:36:20.733449 capybara cryptsetup-close-dynamo.sh[45633]: 21:36:20.731270842 crypsetup-dynamo.service: cryptsetup close luks-dynamo
Sep 04 21:36:20.735560 capybara systemd[1]: Finished User Runtime Directory /run/user/3000.
Sep 04 21:36:20.737856 capybara systemd[1]: Stopping User Runtime Directory /run/user/3000...
Sep 04 21:36:20.744957 capybara systemd-logind[1756]: New session 28 of user dynamo.
Sep 04 21:36:20.750115 capybara systemd[1]: run-user-3000.mount: Deactivated successfully.
Sep 04 21:36:20.751762 capybara systemd[1]: user-runtime-dir@3000.service: Deactivated successfully.
Sep 04 21:36:20.754806 capybara systemd[1]: Starting User Runtime Directory /run/user/3000...
Sep 04 21:36:20.768138 capybara systemd[1]: Finished User Runtime Directory /run/user/3000.
Sep 04 21:36:20.787248 capybara systemd-homed[1752]: block device /sys/devices/virtual/block/dm-3 has been removed.
Sep 04 21:36:20.788571 capybara systemd-homed[1752]: block device /sys/devices/virtual/block/dm-3 has been removed.
Sep 04 21:36:20.856171 capybara cryptsetup-close-dynamo.sh[45633]: 21:36:20.855938833 crypsetup-dynamo.service: cryptsetup close luks-dynamo, exit code=0
Sep 04 21:36:20.856669 capybara systemd[1]: cryptsetup-dynamo.service: Deactivated successfully.
Sep 04 21:36:20.856998 capybara systemd[1]: Stopped cryptsetup-dynamo.service.
Sep 04 21:36:20.859196 capybara systemd[1]: Mounting /home/dynamo...
Sep 04 21:36:20.862728 capybara mount[45661]: mount: /home/dynamo: special device /dev/mapper/luks-dynamo does not exist.
Sep 04 21:36:20.863312 capybara systemd[1]: home-dynamo.mount: Mount process exited, code=exited, status=32/n/a
Sep 04 21:36:20.863323 capybara systemd[1]: home-dynamo.mount: Failed with result 'exit-code'.
Sep 04 21:36:20.863665 capybara systemd[1]: Failed to mount /home/dynamo.
Sep 04 21:36:20.863761 capybara systemd[1]: Dependency failed for Session 28 of User dynamo.
Sep 04 21:36:20.863800 capybara systemd[1]: session-28.scope: Job session-28.scope/start failed with result 'dependency'.
Sep 04 21:36:20.863813 capybara systemd[1]: Dependency failed for User Manager for UID 3000.
Sep 04 21:36:20.863873 capybara systemd[1]: user@3000.service: Job user@3000.service/start failed with result 'dependency'.

The problematic part is here

Sep 04 21:36:20.722587 capybara systemd[1]: Finished cryptsetup-dynamo.service.
Sep 04 21:36:20.726978 capybara systemd[1]: Stopping cryptsetup-dynamo.service...

I can’t explain why is “Stopping” running so early there… Note that this was working as expected on F34 and only started to fail like this after upgrading to F35.

Any help is appreciated. I can provide more info or details if needed.

Are you able to mount the filesystem outside of the service?

Does running the service manually after login work?

1 Like

Hi Elliott, as I mentioned above, I am able to mount it automatically via PAM on user login, the problem is that the service I use to re-lock the encrypted volume when the user logs out is executed right after the volume is opened and thus breaking login. On F34 the “re-lock” service run as expected after the user logged out and the FS was unmounted. On F35 the “re-lock” service is triggered unexpectedly during the login.

When I disable this “encrypt after logout” service, the opening and mounting of the volume through PAM and systemd works well as there is no process to close the encrypted volume unexpectedly. However, this leaves me with an open volume also after I log out, which is the undesired part of the current situation on F35.