Help with gracefully importing and exporting multipath iscsi backed zfs pool during boot/shutdown

Ask Fedora
,
21 
[Unit]
Description=Import and load-keys and mount storage pool
Requires=multipathd.service
After=network-online.target
After=multipathd.service
After=iscsid.service
Before=nfs-server.service

[Service]
Type=oneshot
RemainAfterExit=yes
RestartSec=1
ExecStartPre=/usr/bin/test -e /dev/disk/by-id/wwn-0x60014054244e8dce0484ffc98805292a
#ExecCondition=/usr/bin/test -e /dev/disk/by-id/wwn-0x60014054244e8dce0484ffc98805292a
ExecStart=/usr/sbin/zpool import storage
ExecStart=/usr/sbin/zfs load-key storage
ExecStart=/usr/sbin/zfs mount -a
ExecStop=/usr/sbin/zpool export storage
ExecStop=/usr/bin/sleep 15

[Install]
WantedBy=network-online.target



# systemctl status zfs-storage-pool-import-loadkey-mount.service 
× zfs-storage-pool-import-loadkey-mount.service - Import and load-keys and mount storage pool
     Loaded: loaded (/etc/systemd/system/zfs-storage-pool-import-loadkey-mount.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Fri 2025-09-19 00:11:03 EEST; 1min 5s ago
 Invocation: 044113c8116e42ff90bc87947bf8c56b
    Process: 1915 ExecStartPre=/usr/bin/test -e /dev/disk/by-id/wwn-0x60014054244e8dce0484ffc98805292a (code=exited, status=1/FAILURE)
   Mem peak: 1.3M
        CPU: 5ms

Sep 19 00:11:03 k8p-clone systemd[1]: Starting zfs-storage-pool-import-loadkey-mount.service - Import and load-keys and mount storage pool...
Sep 19 00:11:03 k8p-clone systemd[1]: zfs-storage-pool-import-loadkey-mount.service: Control process exited, code=exited, status=1/FAILURE
Sep 19 00:11:03 k8p-clone systemd[1]: zfs-storage-pool-import-loadkey-mount.service: Failed with result 'exit-code'.
Sep 19 00:11:03 k8p-clone systemd[1]: Failed to start zfs-storage-pool-import-loadkey-mount.service - Import and load-keys and mount storage pool.

Edit: running it manually works, just like it worked with ExecCondition=/usr/bin/test -e

22

You are missing Restart=on-failure.

23

That’s because by the time you run it manually, the ISCSI dev node is present. The problem is that it isn’t retrying (because you were missing that Restart=on-failure setting).

24

success! after reboot:

root@k8p-clone:~# zpool list
NAME      SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
storage  9.50G  1.15M  9.50G        -         -     1%     0%  1.00x    ONLINE  -
zroot    97.5G  6.92G  90.6G        -         -    18%     7%  1.00x    ONLINE  -
root@k8p-clone:~# systemctl status zfs-storage-pool-import-loadkey-mount.service 
● zfs-storage-pool-import-loadkey-mount.service - Import and load-keys and mount storage pool
     Loaded: loaded (/etc/systemd/system/zfs-storage-pool-import-loadkey-mount.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: active (exited) since Fri 2025-09-19 00:16:39 EEST; 15s ago
 Invocation: f44f091fbe5644eb8e763a86373124bc
    Process: 2032 ExecStartPre=/usr/bin/test -e /dev/disk/by-id/wwn-0x60014054244e8dce0484ffc98805292a (code=exited, status=0/SUCCESS)
    Process: 2037 ExecStart=/usr/sbin/zpool import storage (code=exited, status=0/SUCCESS)
    Process: 2234 ExecStart=/usr/sbin/zfs load-key storage (code=exited, status=0/SUCCESS)
    Process: 2273 ExecStart=/usr/sbin/zfs mount -a (code=exited, status=0/SUCCESS)
   Main PID: 2273 (code=exited, status=0/SUCCESS)
   Mem peak: 6.6M
        CPU: 102ms

Sep 19 00:16:38 k8p-clone systemd[1]: zfs-storage-pool-import-loadkey-mount.service: Scheduled restart job, restart counter is at 1.
Sep 19 00:16:38 k8p-clone systemd[1]: Starting zfs-storage-pool-import-loadkey-mount.service - Import and load-keys and mount storage pool...
Sep 19 00:16:39 k8p-clone systemd[1]: Finished zfs-storage-pool-import-loadkey-mount.service - Import and load-keys and mount storage pool.
25

final service looks like:

[Unit]
Description=Import and load-keys and mount storage pool
Requires=multipathd.service
After=network-online.target
After=multipathd.service
After=iscsid.service
Before=nfs-server.service

[Service]
Type=oneshot
RemainAfterExit=yes
RestartSec=1
TimeoutStartSec=120
Restart=on-failure
ExecStartPre=/usr/bin/udevadm settle -E /dev/disk/by-id/dm-name-storage
ExecStart=/usr/sbin/zpool import -o cachefile=/etc/zfs/storage.cache storage
ExecStart=-/usr/sbin/zfs load-key storage
ExecStart=-/usr/sbin/zfs mount -a
ExecStop=/usr/sbin/zpool export storage
ExecStop=/usr/bin/sleep 15

[Install]
WantedBy=multi-user.target

edit: also must do:

zpool set cachefile=/etc/zfs/storage.cache storage

edit2: changed to ExecStartPre=-/usr/bin/udevadm settle -E /dev/disk/by-id/dm-name-storage to test that the multipath dev node(s) exist. This seems to be the best way to do it for now

But again, this has to be a bug right? something the fedora team should take a look at?
edit: ignore the above, not a bug

26

I’m not sure what you mean? In general, any ZFS-specific problem isn’t going to get much attention by the Fedora devs.

I don’t see a bug. The OS isn’t expected to know what the dependencies are for your remote storage or how they need to be ordered. The error code for that sort of thing is “Contact your system administrator.” :slightly_smiling_face:

27

well i mean iscsid and multipathd are related to storage, so auto-importing zfs pools by cache or by scan should happen after the iscsi multipath is up and running.. but i get your point

anyways..

I can’t tell you how thankful i am for your help and the time and effort you gave, i would have NEVER got this working by myself, except for manually doing this after each reboot….

28

Yes, you should do that.

29

oh.. i might have copypasted an old line by mistake.. i had that there

30

Maybe that could be filed against the OpenZFS project as an enhancement request. Not all ZFS pools are on iSCSI though, so the import services would need to detect that condition somehow or else there would need to be separate pre-made services for that use case that people could manually choose.

31

I’ll edit the success! post and correct the multi-user.target line, and mark it as a solution.

Hopefully this thread will help others with the same issue.

THANK YOU @glb lol..

edit: just a little addition, this test system has a single test disk exported, but on my actual system it will be two disks in zpool mirror.. so i will have to have both wwn disks there, or try using some other file from /dev/mapper or something.. but that’s for another day

32

some addition here.. it might not be a good idea to use the wwn-* file for a pool with multiple vdev’s. multipathd creates dm-1 for this single disk pool, but there are many options:

# ls -la /dev/mapper/ | grep dm-1
lrwxrwxrwx.  1 root root       7 Sep 19 01:13 storage -> ../dm-1


# ls -la /dev/disk/by-id/ | grep dm-1
lrwxrwxrwx. 1 root root  10 Sep 19 01:13 dm-name-storage -> ../../dm-1
lrwxrwxrwx. 1 root root  10 Sep 19 01:13 dm-uuid-mpath-360014054244e8dce0484ffc98805292a -> ../../dm-1
lrwxrwxrwx. 1 root root  10 Sep 19 01:13 scsi-360014054244e8dce0484ffc98805292a -> ../../dm-1
lrwxrwxrwx. 1 root root  10 Sep 19 01:13 wwn-0x60014054244e8dce0484ffc98805292a -> ../../dm-1

i just tried importing the pool using zpool import -d /dev/disk/by-id/dm-uuid-mpath-360014054244e8dce0484ffc98805292a storage and that seems to be the more appropriate file to use with multiple vdev’s

also i know from experience, if i was to delete the targetcli export on the server and recreate it, it would get a different wwn. So maybe using the multipathd alias ‘storage’ would be the best option..

now actually using /dev/disk/by-id/dm-name-storage in the systemd service.. but they all seem to work

33

There is another caveat that you might want to consider.

From man systemd.service:

Note that if any of the commands specified in ExecStartPre=, ExecStart=, or ExecStartPost= fail (and are not prefixed with “-”, see above) or time out before the service is fully up, execution continues with commands specified in ExecStopPost=, the commands in ExecStop= are skipped.

With your current execution stack:

ExecStart=/usr/sbin/zpool import storage
ExecStart=/usr/sbin/zfs load-key storage
ExecStart=/usr/sbin/zfs mount -a
ExecStop=/usr/sbin/zpool export storage

The zpool export ... command will not be run on shutdown if either of the zfs load-key ... or zfs mount ... commands fail, even though the pool may have been successfully imported.

It might be better to change that to:

ExecStart=/usr/sbin/zpool import storage
ExecStart=-/usr/sbin/zfs load-key storage
ExecStart=-/usr/sbin/zfs mount -a
ExecStop=/usr/sbin/zpool export storage

So that if zpool import ... succeeds, a corresponding zpool export ... will always be run.

Better yet would be to make a whole separate service for the zfs load-key ... and zfs mount ... commands and order that service after your zfs-storage-pool-import-loadkey-mount service.

If you don’t do one of the above, you might hit a situation where your system hangs on shutdown again because your zfs pool cannot export after the underlying storage has disappeared.

Just FYI. :slightly_smiling_face:

34

thanks. I been looking at the zfs-mount-generator manpages, and it describes things like Wants=, After=: network-online.target etc.. but all this is way above me, i just need it to work.. i’ll test adding the - character..

Right place would maybe be openzfs github issues page for all this..

35

One more thing … :slightly_smiling_face:

In your marked solution, you did not set something like -c /etc/zfs/storage.cache on your zpool import ... line. You should do that as was suggested in the documentation (emphasis added):

If you omit that -c ... when importing your pool, the metadata about the pool gets stored in /etc/zfs/zpool.cache by default. That zpool.cache file with your storage metadata will (eventually) end up in your initramfs (when you happen to generate one while your storage pool is imported). On your next reboot, the zfs-import-cache service will run during early boot, see that cache file in the initramfs, and try to mount your storage pool. This will fail and probably increase your boot time significantly (if not hang it completely).

36

i do after reboot when storage pool is imported:

root@k8p-clone:~# zpool set cachefile=/etc/zfs/storage.cache storage
root@k8p-clone:~# zpool export storage
root@k8p-clone:~# zpool import -c /etc/zfs/storage.cache storage
invalid or corrupt cache file contents: invalid or missing cache file
cannot import 'storage': no such pool available


root@k8p-clone:~# ls -la /etc/zfs/*.cache
-rw-r--r--. 1 root root 0 Sep 19 10:47 /etc/zfs/storage.cache
-rw-r--r--. 1 root root 0 Sep 19 10:47 /etc/zfs/zpool.cache

both files are empty..

37

thing is, i was getting ready to post in ‘“our other thread” just before i saw your update above. Because, after getting all this to work, kernel-update resulted in a unbootable system.

I was doing snapshots and rollbacks to figure out why..

system update and zfs-update worked fine, but after kernel-update, system is unbootable. All i have is a screenshot. The zfs-import-cache error is there, but it was there before as well.. Now i’m starting to think these issues are connected.

So again, with this storage pool import service working like earlier, doing a kernel-update results in:

k8p-clone_failed_kernel_update_19_09_2025
k8p-clone_failed_kernel_update_19_09_20251920×1451 472 KB

So far i’ve tried rolling back to before kernel-update and:

1: delete the /etc/dkms/framework.conf.d/override.conf file → didn’t help

2: manually running dracut --regenerate-all --force after kernel update → didn’t help

3: completely deleting your dnf wrapper scripts, and upgrading system incl. kernel → didn’t help

So these things are now probably linked somehow.. maybe?

38

This is what i’m doing now:

export storage pool

set iscsi node login to manual

disable zfs-storage-pool-import-loadkey-mount.service

regenerate zroot pool cache, resulting in zpool.cache now having data:

root@k8p-clone:~# zpool set cachefile=/etc/zfs/zpool.cache zroot
root@k8p-clone:~# ls -la /etc/zfs/*.cache
-rw-r--r--. 1 root root    0 Sep 19 10:47 /etc/zfs/storage.cache
-rw-r--r--. 1 root root 1520 Sep 19 11:16 /etc/zfs/zpool.cache

update initramfs:

# dracut --regenerate-all --force
dracut-install: Failed to find module 'zfs'
dracut[E]: FAILED:  /usr/lib/dracut/dracut-install -D /var/tmp/dracut.dXkNRHT/initramfs -H --kerneldir /usr/lib/modules/6.14.4-300.fc42.x86_64 -m zfs
dracut[F]: installkernel failed in module zfs
dracut-install: Failed to find module 'zfs'
dracut[E]: FAILED:  /usr/lib/dracut/dracut-install -D /var/tmp/dracut.dtgxvNP/initramfs -H --kerneldir /usr/lib/modules/6.15.5-200.fc42.x86_64 -m zfs
dracut[F]: installkernel failed in module zfs

these errors are because i have post_transaction=“” in /etc/dkms/framework.conf.d/override.conf right? It’s not happening to my current running kernel:

# uname -r
6.15.7-200.fc42.x86_64

reboot

update-kernel

# kernel-update 

Kernel update kernel-6.16.7-200.fc42.x86_64 is available.
Your current version of OpenZFS is zfs-2.3.4-1.fc42.x86_64.

wth.. i upgraded the kernel to kernel-6.16.7-200, and the system boots, but to 6.15.7-200.fc42.x86_64

6.16.7-200 is there in /boot, but is not being picked up by zfsbootmanager..

oh.. no initramfs for 6.16.7

it fails to generate initramfs for 2 kernels:

# dracut --regenerate-all --force
dracut-install: Failed to find module 'zfs'
dracut[E]: FAILED:  /usr/lib/dracut/dracut-install -D /var/tmp/dracut.dMtmxdR/initramfs -H --kerneldir /usr/lib/modules/6.15.5-200.fc42.x86_64 -m zfs
dracut[F]: installkernel failed in module zfs
dracut-install: Failed to find module 'zfs'
dracut[E]: FAILED:  /usr/lib/dracut/dracut-install -D /var/tmp/dracut.di0cpo7/initramfs -H --kerneldir /usr/lib/modules/6.16.7-200.fc42.x86_64 -m zfs
dracut[F]: installkernel failed in module zfs

i dunno anymore.. will have to rollback to some point and try again. I can rollback to before i ever set up the iscsi link, but it’s pretty much work to redo.. to be continued, gotta go afk now

39

They are probably because you are using --regenerate-all. You shouldn’t use that flag. It has the potential to break otherwise-working older initramfs images.

My suggestion is to never use --regenerate-all (I never use it). Also, it is a good practice to run dkms status before running dracut -f and make sure that a zfs module has been successfully built for the latest kernel. If the zfs module hasn’t successfully built, there are dkms commands to can run to get it to retry. I update my zfs packages and kernel packages separately (using those zfs-update and kernel-update scripts).

It takes a little while to figure out all the quirks around running a Fedora on ZFS system. :slightly_smiling_face:

40

That was just because i was trying things on an unbootable system.. normally i keep the post_transaction=“” in /etc/dkms/framework.conf.d/override.conf per your recommendation.

The issue remains this:

again, after kernel update system is unbootable, and i need to find out why..

after reboot, on working system after zfs-update and before kernel-update, we look like this:

root@k8p-clone:~# zpool list
NAME      SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
storage  9.50G  1.14M  9.50G        -         -     1%     0%  1.00x    ONLINE  -
zroot    97.5G  7.01G  90.5G        -         -    19%     7%  1.00x    ONLINE  -
root@k8p-clone:~# ls -la /etc/zfs/*.cache
-rw-r--r--. 1 root root 1624 Sep 19 17:05 /etc/zfs/zpool.cache

So at this point, if i do a kernel update, the storage pool info is in /etc/zfs/zpool.cache along with zroot because storage pool has no separate storage.cache file.

/etc/zfs/zpool.cache gets then included in initramfs at kernel update, and the system becomes unbootable, like in the screenshot above.

but if i try to generate a /etc/zfs/storage.cache, well, see above..