Help with gracefully importing and exporting multipath iscsi backed zfs pool during boot/shutdown

Ask Fedora
,
41

I’ve got to run in a few minutes and I’ll be away for a few hours. At a very quick glance, I don’t see the error message (other than that one about the zfs module not being built, but that wouldn’t have anything to do with cachefile being set or not).

FWIW, here is a quick dump of some of the scripts from one of my working systems.

# zpool get cachefile
NAME  PROPERTY   VALUE                SOURCE
bak0  cachefile  -                    default
disk  cachefile  /etc/zfs/disk.cache  local
root  cachefile  none                 local
zfs3  cachefile  /etc/zfs/zfs3.cache  local

# systemctl cat filesystem-disk.service
# /etc/systemd/system/filesystem-disk.service
[Unit]
Description=filesystem %j
Before=filesystems.target
Conflicts=shutdown.target

[Service]
Type=oneshot
ExecStart=/usr/sbin/zpool import -d /dev/disk/by-path -o cachefile=/etc/zfs/%j.cache %j
ExecStop=/usr/sbin/zpool export %j
Restart=on-failure
RestartSec=10
RemainAfterExit=yes
TimeoutSec=0

[Install]
WantedBy=filesystems.target

# /usr/lib/systemd/system/service.d/10-timeout-abort.conf
# This file is part of the systemd package.
# See https://fedoraproject.org/wiki/Changes/Shorter_Shutdown_Timer.
#
# To facilitate debugging when a service fails to stop cleanly,
# TimeoutStopFailureMode=abort is set to "crash" services that fail to stop in
# the time allotted. This will cause the service to be terminated with SIGABRT
# and a coredump to be generated.
#
# To undo this configuration change, create a mask file:
#   sudo mkdir -p /etc/systemd/system/service.d
#   sudo ln -sv /dev/null /etc/systemd/system/service.d/10-timeout-abort.conf

[Service]
TimeoutStopFailureMode=abort

# systemctl cat filesystem-zfs3.service
# /etc/systemd/system/filesystem-zfs3.service
[Unit]
Description=filesystem %j
Before=filesystems.target
Conflicts=shutdown.target

[Service]
Type=oneshot
ExecStart=/usr/sbin/zpool import -d /dev/disk/by-path -o cachefile=/etc/zfs/%j.cache %j
ExecStop=/usr/sbin/zpool export %j
Restart=on-failure
RestartSec=10
RemainAfterExit=yes
TimeoutSec=0

[Install]
WantedBy=filesystems.target

# /usr/lib/systemd/system/service.d/10-timeout-abort.conf
# This file is part of the systemd package.
# See https://fedoraproject.org/wiki/Changes/Shorter_Shutdown_Timer.
#
# To facilitate debugging when a service fails to stop cleanly,
# TimeoutStopFailureMode=abort is set to "crash" services that fail to stop in
# the time allotted. This will cause the service to be terminated with SIGABRT
# and a coredump to be generated.
#
# To undo this configuration change, create a mask file:
#   sudo mkdir -p /etc/systemd/system/service.d
#   sudo ln -sv /dev/null /etc/systemd/system/service.d/10-timeout-abort.conf

[Service]
TimeoutStopFailureMode=abort

# ls -al /etc/zfs/*.cache
-rw-r--r--. 1 root root    0 Aug 22 10:26 /etc/zfs/bak0.cache
-rw-r--r--. 1 root root 2352 Aug 22 10:14 /etc/zfs/disk.cache
-rw-r--r--. 1 root root 3780 Aug 12 10:26 /etc/zfs/zfs3.cache
-rw-r--r--. 1 root root 6300 Aug 22 11:09 /etc/zfs/zpool.cache

I’m using -o cachefile=/etc/zfs/%j.cache, but I thought that was equivalent to -c .... I don’t know. I’ll be back to take a better look at what you’ve reported in a few hours.

42

-o cachefile= works with import:
after boot of working system:

root@k8p-clone:~# zpool list
NAME      SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
storage  9.50G  1.02M  9.50G        -         -     1%     0%  1.00x    ONLINE  -
zroot    97.5G  7.01G  90.5G        -         -    19%     7%  1.00x    ONLINE  -
root@k8p-clone:~# zpool get cachefile
NAME     PROPERTY   VALUE      SOURCE
storage  cachefile  -          default
zroot    cachefile  none       local
root@k8p-clone:~# ls -la /etc/zfs/*.cache
-rw-r--r--. 1 root root 1624 Sep 19 17:55 /etc/zfs/zpool.cache
root@k8p-clone:~# zpool set cachefile=/etc/zfs/storage.cache storage
root@k8p-clone:~# zpool get cachefile
NAME     PROPERTY   VALUE                   SOURCE
storage  cachefile  /etc/zfs/storage.cache  local
zroot    cachefile  none                    local
root@k8p-clone:~# ls -la /etc/zfs/*.cache
-rw-r--r--. 1 root root 1624 Sep 19 20:56 /etc/zfs/storage.cache
-rw-r--r--. 1 root root    0 Sep 19 20:56 /etc/zfs/zpool.cache
root@k8p-clone:~# zpool export storage
root@k8p-clone:~# zpool import -o cachefile=/etc/zfs/storage.cache storage
root@k8p-clone:~# zpool get cachefile
NAME     PROPERTY   VALUE                   SOURCE
storage  cachefile  /etc/zfs/storage.cache  local
zroot    cachefile  none                    local
root@k8p-clone:~# ls -la /etc/zfs/*.cache
-rw-r--r--. 1 root root 1624 Sep 19 20:57 /etc/zfs/storage.cache
-rw-r--r--. 1 root root    0 Sep 19 20:57 /etc/zfs/zpool.cache

now i add ExecStart=/usr/sbin/zpool import -o cachefile=/etc/zfs/storage.cache storage
to /etc/systemd/system/zfs-storage-pool-import-loadkey-mount.service

root@k8p-clone:~# systemctl daemon-reload 
root@k8p-clone:~# reboot

root@k8p-clone:~# zpool list
NAME      SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
storage  9.50G  1.15M  9.50G        -         -     1%     0%  1.00x    ONLINE  -
zroot    97.5G  7.01G  90.5G        -         -    19%     7%  1.00x    ONLINE  -
root@k8p-clone:~# zpool get cachefile
NAME     PROPERTY   VALUE                   SOURCE
storage  cachefile  /etc/zfs/storage.cache  local
zroot    cachefile  none                    local
root@k8p-clone:~# ls -la /etc/zfs/*.cache
-rw-r--r--. 1 root root 1624 Sep 19 21:03 /etc/zfs/storage.cache
-rw-r--r--. 1 root root    0 Sep 19 21:03 /etc/zfs/zpool.cache
root@k8p-clone:~# kernel-update

Kernel update kernel-6.16.7-200.fc42.x86_64 is available.
Your current version of OpenZFS is zfs-2.3.4-1.fc42.x86_64.

Before proceeding, check https://github.com/openzfs/zfs/releases to veri-
fy that your version of OpenZFS is compatible with this kernel update.

Update to kernel-6.16.7-200.fc42.x86_64 [y/n]?: y

[...]

>>> Scriptlet output:                                                                                                                                                                      
>>> grub2-probe: error: ../grub-core/kern/fs.c:123:unknown filesystem.                                                                                                                     
>>> Sign command: /lib/modules/6.16.7-200.fc42.x86_64/build/scripts/sign-file                                                                                                              
>>> Signing key: /var/lib/dkms/mok.key                                                                                                                                                     
>>> Public certificate (MOK): /var/lib/dkms/mok.pub                                                                                                                                        
>>>                                                                                                                                                                                        
>>> Autoinstall of module zfs/2.3.4 for kernel 6.16.7-200.fc42.x86_64 (x86_64)                                                                                                             
>>> Running the pre_build script................... done.                                                                                                                                  
>>> Building module(s)................ done.                                                                                                                                               
>>> Signing module /var/lib/dkms/zfs/2.3.4/build/module/zfs.ko                                                                                                                             
>>> Signing module /var/lib/dkms/zfs/2.3.4/build/module/spl.ko                                                                                                                             
>>> Running the post_build script... done.                                                                                                                                                 
>>> Installing /lib/modules/6.16.7-200.fc42.x86_64/extra/zfs.ko.xz                                                                                                                         
>>> Installing /lib/modules/6.16.7-200.fc42.x86_64/extra/spl.ko.xz                                                                                                                         
>>> Running depmod...... done.                                                                                                                                                             
>>>                                                                                                                                                                                        
>>> Autoinstall on 6.16.7-200.fc42.x86_64 succeeded for module(s) zfs.                                                                                                                     
>>> grep: /etc/default/grub: No such file or directory                                                                                                                                     
>>>                                                                                                                                                                                        
Complete!
root@k8p-clone:~# reboot

It worked! having a separate cache for storage turns out to be critical.

43

just to make sure, for my actual system with more than one disk, i’d need to have multiple ExecStartPre test entries, ie something like:

[Unit]
Description=Import and load-keys and mount storage pool
Requires=multipathd.service
After=network-online.target
After=multipathd.service
After=iscsid.service
Before=nfs-server.service

[Service]
Type=oneshot
RemainAfterExit=yes
RestartSec=1
Restart=on-failure
ExecStartPre=/usr/bin/test -e /dev/disk/by-id/dm-name-hgst-10t-disk0
ExecStartPre=/usr/bin/test -e /dev/disk/by-id/dm-name-hgst-10t-disk1
ExecStart=/usr/sbin/zpool import -o cachefile=/etc/zfs/storage.cache storage
ExecStart=-/usr/sbin/zfs load-key storage
ExecStart=-/usr/sbin/zfs mount -a
ExecStop=/usr/sbin/zpool export storage
ExecStop=/usr/bin/sleep 15

[Install]
WantedBy=multi-user.target

In above case, both tests would need to succeed before zpool import is invoked right?

Furthermore, i wish i’d know if checking for presence of the disks should be done in /dev/disk/by-id/, /dev/mapper/*, or what they both actually point to, /dev/dm-[n] ?

root@k8p:~# ls -la /dev/disk/by-id/ | grep -e "dm-1" -e "dm-2"
lrwxrwxrwx. 1 root root  10 Sep 17 17:58 dm-name-hgst-10t-disk0 -> ../../dm-1
lrwxrwxrwx. 1 root root  10 Sep 17 17:58 dm-name-hgst-10t-disk1 -> ../../dm-2
lrwxrwxrwx. 1 root root  10 Sep 17 17:58 dm-uuid-mpath-360014059e827fb602ec4d96929f7458c -> ../../dm-1
lrwxrwxrwx. 1 root root  10 Sep 17 17:58 dm-uuid-mpath-36001405b575fb6aa7d44bc79fa6d870c -> ../../dm-2
lrwxrwxrwx. 1 root root  10 Sep 17 17:58 scsi-360014059e827fb602ec4d96929f7458c -> ../../dm-1
lrwxrwxrwx. 1 root root  10 Sep 17 17:58 scsi-36001405b575fb6aa7d44bc79fa6d870c -> ../../dm-2
lrwxrwxrwx. 1 root root  10 Sep 17 17:58 wwn-0x60014059e827fb602ec4d96929f7458c -> ../../dm-1
lrwxrwxrwx. 1 root root  10 Sep 17 17:58 wwn-0x6001405b575fb6aa7d44bc79fa6d870c -> ../../dm-2

root@k8p:~# ls -la /dev/mapper/ | grep -e "dm-1" -e "dm-2"
lrwxrwxrwx.  1 root root       7 Sep 17 17:58 hgst-10t-disk0 -> ../dm-1
lrwxrwxrwx.  1 root root       7 Sep 17 17:58 hgst-10t-disk1 -> ../dm-2

root@k8p:~# ls -la /dev/ | grep -e "dm-1" -e "dm-2"
brw-rw----.  1 root disk    252,     1 Sep 17 17:58 dm-1
brw-rw----.  1 root disk    252,     2 Sep 17 17:58 dm-2

Do you have any insight on this?

edit: i guess using dm-1 and dm-2 is not a good idea, since that would change if dm-0 would be removed from earlier in the boot…

44

iirc, this is actually 5 seconds?

“ZFS issues infrequent flushes (every 5 second or so) after the uberblock updates.”
https://docs.oracle.com/cd/E26505_01/html/E37386/chapterzfs-6.html

45

It looks like you are right:

$ cat /sys/module/zfs/parameters/zfs_txg_timeout 
5

I’m not sure if that is the right number to go by. I just know from experience that your system can hang on shutdown if you don’t give that zfs export ... command time to finish its background tasks. I’m guessing that it is highly unlikely that there will be greater than zfs_txg_timeout seconds worth of “backlog” that needs to be flushed when you run zfs export ...

It seems like there should be some better “signal” you can check to actually know that the export is really done, but I don’t know what it is.

46

I would try to verify the presence of the dev nodes before attempting the zfs import ... just because I think that import command is somewhat expensive with the way it scans all the nodes under /dev and, perhaps, blocks other things that might be attempting to run in parallel on system startup (e.g., if you have several pools you are trying to import, it might take much longer for them to import if they are all constantly waiting on each other to finish scanning). But you don’t “have” to run that pre-check. You can just let the zfs import ... command fail and retry a few times.[1] If you don’t have a consistent name of some sort that you can check, you might have to do it that way.

To reduce the “disruption” of that zfs import ... command scanning all the device nodes under /dev somewhat and possibly prevent it from having to double-check the same underlying device several times, I would recommend adding something like -d /dev/disk/by-id or -d /dev/disk/by-path. I think whatever you use will be what shows in the output of zpool list -v. If /dev/mapper contains the drives you need, but not others, that might be the better setting for -d ... so zfs import ... will have fewer incorrect devices to potentially scan.

Another approach that might work under some circumstances would be to create your own subdirectory under /dev that would only contain the drives for the array you are interested in. For example, if you had an external JBOD enclosure attached through a PCI card with PCI_ID 1095:3132, you could write a udev rule like the following to create a /dev/jbod directory with symlinks to all your enclosure’s drives.

ACTION=="remove", GOTO="jbod_end"

SUBSYSTEM=="pci", ENV{PCI_ID}=="1095:3132", TAG="JBOD"
SUBSYSTEM=="block", TAGS=="JBOD", ENV{ID_WWN_WITH_EXTENSION}=="?*", SYMLINK+="jbod/wwn-$env{ID_WWN_WITH_EXTENSION}"

LABEL="jbod_end"

You could then have your ExecStartPre= line check the number of drives under /dev/jbod before proceeding.

  1. Well, it depends on your underlying pool configuration. For example, you wouldn’t want a pool with mirrored drives to come online with missing drives just because it didn’t wait long enough for all the remote drives to show up. ↩︎

47

just as a detail, the VM does shut down faster than 15 seconds, maybe i should try to find the logs..

edit: there still seems to be some issue, here is presumably from the shutdown:

# journalctl -u zfs-storage-pool-import-loadkey-mount.service -b
Sep 20 18:29:25 k8p-clone systemd[1]: Starting zfs-storage-pool-import-loadkey-mount.service - Import and load-keys and mount storage pool...
Sep 20 18:29:25 k8p-clone systemd[1]: zfs-storage-pool-import-loadkey-mount.service: Control process exited, code=exited, status=1/FAILURE
Sep 20 18:29:25 k8p-clone systemd[1]: zfs-storage-pool-import-loadkey-mount.service: Failed with result 'exit-code'.
Sep 20 18:29:25 k8p-clone systemd[1]: Failed to start zfs-storage-pool-import-loadkey-mount.service - Import and load-keys and mount storage pool.
Sep 20 18:29:26 k8p-clone systemd[1]: zfs-storage-pool-import-loadkey-mount.service: Scheduled restart job, restart counter is at 1.
Sep 20 18:29:26 k8p-clone systemd[1]: Starting zfs-storage-pool-import-loadkey-mount.service - Import and load-keys and mount storage pool...
Sep 20 18:29:26 k8p-clone systemd[1]: Finished zfs-storage-pool-import-loadkey-mount.service - Import and load-keys and mount storage pool.

edit2: well it starts and stops without error on a live system.. i dunno
edit3: nah, that error is the first time it looks for the dev nodes, first time usually fails

48

I don’t see an issue. The first “failure” is likely just the ExecStartPre test failing. Then it waits 1 second and tries again and everything succeeds.

You might need Conflicts=shutdown.target in the Unit section to be sure that the service runs the ExecStop commands on shutdown/restart. I thought that condition was applied automatically on most services, but I’m not sure. (It might also be possible that systemd is being clever and killing the sleep command if there is nothing else left to terminate.)

49

manually stopping the service waits 15 seconds, but shutdown usually seems to go faster, i haven’t used a timer to check lol.. But nah it’s time to implement this on real system.

The amounts of reboots it took to get this far would have been impossible to do on the actual system, i would have gone insane..

Can’t thank you enough for all the help, just amazing..

50

well… not so fast, lol

Implementing on real system it worked as expected until kernel update, after which some timings seem to have changed.
After boot:

# zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zroot  3.62T   373G  3.26T        -         -    16%    10%  1.00x    ONLINE  -

# systemctl status zfs-storage-pool-import-loadkey-mount.service
× zfs-storage-pool-import-loadkey-mount.service - Import and load-keys and mount storage pool
     Loaded: loaded (/etc/systemd/system/zfs-storage-pool-import-loadkey-mount.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Sun 2025-09-21 17:52:48 EEST; 3min 44s ago
 Invocation: 27a84d8cf0cd4f2fa6acfd1ca32cd22a
    Process: 2436 ExecStartPre=/usr/bin/test -e /dev/disk/by-id/dm-name-hgst-10t-disk0 (code=exited, status=1/FAILURE)

Sep 21 17:52:48 k8p systemd[1]: zfs-storage-pool-import-loadkey-mount.service: Scheduled restart job, restart counter is at 5.
Sep 21 17:52:48 k8p systemd[1]: zfs-storage-pool-import-loadkey-mount.service: Start request repeated too quickly.
Sep 21 17:52:48 k8p systemd[1]: zfs-storage-pool-import-loadkey-mount.service: Failed with result 'exit-code'.

# systemctl start zfs-storage-pool-import-loadkey-mount.service

# systemctl status zfs-storage-pool-import-loadkey-mount.service
● zfs-storage-pool-import-loadkey-mount.service - Import and load-keys and mount storage pool
     Loaded: loaded (/etc/systemd/system/zfs-storage-pool-import-loadkey-mount.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: active (exited) since Sun 2025-09-21 17:57:10 EEST; 1s ago
 Invocation: bd8e30438202433fbab2b740ee4f468c
    Process: 4604 ExecStartPre=/usr/bin/test -e /dev/disk/by-id/dm-name-hgst-10t-disk0 (code=exited, status=0/SUCCESS)
    Process: 4606 ExecStartPre=/usr/bin/test -e /dev/disk/by-id/dm-name-hgst-10t-disk1 (code=exited, status=0/SUCCESS)
    Process: 4607 ExecStart=/usr/sbin/zpool import -o cachefile=/etc/zfs/storage.cache storage (code=exited, status=0/SUCCESS)
    Process: 4846 ExecStart=/usr/sbin/zfs load-key storage (code=exited, status=0/SUCCESS)
    Process: 4885 ExecStart=/usr/sbin/zfs mount -a (code=exited, status=0/SUCCESS)
   Main PID: 4885 (code=exited, status=0/SUCCESS)
   Mem peak: 12.3M
        CPU: 148ms

Sep 21 17:57:07 k8p systemd[1]: Starting zfs-storage-pool-import-loadkey-mount.service - Import and load-keys and mount storage pool...
Sep 21 17:57:10 k8p systemd[1]: Finished zfs-storage-pool-import-loadkey-mount.service - Import and load-keys and mount storage pool.

# zpool list
NAME      SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
storage  9.09T  4.22T  4.87T        -         -    12%    46%  1.00x    ONLINE  -
zroot    3.62T   373G  3.26T        -         -    16%    10%  1.00x    ONLINE  -

this is with two disks, ie:

[Unit]
Description=Import and load-keys and mount storage pool
Requires=multipathd.service
After=network-online.target
After=multipathd.service
After=iscsid.service
Before=nfs-server.service

[Service]
Type=oneshot
RemainAfterExit=yes
RestartSec=1
Restart=on-failure
ExecStartPre=/usr/bin/test -e /dev/disk/by-id/dm-name-hgst-10t-disk0
ExecStartPre=/usr/bin/test -e /dev/disk/by-id/dm-name-hgst-10t-disk1
ExecStart=/usr/sbin/zpool import -o cachefile=/etc/zfs/storage.cache storage
ExecStart=-/usr/sbin/zfs load-key storage
ExecStart=-/usr/sbin/zfs mount -a
ExecStop=/usr/sbin/zpool export storage
ExecStop=/usr/bin/sleep 15

[Install]
WantedBy=multi-user.target

So maybe it needs a sleep command yet somewhere?

edit: looking at logs, iscsid and multipathd are still working after the service runs..

Sep 21 17:52:47 k8p iscsid[2112]: iscsid: connect to 10.0.0.240:3260 failed (No route to host)
Sep 21 17:52:47 k8p sddm-helper-start-wayland[2172]: "QSGContext::initialize: depth buffer support missing, expect rendering errors\nQSGContext::initialize: stencil buffer support missing, expect rendering errors\n"
Sep 21 17:52:47 k8p kernel: amdgpu 0000:c6:00.0: [drm] *ERROR* Got unknown request 0x30 (SINK_EVENT_NOTIFY)
Sep 21 17:52:47 k8p systemd[1]: zfs-storage-pool-import-loadkey-mount.service: Scheduled restart job, restart counter is at 4.
Sep 21 17:52:47 k8p systemd[1]: Starting zfs-storage-pool-import-loadkey-mount.service - Import and load-keys and mount storage pool...
Sep 21 17:52:47 k8p systemd[1]: zfs-storage-pool-import-loadkey-mount.service: Control process exited, code=exited, status=1/FAILURE
Sep 21 17:52:47 k8p systemd[1]: zfs-storage-pool-import-loadkey-mount.service: Failed with result 'exit-code'.
Sep 21 17:52:47 k8p systemd[1]: Failed to start zfs-storage-pool-import-loadkey-mount.service - Import and load-keys and mount storage pool.
Sep 21 17:52:47 k8p audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=zfs-storage-pool-import-loadkey-mount comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Sep 21 17:52:48 k8p kernel: scsi host0: iSCSI Initiator over TCP/IP
Sep 21 17:52:48 k8p iscsid[2112]: iscsid: Connection1:0 to [target: iqn.2005-01.env.pve.pve2.iscsi:k8p, portal: 192.168.2.240,3260] through [iface: default] is operational now
Sep 21 17:52:48 k8p kernel: scsi 0:0:0:0: Direct-Access     LIO-ORG  hgst-10tb-disk0  4.0  PQ: 0 ANSI: 6
Sep 21 17:52:48 k8p kernel: scsi 0:0:0:0: alua: supports implicit and explicit TPGS
Sep 21 17:52:48 k8p kernel: scsi 0:0:0:0: alua: device naa.60014059e827fb602ec4d96929f7458c port group 0 rel port 1
Sep 21 17:52:48 k8p kernel: sd 0:0:0:0: Attached scsi generic sg0 type 0
Sep 21 17:52:48 k8p kernel: sd 0:0:0:0: [sda] 19532873728 512-byte logical blocks: (10.0 TB/9.10 TiB)
Sep 21 17:52:48 k8p kernel: sd 0:0:0:0: [sda] 4096-byte physical blocks
Sep 21 17:52:48 k8p kernel: sd 0:0:0:0: [sda] Write Protect is off
Sep 21 17:52:48 k8p kernel: sd 0:0:0:0: [sda] Mode Sense: 43 00 10 08
Sep 21 17:52:48 k8p kernel: sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
Sep 21 17:52:48 k8p kernel: scsi 0:0:0:1: Direct-Access     LIO-ORG  hgst-10tb-disk1  4.0  PQ: 0 ANSI: 6
Sep 21 17:52:48 k8p kernel: sd 0:0:0:0: [sda] Preferred minimum I/O size 4096 bytes
Sep 21 17:52:48 k8p kernel: sd 0:0:0:0: [sda] Optimal transfer size 33550336 bytes
Sep 21 17:52:48 k8p kernel: scsi 0:0:0:1: alua: supports implicit and explicit TPGS
Sep 21 17:52:48 k8p kernel: scsi 0:0:0:1: alua: device naa.6001405b575fb6aa7d44bc79fa6d870c port group 0 rel port 1
Sep 21 17:52:48 k8p kernel: sd 0:0:0:1: Attached scsi generic sg1 type 0
Sep 21 17:52:48 k8p kernel: sd 0:0:0:1: [sdb] 19532873728 512-byte logical blocks: (10.0 TB/9.10 TiB)
Sep 21 17:52:48 k8p kernel: sd 0:0:0:1: [sdb] 4096-byte physical blocks
Sep 21 17:52:48 k8p kernel: sd 0:0:0:1: [sdb] Write Protect is off
Sep 21 17:52:48 k8p kernel: sd 0:0:0:1: [sdb] Mode Sense: 43 00 10 08
Sep 21 17:52:48 k8p kernel: sd 0:0:0:1: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
Sep 21 17:52:48 k8p iscsid[2442]: iscsid: connection1:0 IPC qtask write failed: Broken pipe
Sep 21 17:52:48 k8p kernel: sd 0:0:0:1: [sdb] Preferred minimum I/O size 4096 bytes
Sep 21 17:52:48 k8p kernel: sd 0:0:0:1: [sdb] Optimal transfer size 33550336 bytes
Sep 21 17:52:48 k8p kernel:  sda: sda1 sda9
Sep 21 17:52:48 k8p kernel: sd 0:0:0:0: [sda] Attached SCSI disk
Sep 21 17:52:48 k8p kernel:  sdb: sdb1 sdb9
Sep 21 17:52:48 k8p kernel: sd 0:0:0:1: [sdb] Attached SCSI disk
Sep 21 17:52:48 k8p multipathd[1262]: libmp_mapinfo: map hgst-10t-disk1 doesn't exist
Sep 21 17:52:48 k8p multipathd[1262]: libmp_mapinfo: map hgst-10t-disk1 doesn't exist
Sep 21 17:52:48 k8p multipathd[1262]: hgst-10t-disk1: addmap [0 19532873728 multipath 1 queue_if_no_path 1 alua 1 1 service-time 0 1 1 8:16 1]
Sep 21 17:52:48 k8p systemd-homed[1768]: block device /sys/devices/platform/host0/session1/target0:0:0/0:0:0:1/block/sdb/sdb1 has been removed.
Sep 21 17:52:48 k8p systemd-homed[1768]: block device /sys/devices/platform/host0/session1/target0:0:0/0:0:0:1/block/sdb/sdb9 has been removed.
Sep 21 17:52:48 k8p kernel: device-mapper: multipath service-time: version 0.3.0 loaded
Sep 21 17:52:48 k8p systemd-homed[1768]: block device /sys/devices/platform/host0/session1/target0:0:0/0:0:0:0/block/sda/sda1 has been removed.
Sep 21 17:52:48 k8p systemd-homed[1768]: block device /sys/devices/platform/host0/session1/target0:0:0/0:0:0:0/block/sda/sda9 has been removed.
Sep 21 17:52:48 k8p multipathd[1262]: sdb [8:16]: path added to devmap hgst-10t-disk1
Sep 21 17:52:48 k8p multipathd[1262]: libmp_mapinfo: map hgst-10t-disk0 doesn't exist
Sep 21 17:52:48 k8p multipathd[1262]: libmp_mapinfo: map hgst-10t-disk0 doesn't exist
Sep 21 17:52:48 k8p multipathd[1262]: hgst-10t-disk0: addmap [0 19532873728 multipath 1 queue_if_no_path 1 alua 1 1 service-time 0 1 1 8:0 1]
Sep 21 17:52:48 k8p kernel: sd 0:0:0:1: alua: transition timeout set to 60 seconds
Sep 21 17:52:48 k8p kernel: sd 0:0:0:1: alua: port group 00 state A non-preferred supports TOlUSNA
Sep 21 17:52:48 k8p multipathd[1262]: sda [8:0]: path added to devmap hgst-10t-disk0
Sep 21 17:52:48 k8p kernel: sd 0:0:0:0: alua: transition timeout set to 60 seconds
Sep 21 17:52:48 k8p kernel: sd 0:0:0:0: alua: port group 00 state A non-preferred supports TOlUSNA
Sep 21 17:52:48 k8p systemd[1]: zfs-storage-pool-import-loadkey-mount.service: Scheduled restart job, restart counter is at 5.
Sep 21 17:52:48 k8p audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=zfs-storage-pool-import-loadkey-mount comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Sep 21 17:52:48 k8p systemd[1]: zfs-storage-pool-import-loadkey-mount.service: Start request repeated too quickly.
Sep 21 17:52:48 k8p systemd[1]: zfs-storage-pool-import-loadkey-mount.service: Failed with result 'exit-code'.
Sep 21 17:52:48 k8p systemd[1]: Failed to start zfs-storage-pool-import-loadkey-mount.service - Import and load-keys and mount storage pool.
Sep 21 17:52:49 k8p multipathd[1262]: hgst-10t-disk0: sda - directio checker reports path is up
Sep 21 17:52:49 k8p multipathd[1262]: 8:0: reinstated
Sep 21 17:52:49 k8p multipathd[1262]: hgst-10t-disk0: remaining active paths: 1
Sep 21 17:52:52 k8p kernel: scsi host1: iSCSI Initiator over TCP/IP
Sep 21 17:52:52 k8p iscsid[2112]: iscsid: Connection2:0 to [target: iqn.2005-01.env.pve.pve2.iscsi:k8p, portal: 10.0.0.240,3260] through [iface: default] is operational now
Sep 21 17:52:52 k8p kernel: scsi 1:0:0:0: Direct-Access     LIO-ORG  hgst-10tb-disk0  4.0  PQ: 0 ANSI: 6
Sep 21 17:52:52 k8p kernel: scsi 1:0:0:0: alua: supports implicit and explicit TPGS
Sep 21 17:52:52 k8p kernel: scsi 1:0:0:0: alua: device naa.60014059e827fb602ec4d96929f7458c port group 0 rel port 1
Sep 21 17:52:52 k8p kernel: sd 1:0:0:0: Attached scsi generic sg2 type 0
Sep 21 17:52:52 k8p kernel: sd 1:0:0:0: [sdc] 19532873728 512-byte logical blocks: (10.0 TB/9.10 TiB)
Sep 21 17:52:52 k8p kernel: sd 1:0:0:0: [sdc] 4096-byte physical blocks
Sep 21 17:52:52 k8p kernel: sd 1:0:0:0: [sdc] Write Protect is off
Sep 21 17:52:52 k8p kernel: sd 1:0:0:0: [sdc] Mode Sense: 43 00 10 08
Sep 21 17:52:52 k8p kernel: sd 1:0:0:0: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA
Sep 21 17:52:52 k8p kernel: scsi 1:0:0:1: Direct-Access     LIO-ORG  hgst-10tb-disk1  4.0  PQ: 0 ANSI: 6
Sep 21 17:52:52 k8p kernel: sd 1:0:0:0: [sdc] Preferred minimum I/O size 4096 bytes
Sep 21 17:52:52 k8p kernel: sd 1:0:0:0: [sdc] Optimal transfer size 33550336 bytes
Sep 21 17:52:52 k8p iscsid[2545]: iscsid: connection2:0 IPC qtask write failed: Broken pipe
Sep 21 17:52:52 k8p kernel: scsi 1:0:0:1: alua: supports implicit and explicit TPGS
Sep 21 17:52:52 k8p kernel: scsi 1:0:0:1: alua: device naa.6001405b575fb6aa7d44bc79fa6d870c port group 0 rel port 1
Sep 21 17:52:52 k8p kernel: sd 1:0:0:1: Attached scsi generic sg3 type 0
Sep 21 17:52:52 k8p kernel: sd 1:0:0:1: [sdd] 19532873728 512-byte logical blocks: (10.0 TB/9.10 TiB)
Sep 21 17:52:52 k8p kernel: sd 1:0:0:1: [sdd] 4096-byte physical blocks
Sep 21 17:52:52 k8p kernel: sd 1:0:0:1: [sdd] Write Protect is off
Sep 21 17:52:52 k8p kernel: sd 1:0:0:1: [sdd] Mode Sense: 43 00 10 08
Sep 21 17:52:52 k8p kernel: sd 1:0:0:1: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA
Sep 21 17:52:52 k8p kernel: sd 1:0:0:1: [sdd] Preferred minimum I/O size 4096 bytes
Sep 21 17:52:52 k8p kernel: sd 1:0:0:1: [sdd] Optimal transfer size 33550336 bytes
Sep 21 17:52:52 k8p kernel:  sdc: sdc1 sdc9
Sep 21 17:52:52 k8p kernel: sd 1:0:0:0: [sdc] Attached SCSI disk
Sep 21 17:52:52 k8p kernel:  sdd: sdd1 sdd9
Sep 21 17:52:52 k8p kernel: sd 1:0:0:1: [sdd] Attached SCSI disk
Sep 21 17:52:52 k8p multipathd[1262]: hgst-10t-disk0: reload [0 19532873728 multipath 1 queue_if_no_path 1 alua 1 1 service-time 0 2 1 8:0 1 8:32 1]
Sep 21 17:52:52 k8p systemd-homed[1768]: block device /sys/devices/platform/host1/session2/target1:0:0/1:0:0:0/block/sdc/sdc1 has been removed.
Sep 21 17:52:52 k8p systemd-homed[1768]: block device /sys/devices/platform/host1/session2/target1:0:0/1:0:0:0/block/sdc/sdc9 has been removed.
Sep 21 17:52:52 k8p systemd-homed[1768]: block device /sys/devices/platform/host1/session2/target1:0:0/1:0:0:1/block/sdd/sdd9 has been removed.
Sep 21 17:52:52 k8p systemd-homed[1768]: block device /sys/devices/platform/host1/session2/target1:0:0/1:0:0:1/block/sdd/sdd1 has been removed.
Sep 21 17:52:52 k8p multipathd[1262]: sdc [8:32]: path added to devmap hgst-10t-disk0
Sep 21 17:52:52 k8p multipathd[1262]: hgst-10t-disk1: reload [0 19532873728 multipath 1 queue_if_no_path 1 alua 1 1 service-time 0 2 1 8:16 1 8:48 1]
Sep 21 17:52:52 k8p multipathd[1262]: sdd [8:48]: path added to devmap hgst-10t-disk1

edit2: i added ExecStart=/usr/bin/sleep 10 before the two ExecStartPre commands, and now it seems ok.. wish there was a more clean way..

51

You can increase the time between retries with that RestartSec= option. You can also change StartLimitIntervalSec= and StartLimitBurst= (in the [Unit] section) if you want your service to retry more than the default of 5 times.

There are even options to configure an exponential backoff algorithm for the restart frequency, but I’ve never attempted that.

P.S. It looks like these are the defaults for all services in Fedora Linux:

$ grep StartLimit /etc/systemd/system.conf 
#DefaultStartLimitIntervalSec=10s
#DefaultStartLimitBurst=5

I wouldn’t change the global default for all services though. I would just adjust the settings for the service you are trying to get working.

52

yeah i increased RestartSec=2 and also added that 10 second delay before it starts checking if the nodes exist.. works now

53 
[Unit]
Description=Import and load-keys and mount storage pool
Requires=multipathd.service
After=network-online.target
After=multipathd.service
After=iscsid.service
Before=nfs-server.service

[Service]
Type=oneshot
RemainAfterExit=yes
RestartSec=2
Restart=on-failure
ExecStart=/usr/bin/sleep 10
ExecStartPre=/usr/bin/test -e /dev/disk/by-id/dm-name-hgst-10t-disk0
ExecStartPre=/usr/bin/test -e /dev/disk/by-id/dm-name-hgst-10t-disk1
ExecStart=/usr/sbin/zpool import -o cachefile=/etc/zfs/storage.cache storage
ExecStart=-/usr/sbin/zfs load-key storage
ExecStart=-/usr/sbin/zfs mount -a
ExecStop=/usr/sbin/zpool export storage
ExecStop=/usr/bin/sleep 15

[Install]
WantedBy=multi-user.target
54

I think you meant “ExecStartPre=/usr/bin/sleep 10”. Otherwise, the sleep won’t happen until after the tests are run.

An exponential backoff algo is probably the best solution for this type of service where things might be ready immediately, or it could be a (relatively) long time. But I’ve never done that before (it’s a newish feature in systemd).

55

right, ofcourse.. well i don’t even know if that one boot was a one-off, or what.. it worked with ExecStart=/usr/bin/sleep 10 and now gonna test with ExecStartPre=/usr/bin/sleep 10

another reboot.. sigh.. edit.. seems fine

56

Still any of this doesn’t change the fact that multipathd is still working long after my service starts.. and it shouldn’t, it should have been SUCCESS by then..

That’s the real issue

57

i don’t know how to check, or how .target stuff works.. like network-online.target
maybe multipathd has some kind of function like that…

anyways, thanks again for the help

58

It could be that multipathd is working correctly and things are actually “up” but the udev subsystem just hasn’t caught up yet and created the symlinks under /dev that you are look for. You might try using ExectStartPre=/usr/sbin/udevadm settle instead of your sleep command.

59

good idea.. but not tonight, i have rebooted enough lol

60

actually.. from systemd-udev-settle.service manpages:

“This service calls udevadm settle to wait until all events that have been queued by udev(7) have been processed. It is a crude way to wait until “all” hardware has been discovered. Services may pull in this service and order themselves after it to wait for the udev queue to be empty.”

This might be the key.. will test tomorrow. Thanks for the tip!

EDIT: wait… the manpages continue:

"Using this service is not recommended. There can be no guarantee that hardware is fully discovered at any specific time, because the kernel does hardware detection asynchronously, and certain buses and devices take a very long time to become ready, and also additional hardware may be plugged in at any time.

  •   Instead, services should subscribe to udev events and react to any new hardware as it is discovered. Services that, based on configuration, expect certain devices to appear, may warn or report failure after a timeout. This timeout should be tailored to the hardware type. Waiting for systemd-udev-settle.service*

  •   usually slows boot significantly, because it means waiting for all unrelated events too."*

and checking the service:

# systemctl status systemd-udev-settle.service status
Unit status.service could not be found.
● systemd-udev-settle.service - Wait for udev To Complete Device Initialization
     Loaded: loaded (/usr/lib/systemd/system/systemd-udev-settle.service; static)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: active (exited) since Sun 2025-09-21 19:14:03 EEST; 1h 23min ago
 Invocation: 8943077aa5e349028a81781b3bcbd764
       Docs: man:systemd-udev-settle.service(8)
    Process: 1306 ExecStart=udevadm settle (code=exited, status=0/SUCCESS)
   Main PID: 1306 (code=exited, status=0/SUCCESS)
   Mem peak: 2.8M
        CPU: 9ms

Sep 21 19:14:02 k8p systemd[1]: Starting systemd-udev-settle.service - Wait for udev To Complete Device Initialization...
Sep 21 19:14:02 k8p udevadm[1306]: systemd-udev-settle.service is deprecated. Please fix zfs-import-cache.service not to pull it in.
Sep 21 19:14:03 k8p systemd[1]: Finished systemd-udev-settle.service - Wait for udev To Complete Device Initialization.

interesting..