Help with gracefully importing and exporting multipath iscsi backed zfs pool during boot/shutdown

Ask Fedora
,
1

Hi, i’m trying to run a zfs storage pool on block device exported over iscsi with multipath enabled.

At boot, zfs-import-cache service fails, obviously because the iscsi link is not ready.

After boot is complete, i can manually import the pool fine.

at shutdown/reboot, there are some errors shown in logs below, and finally the system fails to poweoff, with error message shown in spice screenshot.

This is a test VM with a test zfs pool, so no risk of dataloss. I wanted to ask for tips/advice on doing this the right way:

The setup is as follows:

the block device is exported using targetcli as follows:

o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 3]

…

 | | o- k8p-clone-storage ...................................... [/dev/zvol/rpool/k8p-clone-storage (10.0GiB) write-thru activated]
  | |   o- alua ................................................................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]

…

 | o- iqn.2005-01.env.pve.pve2.iscsi:k8p-clone .......................................................................... [TPGs: 1]
  |   o- tpg1 .......................................................................................... [no-gen-acls, auth per-acl]
  |     o- acls .......................................................................................................... [ACLs: 1]
  |     | o- iqn.1994-05.com.redhat:bbd7d03ff410 ...................................................... [1-way auth, Mapped LUNs: 1]
  |     |   o- mapped_lun0 ..................................................................... [lun0 block/k8p-clone-storage (rw)]
  |     o- luns .......................................................................................................... [LUNs: 1]
  |     | o- lun0 ................................. [block/k8p-clone-storage (/dev/zvol/rpool/k8p-clone-storage) (default_tg_pt_gp)]
  |     o- portals .................................................................................................... [Portals: 2]
  |       o- 10.0.0.240:3260 .................................................................................................. [OK]
  |       o- 192.168.2.240:3260 ............................................................................................... [OK]

link is up and running:

# multipathd show maps topology
storage (360014054244e8dce0484ffc98805292a) dm-1 LIO-ORG,k8p-clone-stora
size=10G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=enabled
  |- 7:0:0:0 sdb 8:16 active ready running
  `- 8:0:0:0 sdc 8:32 active ready running


# cat /etc/multipath.conf 
defaults {
        verbosity 2
        polling_interval 5
        max_polling_interval 20
        reassign_maps "no"
        path_selector "service-time 0"
        path_grouping_policy "failover"
        uid_attribute "ID_SERIAL"
        prio "const"
        prio_args ""
        features "0"
        path_checker "tur"
        alias_prefix "mpath"
        failback "manual"
        rr_min_io 1000
        rr_min_io_rq 1
        max_fds "max"
        rr_weight "uniform"
        queue_without_daemon "no"
        allow_usb_devices "no"
        flush_on_last_del "unused"
        user_friendly_names "no"
        fast_io_fail_tmo 5
        log_checker_err "always"
        all_tg_pt "no"
        retain_attached_hw_handler "yes"
        detect_prio "yes"
        detect_checker "yes"
        detect_pgpolicy "yes"
        detect_pgpolicy_use_tpg "no"
        force_sync "no"
        strict_timing "no"
        deferred_remove "no"
        delay_watch_checks "no"
        delay_wait_checks "no"
        san_path_err_threshold "no"
        san_path_err_forget_rate "no"
        san_path_err_recovery_time "no"
        marginal_path_err_sample_time "no"
        marginal_path_err_rate_threshold "no"
        marginal_path_err_recheck_gap_time "no"
        marginal_path_double_failed_time "no"
        find_multipaths "off"
        uxsock_timeout 4000
        retrigger_tries 3
        retrigger_delay 10
        missing_uev_wait_timeout 30
        skip_kpartx "no"
        remove_retries 0
        ghost_delay "no"
        auto_resize "never"
        find_multipaths_timeout -10
        enable_foreign "NONE"
        marginal_pathgroups "off"
        recheck_wwid "no"
}
blacklist {
        devnode "!^(sd[a-z]|dasd[a-z]|nvme[0-9])"
        device {
                vendor "SGI"
                product "Universal Xport"
        }
        device {
                vendor "^DGC"
                product "LUNZ"
        }
        device {
                vendor "EMC"
                product "LUNZ"
        }
        device {
                vendor "DELL"
                product "Universal Xport"
        }
        device {
                vendor "FUJITSU"
                product "Universal Xport"
        }
        device {
                vendor "IBM"
                product "Universal Xport"
        }
        device {
                vendor "IBM"
                product "S/390"
        }
        device {
                vendor "LENOVO"
                product "Universal Xport"
        }
        device {
                vendor "(NETAPP|LSI|ENGENIO)"
                product "Universal Xport"
        }
        device {
                vendor "STK"
                product "Universal Xport"
        }
        device {
                vendor "SUN"
                product "Universal Xport"
        }
        device {
                vendor "(Intel|INTEL)"
                product "VTrak V-LUN"
        }
        device {
                vendor "Promise"
                product "VTrak V-LUN"
        }
        device {
                vendor "Promise"
                product "Vess V-LUN"
        }
}
blacklist_exceptions {
                wwid "360014054244e8dce0484ffc98805292a"
}
devices {
        device {
                vendor "(LIO-ORG|SUSE)"
                product ".*"
                path_grouping_policy "group_by_prio"
                path_checker "directio"
                hardware_handler "1 alua"
                prio "alua"
                failback "immediate"
                no_path_retry 12
                detect_checker "no"
        }
}
overrides {
}
multipaths {
        multipath {
                wwid "360014054244e8dce0484ffc98805292a"
                alias "storage"
        }
}


# ls -la /dev/disk/by-id/ | grep dm-1
lrwxrwxrwx. 1 root root  10 Sep 18 07:59 dm-name-storage -> ../../dm-1
lrwxrwxrwx. 1 root root  10 Sep 18 07:59 dm-uuid-mpath-360014054244e8dce0484ffc98805292a -> ../../dm-1
lrwxrwxrwx. 1 root root  10 Sep 18 07:59 scsi-360014054244e8dce0484ffc98805292a -> ../../dm-1
lrwxrwxrwx. 1 root root  10 Sep 18 07:59 wwn-0x60014054244e8dce0484ffc98805292a -> ../../dm-1

after boot, i can import pool fine:

# zpool import storage
# zpool list
NAME      SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
storage  9.50G  1.13M  9.50G        -         -     1%     0%  1.00x    ONLINE  -
zroot    97.5G  6.91G  90.6G        -         -    19%     7%  1.00x    ONLINE  -
# zpool status storage 
  pool: storage
 state: ONLINE
status: One or more devices are configured to use a non-native block size.
        Expect reduced performance.
action: Replace affected devices with devices that support the
        configured block size, or migrate data to a properly configured
        pool.
config:

        NAME                                      STATE     READ WRITE CKSUM
        storage                                   ONLINE       0     0     0
          wwn-0x60014054244e8dce0484ffc98805292a  ONLINE       0     0     0  block size: 4096B configured, 16384B native

relevant log entries from boot:

Sep 17 18:20:02 k8p-clone multipathd[1020]: libmp_mapinfo: map 0QEMU_QEMU_HARDDISK_drive-scsi0 doesn't exist
Sep 17 18:20:02 k8p-clone multipathd[1020]: sda: setting scsi timeouts is unsupported for protocol scsi:unspec
Sep 17 18:20:02 k8p-clone multipathd[1020]: libmp_mapinfo: map 0QEMU_QEMU_HARDDISK_drive-scsi0 doesn't exist
Sep 17 18:20:02 k8p-clone multipathd[1020]: 0QEMU_QEMU_HARDDISK_drive-scsi0: addmap [0 209715200 multipath 0 0 1 1 service-time 0 1 1 8:0 1]
Sep 17 18:20:02 k8p-clone kernel: Adding 8388604k swap on /dev/zram0.  Priority:100 extents:1 across:8388604k SSDsc
Sep 17 18:20:02 k8p-clone systemd[1]: Activated swap dev-zram0.swap - Compressed Swap on /dev/zram0.
Sep 17 18:20:02 k8p-clone systemd[1]: Reached target swap.target - Swaps.
Sep 17 18:20:02 k8p-clone multipathd[1020]: libdevmapper: ioctl/libdm-iface.c(1995): device-mapper: reload ioctl on 0QEMU_QEMU_HARDDISK_drive-scsi0 (252:>
Sep 17 18:20:02 k8p-clone kernel: device-mapper: multipath service-time: version 0.3.0 loaded
Sep 17 18:20:02 k8p-clone kernel: device-mapper: table: 252:0: multipath: error getting device (-EBUSY)
Sep 17 18:20:02 k8p-clone kernel: device-mapper: ioctl: error adding target to table
Sep 17 18:20:02 k8p-clone multipathd[1020]: dm_addmap: libdm task=0 error: Device or resource busy
Sep 17 18:20:02 k8p-clone multipathd[1020]: libmp_mapinfo: map 0QEMU_QEMU_HARDDISK_drive-scsi0 doesn't exist
Sep 17 18:20:02 k8p-clone multipathd[1020]: 0QEMU_QEMU_HARDDISK_drive-scsi0: failed in domap for addition of new path sda
Sep 17 18:20:02 k8p-clone multipathd[1020]: uevent trigger error


Sep 17 18:20:32 k8p-clone systemd[1]: Starting zfs-import-cache.service - Import ZFS pools by cache file...
Sep 17 18:20:32 k8p-clone (zpool)[1471]: zfs-import-cache.service: Referenced but unset environment variable evaluates to an empty string: ZPOOL_IMPORT_O>

[...]

Sep 17 18:20:34 k8p-clone zpool[1471]: cannot import 'storage': no such pool or dataset
Sep 17 18:20:34 k8p-clone zpool[1471]: cannot import '(null)': no such pool available
Sep 17 18:20:34 k8p-clone zpool[1471]:         Destroy and re-create the pool from
Sep 17 18:20:34 k8p-clone zpool[1471]:         a backup source.
Sep 17 18:20:34 k8p-clone zpool[1471]: cachefile import failed, retrying
Sep 17 18:20:34 k8p-clone systemd[1]: zfs-import-cache.service: Main process exited, code=exited, status=1/FAILURE
Sep 17 18:20:34 k8p-clone systemd[1]: zfs-import-cache.service: Failed with result 'exit-code'.
Sep 17 18:20:34 k8p-clone systemd[1]: Failed to start zfs-import-cache.service - Import ZFS pools by cache file.

relevant log entry from shutdown:

Sep 17 18:26:59 k8p-clone systemd[1]: Stopping multipathd.service - Device-Mapper Multipath Device Controller...
Sep 17 18:26:59 k8p-clone multipathd[1020]: multipathd: shut down
Sep 17 18:26:59 k8p-clone systemd[1]: Stopping lvm2-monitor.service - Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling...
Sep 17 18:26:59 k8p-clone systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully.
Sep 17 18:26:59 k8p-clone systemd[1]: Stopped systemd-tmpfiles-setup-dev.service - Create Static Device Nodes in /dev.
Sep 17 18:26:59 k8p-clone audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-tmpfiles-s>
Sep 17 18:26:59 k8p-clone systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully.
Sep 17 18:26:59 k8p-clone systemd[1]: Stopped systemd-tmpfiles-setup-dev-early.service - Create Static Device Nodes in /dev gracefully.
Sep 17 18:26:59 k8p-clone audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-tmpfiles-s>
Sep 17 18:26:59 k8p-clone kernel: audit: type=1131 audit(1758122819.348:392): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0>
Sep 17 18:26:59 k8p-clone kernel: audit: type=1131 audit(1758122819.348:393): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0>
Sep 17 18:26:59 k8p-clone kernel: I/O error, dev dm-1, sector 0 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 0
Sep 17 18:26:59 k8p-clone systemd[1]: multipathd.service: Deactivated successfully.
Sep 17 18:26:59 k8p-clone systemd[1]: Stopped multipathd.service - Device-Mapper Multipath Device Controller.

results in hanged system:

spice_iscsi
spice_iscsi1494×929 185 KB

specifically calling on @glb for this, as he is very knowledgeable about these things.

Few notes:

i don’t know if i should configure zfs vdev stuff for this:

# ls -la /etc/zfs/vdev_id.conf*
-rw-r--r--. 1 root root 165 Jun 19 19:31 /etc/zfs/vdev_id.conf.alias.example
-rw-r--r--. 1 root root 166 Jun 19 19:31 /etc/zfs/vdev_id.conf.multipath.example
-rw-r--r--. 1 root root 616 Jun 19 19:31 /etc/zfs/vdev_id.conf.sas_direct.example
-rw-r--r--. 1 root root 152 Jun 19 19:31 /etc/zfs/vdev_id.conf.sas_switch.example
-rw-r--r--. 1 root root 254 Jun 19 19:31 /etc/zfs/vdev_id.conf.scsi.example

as documented here:

https://openzfs.github.io/openzfs-docs/man/master/5/vdev_id.conf.5.html

some references i found:

https://github.com/openzfs/zfs/issues/15929

https://github.com/openzfs/zfs/issues/3918

Thanks for any tips/insight!

2

one string in the boot logs was cut short, but seems interesting:

Sep 17 18:20:32 k8p-clone (zpool)[1471]: zfs-import-cache.service: Referenced but unset environment variable evaluates to an empty string: ZPOOL_IMPORT_OPTS
3

Hello again Unused Hardhat :slightly_smiling_face:

IIRC, that ZPOOL_IMPORT_OPTS variable was/is a way of passing -f (force) from the kernel command line. I think there was something broken about it when I last looked at it, but I never bothered to file a report upstream. Unless you need it, I wouldn’t worry too much about that one.

The zfs-import-cache service reads from a zpool.cache file under /etc/zfs. The contents of that /etc/zfs directory are packaged into the initramfs so that the pool configuration data is available during early boot.

There is some documentation about it in the zpoolprops man page.

cachefile=path|none
Controls the location of where the pool configuration is cached. Discovering all pools on system startup requires a cached copy of the configuration data that is stored on the root file system. All pools in this cache are automatically imported when the system boots. Some environments, such as install and clustering, need to cache this information in a different location so that pools are not automatically imported. Setting this property caches the pool configuration in a different location that can later be imported with zpool import -c. Setting it to the value none creates a temporary pool that is never cached, and the “” (empty string) uses the default location.

Personally, I disable (mask) the whole zfs-import-cache service and use a customized version of zfs-import-scan to import my pools. But just moving the cache file for that iSCSI-backed pool to a custom location and then making a zfs-import-custom-cache service that runs when you want it to might work better for your situation.

For handling shutdown, I would add a ExecStop=/usr/sbin/zpool export <your-pool-name> line to your zfs-import-custom-cache service and make sure that the service is ordered after multipathd.service (and anything else that your pool might depend on) and make sure that you set a Requires=multipathd.service as well. On system shutdown, systemd will stop services in the reverse order that they are started. I would also add a ExecStop=-/usr/bin/sleep <short-delay-in-seconds> after the ExecStop=/usr/sbin/zpool export ... line because the zpool export line will return before it has finished flushing all the data to the disks. I’m not sure what the correct delay would be, but IIRC the default configuration for ZFS flushes all data to disk every 15 seconds, so I think a 15 second delay should be adequate. There is probably a better way to do this, but that is what I found works for a vaguely similar system that I am running (I’m using AoE, not iSCSI).

FWIW, here is the custom service file I use on one of my systems.

# cat /etc/systemd/system/filesystem@.service
[Unit]
Description=%i
Requires=etherdrives.target
Requires=modprobe@aoe.service
Requires=network-online.target
After=etherdrives.target
After=modprobe@aoe.service
After=network-online.target
Before=filesystems.target

[Service]
Type=oneshot
ExecStartPre=/usr/bin/bash -c 'while [[ $(aoe-stat | grep "^\s*e[01]\..*up\s*$" | wc -l) != 10 ]]; do /usr/sbin/aoe-discover; sleep 10; done'
ExecStart=/usr/sbin/zpool import -d /dev/etherd -o cachefile=/etc/zfs/%i.cache %i
ExecStop=/usr/sbin/zpool export %i
ExecStop=-/usr/bin/sleep 15
Restart=on-failure
ManagedOOMPreference=omit
RemainAfterExit=yes
TimeoutSec=infinity

[Install]
RequiredBy=filesystems.target

HTH

4

Thanks. Some further testing reveals that shutdown error only occurs when the storage pool is imported when system is shutdown. Multipath is not the reason, i tried disabling it

I have another f42 system with a pool on local disks, which successfully imports the pool automatically (because it’s available when zfs-import-cache runs, and then i have a small systemd service for that load-key’s and mounts the pool, and i have no problem there, without any manual export command..

so this seems to be a bug related to iscsi and zfs pool export interaction at shutdown.

disabling the zfs-import-cache.service renders the system unbootable. This is a zfs on root system (yes, the same system you know very well)..

I’ll just ignore the cache error at boot, and try to create a custom systemd service as per your advice, and report back.. Many thanks for your help

5

Beware that there is a little bit of a timing issue with regard to the contents of that cache file and when you (re)generate your initramfs. If your pool is imported when you run dracut, then the version of the cache file with the imported pool will get embedded in the initramfs and, consequently, dracut will try to import the pool early in the boot process (probably before the dependencies are available). If, on the other hand, your pool happens to be exported when you run dracut, then it will not be in the version of zpool.cache file that is in your initramfs and your system will not attempt to import that pool early in the boot process.

Yes, you would need to customize zfs-import-scan.service to explicity import your root pool (and regenerate your initramfs) in order to switch from using zfs-import-cache to zfs-import-scan (or some other custom service). I just don’t like the way zfs-import-cache ends up caching ZFS pool configurations depending on the state of your system when you last ran dracut. I prefer to be more explicit about what gets imported and when, but it is all a matter of taste – whatever works for you. :slightly_smiling_face:

6

Man, i want simplicity, and as little deviation from standard system as possible. Even small stuff takes a long time for me, as i’m not a elite hax0r like you :stuck_out_tongue:

Am i right though to consider this an actual fedora bug, because just because you run a zfs pool over iscsi should not cause such errors on shutdown?

Anyways, here is what i came up with:

[Unit]
Description=Import and load-keys and mount storage pool
Requires=multipathd.service
After=network-online.target
After=multipathd.service
After=iscsid.service
Before=nfs-server.service

[Service]
Type=oneshot
RemainAfterExit=yes

ExecStart=/usr/bin/bash -c '/usr/sbin/zpool import storage && zfs load-key storage && zfs mount -a'
ExecStop=/usr/bin/bash -c '/usr/sbin/zpool export storage'
ExecStop=/usr/bin/bash -c '/usr/bin/sleep 15'

[Install]
WantedBy=network-online.target

i have no idea if WantedBy=network-online.target is correct, i tried RequiredBy=filesystems.target but that gave an error:

Unit /etc/systemd/system/zfs-storage-pool-import-loadkey-mount.service is added as a dependency to a non-existent unit filesystems.target.

But it seems to work:

# zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zroot  97.5G  6.91G  90.6G        -         -    19%     7%  1.00x    ONLINE  -

root@k8p-clone:~# systemctl start zfs-storage-pool-import-loadkey-mount.service 
root@k8p-clone:~# systemctl status zfs-storage-pool-import-loadkey-mount.service 
● zfs-storage-pool-import-loadkey-mount.service - Import and load-keys and mount storage pool
     Loaded: loaded (/etc/systemd/system/zfs-storage-pool-import-loadkey-mount.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: active (exited) since Thu 2025-09-18 22:53:18 EEST; 7s ago
 Invocation: 005cde6caf2347e29a861404ff68bacd
    Process: 8396 ExecStart=/usr/bin/bash -c /usr/sbin/zpool import storage && zfs load-key storage && zfs mount -a (code=exited, status=0/SUCCESS)
   Main PID: 8396 (code=exited, status=0/SUCCESS)
   Mem peak: 4.2M
        CPU: 79ms

Sep 18 22:53:17 k8p-clone systemd[1]: Starting zfs-storage-pool-import-loadkey-mount.service - Import and load-keys and mount storage pool...
Sep 18 22:53:18 k8p-clone systemd[1]: Finished zfs-storage-pool-import-loadkey-mount.service - Import and load-keys and mount storage pool.

# zpool list
NAME      SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
storage  9.50G  1.11M  9.50G        -         -     1%     0%  1.00x    ONLINE  -
zroot    97.5G  6.91G  90.6G        -         -    19%     7%  1.00x    ONLINE  -

root@k8p-clone:~# systemctl stop zfs-storage-pool-import-loadkey-mount.service 
root@k8p-clone:~# systemctl status zfs-storage-pool-import-loadkey-mount.service 
○ zfs-storage-pool-import-loadkey-mount.service - Import and load-keys and mount storage pool
     Loaded: loaded (/etc/systemd/system/zfs-storage-pool-import-loadkey-mount.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: inactive (dead) since Thu 2025-09-18 22:53:55 EEST; 1s ago
   Duration: 22.000s
 Invocation: 005cde6caf2347e29a861404ff68bacd
    Process: 8396 ExecStart=/usr/bin/bash -c /usr/sbin/zpool import storage && zfs load-key storage && zfs mount -a (code=exited, status=0/SUCCESS)
    Process: 8670 ExecStop=/usr/bin/bash -c /usr/sbin/zpool export storage (code=exited, status=0/SUCCESS)
    Process: 8683 ExecStop=/usr/bin/bash -c /usr/bin/sleep 15 (code=exited, status=0/SUCCESS)
   Main PID: 8396 (code=exited, status=0/SUCCESS)
   Mem peak: 2.2M
        CPU: 25ms

Sep 18 22:53:17 k8p-clone systemd[1]: Starting zfs-storage-pool-import-loadkey-mount.service - Import and load-keys and mount storage pool...
Sep 18 22:53:18 k8p-clone systemd[1]: Finished zfs-storage-pool-import-loadkey-mount.service - Import and load-keys and mount storage pool.
Sep 18 22:53:40 k8p-clone systemd[1]: Stopping zfs-storage-pool-import-loadkey-mount.service - Import and load-keys and mount storage pool...
Sep 18 22:53:55 k8p-clone systemd[1]: zfs-storage-pool-import-loadkey-mount.service: Deactivated successfully.
Sep 18 22:53:55 k8p-clone systemd[1]: Stopped zfs-storage-pool-import-loadkey-mount.service - Import and load-keys and mount storage pool.

now to test rebooting and shutting down…

7

Oh, yeah, filesystems.target is another custom thing I created on my system. The “normal” setting for WantedBy= would be multi-user.target. Having both After=network-online.target and WantedBy=network-online.target set on the same service seems a bit contradictory to me.[1]

  1. I guess it is not quite a dependency loop, but I still wouldn’t do it that way. A network target shouldn’t “want” a storage service under normal circumstances. :slightly_smiling_face: ↩︎

8

It didn’t work, at boot, multipathd still runs after the service, so it can’t find the pool:

# systemctl status zfs-storage-pool-import-loadkey-mount.service 
× zfs-storage-pool-import-loadkey-mount.service - Import and load-keys and mount storage pool
     Loaded: loaded (/etc/systemd/system/zfs-storage-pool-import-loadkey-mount.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Thu 2025-09-18 22:58:43 EEST; 6min ago
 Invocation: b289675f2e224e9f9870771e4a810a76
    Process: 1921 ExecStart=/usr/bin/bash -c /usr/sbin/zpool import storage && zfs load-key storage && zfs mount -a (code=exited, status=1/FAILURE)
   Main PID: 1921 (code=exited, status=1/FAILURE)
   Mem peak: 4.9M
        CPU: 26ms

Sep 18 22:58:43 k8p-clone systemd[1]: Starting zfs-storage-pool-import-loadkey-mount.service - Import and load-keys and mount storage pool...
Sep 18 22:58:43 k8p-clone bash[1925]: cannot import 'storage': no such pool available
Sep 18 22:58:43 k8p-clone systemd[1]: zfs-storage-pool-import-loadkey-mount.service: Main process exited, code=exited, status=1/FAILURE
Sep 18 22:58:43 k8p-clone systemd[1]: zfs-storage-pool-import-loadkey-mount.service: Failed with result 'exit-code'.
Sep 18 22:58:43 k8p-clone systemd[1]: Failed to start zfs-storage-pool-import-loadkey-mount.service - Import and load-keys and mount storage pool.

So.. change to WantedBy=multi-user.target ?

EDIT: it worked in a sense that shutdown -h now now gives no errors, which is awesome

9

Yes.

That shouldn’t happen with After=multipathd.service set. :thinking:

An ugly hack that might work around the issue would be to add ExecStart=/usr/bin/sleep 15 before your other ExecStart= line.

BTW, with Type=oneshot, you can have multiple ExecStart= lines and they will run in order. If any ExecStart= fails, the remaining are skipped and the whole service fails. So you could write your service as:

ExecStart=/usr/bin/sleep 15
ExecStart=/usr/sbin/zpool import storage
ExecStart=/usr/sbin/zfs load-key storage
ExecStart=/usr/sbin/zfs mount -a
ExecStop=/usr/sbin/zpool export storage
ExecStop=/usr/bin/sleep 15

From man systemd.service:

ExecStart=
Commands that are executed when this service is started.

Unless Type= is oneshot, exactly one command must be given. When Type=oneshot is used, this setting may be used multiple times to define multiple commands to execute. If the empty string is assigned to this option, the list of commands to start is reset, prior assignments of this option will have no effect. If no ExecStart= is specified, then the service must have RemainAfterExit=yes and at least one ExecStop= line set. (Services lacking both ExecStart= and ExecStop= are not valid.)

If more than one command is configured, the commands are invoked sequentially in the order they appear in the unit file. If one of the commands fails (and is not prefixed with “-”), other lines are not executed, and the unit is considered failed.

10

from boot logs: first this:

Sep 18 23:09:51 k8p-clone systemd[1]: zfs-storage-pool-import-loadkey-mount.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ An ExecStart= process belonging to unit zfs-storage-pool-import-loadkey-mount.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 1.
Sep 18 23:09:51 k8p-clone systemd[1]: zfs-storage-pool-import-loadkey-mount.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The unit zfs-storage-pool-import-loadkey-mount.service has entered the 'failed' state with result 'exit-code'.
Sep 18 23:09:51 k8p-clone systemd[1]: Failed to start zfs-storage-pool-import-loadkey-mount.service - Import and load-keys and mount storage pool.
░░ Subject: A start job for unit zfs-storage-pool-import-loadkey-mount.service has failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit zfs-storage-pool-import-loadkey-mount.service has finished with a failure.

then a little down, this:

Sep 18 23:09:51 k8p-clone audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-update-utmp-runlevel comm="systemd>
Sep 18 23:09:51 k8p-clone audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-update-utmp-runlevel comm="systemd">
Sep 18 23:09:51 k8p-clone multipathd[942]: libmp_mapinfo: map storage doesn't exist
Sep 18 23:09:51 k8p-clone multipathd[942]: libmp_mapinfo: map storage doesn't exist
Sep 18 23:09:51 k8p-clone multipathd[942]: storage: addmap [0 20971520 multipath 1 queue_if_no_path 1 alua 1 1 service-time 0 1 1 8:16 1]
Sep 18 23:09:51 k8p-clone multipathd[942]: sdb [8:16]: path added to devmap storage
Sep 18 23:09:51 k8p-clone kernel: sd 7:0:0:0: alua: transition timeout set to 60 seconds
Sep 18 23:09:51 k8p-clone kernel: sd 7:0:0:0: alua: port group 00 state A non-preferred supports TOlUSNA
Sep 18 23:09:51 k8p-clone multipathd[942]: storage: performing delayed actions
Sep 18 23:09:51 k8p-clone multipathd[942]: storage: reload [0 20971520 multipath 1 queue_if_no_path 1 alua 1 1 service-time 0 2 1 8:16 1 8:32 1]

So i have no idea.. i need it to run a little later still, because it seems it runs after multipathd.service, but maybe before the multipath link is complete?

11

ugly indeed, i really don’t want an artificial delay like that at boot….

1 Like
12

There are other dependencies you can set as well. You might, for example, add a line like ConditionPathExists=/dev/disk/by-path/<path-to-your-iscsi-devnode> under the [Unit] section and then also add a Restart=on-failure and RestartSec=1 under the [Service] section.

1 Like
13

will do, a little afk here, then i’ll try all of the above

14

Yeah, figuring out all the right dependencies is much better that sticking a random delay in the pipeline and hoping it is “long enough”. :slightly_smiling_face:

15

Oh, I was thinking that Condition... check in the unit section would cause the service to enter the failed state, but I just checked the documentation and that is not the case.

From man systemd.unit:

Failing conditions or asserts will not result in the unit being moved into the “failed” state.

So you would need to use some sort of ExecCondition=... line in the [Service] section instead if you want it to keep trying until the ISCSI device node appears.

Or, perhaps better yet, I think there might be corresponding systemd unit files that are dynamically generated for each ISCSI device. If you can figure out the right ones for your ZFS pool, you might be able to order your ZFS import service after them.

16

i tried ConditionPathExists=/dev/disk/by-id/wwn-0x60014054244e8dce0484ffc98805292a

didn’t work:

 zfs-storage-pool-import-loadkey-mount.service - Import and load-keys and mount storage pool
     Loaded: loaded (/etc/systemd/system/zfs-storage-pool-import-loadkey-mount.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: inactive (dead)
  Condition: start condition unmet at Thu 2025-09-18 23:45:50 EEST; 2min 56s ago
             └─ ConditionPathExists=/dev/disk/by-id/wwn-0x60014054244e8dce0484ffc98805292a was not met

Sep 18 23:45:50 k8p-clone systemd[1]: zfs-storage-pool-import-loadkey-mount.service - Import and load-keys and mount storage pool was skipped because of an unmet condition check (ConditionPathExists=/dev/disk/by-id/wwn-0x60014054244e8dce0484ffc98805292a).

even though

# ls -la /dev/disk/by-id/wwn-0x60014054244e8dce0484ffc98805292a 
lrwxrwxrwx. 1 root root 10 Sep 18 23:45 /dev/disk/by-id/wwn-0x60014054244e8dce0484ffc98805292a -> ../../dm-1
17 
[Unit]
Description=Import and load-keys and mount storage pool
Requires=multipathd.service
After=network-online.target
After=multipathd.service
After=iscsid.service
Before=nfs-server.service
ConditionPathExists=/dev/disk/by-id/wwn-0x60014054244e8dce0484ffc98805292a

[Service]
Type=oneshot
RemainAfterExit=yes
Restart=on-failure
RestartSec=1

ExecStart=/usr/sbin/zpool import storage
ExecStart=/usr/sbin/zfs load-key storage
ExecStart=/usr/sbin/zfs mount -a
ExecStop=/usr/sbin/zpool export storage
ExecStop=/usr/bin/sleep 15

[Install]
WantedBy=network-online.target
18

Yeah, I was wrong about how that ConditionPathExists works. See my previous post.

Edit: Maybe ExecCondition=/usr/bin/test -e /dev/disk/by-id/wwn-0x60014054244e8dce0484ffc98805292a under the [Service] section would work? The ConditionPathExists= is causing it to fail and never retry, so remove that line.

19 
[Unit]
Description=Import and load-keys and mount storage pool
Requires=multipathd.service
After=network-online.target
After=multipathd.service
After=iscsid.service
Before=nfs-server.service

[Service]
Type=oneshot
RemainAfterExit=yes
Restart=on-failure
RestartSec=1
ExecCondition=/usr/bin/test -e /dev/disk/by-id/wwn-0x60014054244e8dce0484ffc98805292a
ExecStart=/usr/sbin/zpool import storage
ExecStart=/usr/sbin/zfs load-key storage
ExecStart=/usr/sbin/zfs mount -a
ExecStop=/usr/sbin/zpool export storage
ExecStop=/usr/bin/sleep 15

[Install]
WantedBy=network-online.target


# systemctl status zfs-storage-pool-import-loadkey-mount.service 
○ zfs-storage-pool-import-loadkey-mount.service - Import and load-keys and mount storage pool
     Loaded: loaded (/etc/systemd/system/zfs-storage-pool-import-loadkey-mount.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: inactive (dead) (Result: exec-condition) since Thu 2025-09-18 23:57:42 EEST; 26s ago
 Invocation: 2ef1ad9c8d3e4be0b440b44901bd8933
  Condition: start condition unmet at Thu 2025-09-18 23:57:42 EEST; 26s ago
    Process: 1932 ExecCondition=/usr/bin/test -e /dev/disk/by-id/wwn-0x60014054244e8dce0484ffc98805292a (code=exited, status=1/FAILURE)
   Mem peak: 1M
        CPU: 5ms

Sep 18 23:57:42 k8p-clone systemd[1]: Starting zfs-storage-pool-import-loadkey-mount.service - Import and load-keys and mount storage pool...
Sep 18 23:57:42 k8p-clone systemd[1]: zfs-storage-pool-import-loadkey-mount.service: Skipped due to 'exec-condition'.
Sep 18 23:57:42 k8p-clone systemd[1]: Condition check resulted in zfs-storage-pool-import-loadkey-mount.service - Import and load-keys and mount storage pool being skipped.
20

So it’s not retrying? Maybe you need Restart=always then? I guess I need to check the docs again. :slightly_smiling_face:

Edit: OK, so it looks like Restart= does not apply to services that are failed due to ExecCondition=[1]. In that case, try changing it to ExecStartPre=/usr/bin/test -e /dev/disk/by-id/wwn-0x60014054244e8dce0484ffc98805292a.

  1. Unless the ExecCondition command returned with exit code 255 “or abnormally”. These systemd services get pretty convoluted pretty quick. :slightly_smiling_face: ↩︎