And that service runs extremely early in the boot process as indicated by:
DefaultDependencies=no
Before=sysinit.target
You would need to run the udevadm settle command (which appears to be all that the systemd-udev-settle service does) after your multipathd service has queued additional requests to create those dev nodes in the udev system. Just adding ExecStartPre=-/usr/sbin/udevadm settle -t 10 would be the way to do it.
It won’t help to order your service after the systemd-udev-settle.service because it will have finished successfully long before your service runs. The udev event queue will have since been re-populated by multipathd (among other things) and you need to wait for the queue to empty again.
From the udevadm man page (emphasis added):
udevadm settle [options]
Watches the udev event queue, and exits if all current events are handled.
P.S. You might want to make use of the -t option to limit how long it might potentially wait and prefix the command with - to ignore its return code.
-t, --timeout=SECONDS
Maximum number of seconds to wait for the event queue to become empty. The default value is 120 seconds. A value of 0 will check if the queue is empty and always return immediately. A non-zero value will return an exit code of 0 if queue became empty before timeout was reached, non-zero otherwise.
Yeah, I had overlooked that -E option. That sounds like exactly what you are looking for. I wonder if the option can be repeated to check for multiple files? (Eh, it wouldn’t hurt to run the udevadm command multiple times in separate ExecStartPre lines with different -E options. Most of the time, the subsequent runs will return immediately.) If it works, you can replace those test -e ... commands completely with udevadm settle -E ....
But the default timeout for udevadm settle is 120 seconds:
If you don’t want systemd to kill the udevadm settle command prematurely, you should probably set TimeoutStartSec=120 (or greater) in the [Service] section of your systemd service.
One thing i’ve been thinking about, with iscsi backing, if i move files from one dataset to another dataset on the storage pool, that data must travel through the client via ethernet right?
Yes, I think so. If you move files within the same dataset, the filesystem can just update some metadata, but a different dataset would potentially have different compression or encryption and the data would need to pass through your client’s CPU to be decoded and reencoded. Ideally, you want a dedicated full-duplex network connection for your storage. (But that isn’t required.)
P.S. You might update the solution to point to the latest version of your config. I think using udevadm settle -E ... is a better solution than the plain test -e ... command I originally recommended.
Yeah, every dataset is an independent filesystem, even if all the settings are the same. You can’t just move files betweem them in a traditional sense by only updating the file allocation table..
Again, thank you so much for all your help, it’s invaluable.
You shouldn’t ignore the return code from udevadm settle now that you have it testing for the existence of the dev node (i.e. don’t prefix the command with -). I know I said to include - earlier when I was suggesting that you use udevadm settle (without -E ...) to pause and wait for the event queue to empty, but now that you are using that -E ... parameter, the return code is important and there is no point in continuing with the next command – zfs import ... – if that dev node does not exist. Instead, you should let the command return a failure code and cause the whole service to fail. If the service fails, it will retry again with the timeout reinitialized to zero.