BTRFS volume mount fails at boot, but works once system is up

I have a multi-device RAID1 BTRFS volume which fails to mount on boot, but which works fine when I mount it manually once the computer is booted up. I would like to be able to get the volume to mount at boot.

Here are some relevant details:

$ cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Wed Jun 10 16:09:15 2020
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
UUID=4a4582ad-c6ff-4c87-a5aa-b8db16dcc3ed  /      ext4   defaults         0 1
UUID=059b5282-ba55-4352-893b-61fb5d9b0d54  /boot  ext4   defaults  	  0 2
UUID=f89f0a16-fc85-4021-9b0f-8356f2bc9c43  /srv   btrfs  defaults,nofail,x-systemd.requires=/  0 0
UUID=26cd7051-85fa-46d6-b70f-b81b0ff30084  /home  ext4   defaults  	  0 2
UUID=5a271f7a-1276-4055-ac24-62bbbea287f8  none   swap   defaults  	  0 0

The logs reveal that there was a problem:

$ sudo journalctl -p 3 -xb
-- Logs begin at Thu 2020-09-17 15:53:32 BST, end at Thu 2020-10-08 10:08:39 BST. --
Oct 08 10:01:35 localhost.localdomain kernel: BTRFS error (device sde): devid 2 uuid 65c21020-6607-425b-a36b-5455d80ab0d1 is missing
Oct 08 10:01:35 localhost.localdomain kernel: BTRFS error (device sde): failed to read the system array: -2
Oct 08 10:01:35 localhost.localdomain kernel: BTRFS error (device sde): open_ctree failed
Oct 08 10:01:35 localhost.localdomain systemd[1]: Failed to mount /srv.
-- Subject: A start job for unit srv.mount has failed

It is not always the same hard drive which is reported as missing. I have also tried swapping hard drives to different bays, and it does not seem to be a bad hard drive connector.

Once the system has booted I can get the volume to mount by running:

$ sudo systemctl list-units --failed
  UNIT               LOAD   ACTIVE SUB    DESCRIPTION                  
â—Ź srv.mount          loaded failed failed /srv                         

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

1 loaded units listed.
$ sudo systemctl restart srv.mount
$ sudo systemctl list-units --failed
  UNIT LOAD ACTIVE SUB DESCRIPTION
0 loaded units listed.
$ ls /srv
git  nfs  www

The volume also seems to behave perfectly fine afterwards:

$ sudo btrfs device stats /srv
[/dev/sdg].write_io_errs    0
[/dev/sdg].read_io_errs     0
[/dev/sdg].flush_io_errs    0
[/dev/sdg].corruption_errs  0
[/dev/sdg].generation_errs  0
[/dev/sdj].write_io_errs    0
[/dev/sdj].read_io_errs     0
[/dev/sdj].flush_io_errs    0
[/dev/sdj].corruption_errs  0
[/dev/sdj].generation_errs  0
[/dev/sdi].write_io_errs    0
[/dev/sdi].read_io_errs     0
[/dev/sdi].flush_io_errs    0
[/dev/sdi].corruption_errs  0
[/dev/sdi].generation_errs  0
[/dev/sde].write_io_errs    0
[/dev/sde].read_io_errs     0
[/dev/sde].flush_io_errs    0
[/dev/sde].corruption_errs  0
[/dev/sde].generation_errs  0
[/dev/sdh].write_io_errs    0
[/dev/sdh].read_io_errs     0
[/dev/sdh].flush_io_errs    0
[/dev/sdh].corruption_errs  0
[/dev/sdh].generation_errs  0
[/dev/sdf].write_io_errs    0
[/dev/sdf].read_io_errs     0
[/dev/sdf].flush_io_errs    0
[/dev/sdf].corruption_errs  0
[/dev/sdf].generation_errs  0
[/dev/sdc].write_io_errs    0
[/dev/sdc].read_io_errs     0
[/dev/sdc].flush_io_errs    0
[/dev/sdc].corruption_errs  0
[/dev/sdc].generation_errs  0
[/dev/sdd].write_io_errs    0
[/dev/sdd].read_io_errs     0
[/dev/sdd].flush_io_errs    0
[/dev/sdd].corruption_errs  0
[/dev/sdd].generation_errs  0
[/dev/sdb].write_io_errs    0
[/dev/sdb].read_io_errs     0
[/dev/sdb].flush_io_errs    0
[/dev/sdb].corruption_errs  0
[/dev/sdb].generation_errs  0
[/dev/sda].write_io_errs    0
[/dev/sda].read_io_errs     0
[/dev/sda].flush_io_errs    0
[/dev/sda].corruption_errs  0
[/dev/sda].generation_errs  0

Any help would be much appreciated! :grinning:

1 Like

I guess the UUID for /srv in fstab is invalid.

blkid ?

2 Likes

Hi @sixpack13, Thank you for your reply.

A good thought, but I don’t think that is it. Here’s the output from blkid:

$ sudo blkid 
[sudo] password for daniel.may: 
/dev/nvme0n1p1: UUID="059b5282-ba55-4352-893b-61fb5d9b0d54" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="5ced97fa-01"
/dev/nvme0n1p2: UUID="rIxtVb-o9de-jXU4-o4WL-yt9H-2Vkb-cvU0U3" TYPE="LVM2_member" PARTUUID="5ced97fa-02"
/dev/mapper/fedora_oxygen-root00: UUID="4a4582ad-c6ff-4c87-a5aa-b8db16dcc3ed" BLOCK_SIZE="4096" TYPE="ext4"
/dev/mapper/fedora_oxygen-swap00: UUID="5a271f7a-1276-4055-ac24-62bbbea287f8" TYPE="swap"
/dev/mapper/fedora_oxygen-home00: UUID="26cd7051-85fa-46d6-b70f-b81b0ff30084" BLOCK_SIZE="4096" TYPE="ext4"
/dev/sdc: LABEL="BTRFS_RAID1_srv" UUID="f89f0a16-fc85-4021-9b0f-8356f2bc9c43" UUID_SUB="22cb2be1-ebb4-4d3f-8768-78f3b72af5d2" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/sdd: LABEL="BTRFS_RAID1_srv" UUID="f89f0a16-fc85-4021-9b0f-8356f2bc9c43" UUID_SUB="e6f34072-6b6c-4082-b033-507ade585f7a" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/sdb: LABEL="BTRFS_RAID1_srv" UUID="f89f0a16-fc85-4021-9b0f-8356f2bc9c43" UUID_SUB="e17daff1-4c3f-46d3-9a11-785d7179f8c2" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/sde: LABEL="BTRFS_RAID1_srv" UUID="f89f0a16-fc85-4021-9b0f-8356f2bc9c43" UUID_SUB="41dfdcad-aeb4-4495-8049-0604147e53a9" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/sdf: LABEL="BTRFS_RAID1_srv" UUID="f89f0a16-fc85-4021-9b0f-8356f2bc9c43" UUID_SUB="8eee4f65-8f79-4f64-ba90-c686a187a52d" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/sda: LABEL="BTRFS_RAID1_srv" UUID="f89f0a16-fc85-4021-9b0f-8356f2bc9c43" UUID_SUB="9a9b05c9-1817-427d-9ae6-6c8a42c285fe" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/sdg: LABEL="BTRFS_RAID1_srv" UUID="f89f0a16-fc85-4021-9b0f-8356f2bc9c43" UUID_SUB="10d6f5ce-830e-4ef9-bb60-cabf80825626" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/sdj: LABEL="BTRFS_RAID1_srv" UUID="f89f0a16-fc85-4021-9b0f-8356f2bc9c43" UUID_SUB="65c21020-6607-425b-a36b-5455d80ab0d1" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/sdi: LABEL="BTRFS_RAID1_srv" UUID="f89f0a16-fc85-4021-9b0f-8356f2bc9c43" UUID_SUB="8d71fba7-04f7-4dc2-8432-ec51d8aaf40c" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/sdh: LABEL="BTRFS_RAID1_srv" UUID="f89f0a16-fc85-4021-9b0f-8356f2bc9c43" UUID_SUB="81ee9f10-9319-49e7-a15f-3882a66abb44" BLOCK_SIZE="4096" TYPE="btrfs"

The UUID for the volume as a whole seems to match the what is in /etc/fstab.

However it is interesting that the UUID_SUB for the drive it was complaining about is now associated with /dev/sdj which is the last drive letter assigned. So it looks like systemd can’t find a drive at boot, then gives up and the drive appears to the system later. Is it something to do with the systemd dependencies for the srv.mount unit? Can I persude it to wait a little longer to give the drives a chance to all appear? Shouldn’t that be happening anyway? :confused:

maybe all drives together are sucking to much current during start of the devices/box, so that the last drive doesn’t get enough ?

is the problem new ?
was that config working before ?
what has changed since ?

1 Like

I’ve got a hypothesis: the mount command is happening before all devices have been seen by the kernel, and therefore the mount fails. Btrfs expects to see all devices at the time the mount command happens, it will not do automatic degraded mount. NOTE: It is is not advised at this time to include degraded mount option to /etc/fstab as a work around.

If you look at /usr/lib/udev/rules.d/64-btrfs.rules what should be true is that this udev rule causes a delay in attempting to mount multiple device btrfs volume until the kernel sees all devices for that btrfs file sytsem. That is, it shouldn’t be trying to mount it.

The ideal scenario is to try and figure out why the mount attempt is happening, because there might be a bug that needs fixing. And the only way it gets fixed is to understand if there’s a problem and what that problem is. I’ll try to reproduce this in a VM.

Here’s a possible work around. Use the following to /etc/fstab line for this Btrfs file system, instead of the one you have:

x-systemd.automount,noauto,nofail,x-systemd.requires=/

This will delay the mount for this file system until something tries to access the /srv mountpoint. By the time that happens, almost certainly all the drives will have been discovered by the kernel.

UPDATE: OK I tried to reproduce this with a 2-device Btrfs raid1, using the same fstab mount options you have, and can’t reproduce it. Mount is not attempted by systemd with one device missing. Can you use your original fstab options, and add boot parameters rd.udev.debug systemd.log_level=debug and then once booted and logged in do sudo journalctl -b -o short-monotonic --no-hostname > journal.txt and post that text file somewhere? Don’t make the boot parameters permanent, they’ll slow down boot quite a bit. If the problem doesn’t reproduce for you when these boot parameters are used, then boot without those boot parameters and just post the journal somewhere and I’ll see if that’s enough info to figure out why the mount is even being attempted while a device is missing.

4 Likes

Hi @chrismurphy

Thank you very much for your detailed reply and work trying to replicate my problem.

I have posted the result of booting with rd.udev.debug.systemd.log_level=debug and running sudo journalctl -b -o short-monotonic --no-hostname > srv_mount_failure_at_boot-journal.txt here:

https://drive.google.com/file/d/1o1-7smQAjg3LKP98EFfeHbb_jZdNz_jT/view?usp=sharing

EDIT: Changed link to one which should be accessible to everyone.

The same problem replicated itself during this boot, i.e. the volume failed to mount but then mounted and was perfectly fine with a sudo systemctl restart srv.mount once logged in.

Interestingly, once booted I ran:

$ grep "BTRFS" srv_mount_failure_at_boot-journal.txt 
[    8.676368] kernel: BTRFS: device label BTRFS_RAID1_srv devid 8 transid 60687 /dev/sdc scanned by systemd-udevd (726)
[    8.696613] kernel: BTRFS: device label BTRFS_RAID1_srv devid 9 transid 60687 /dev/sda scanned by systemd-udevd (719)
[    8.712305] kernel: BTRFS: device label BTRFS_RAID1_srv devid 7 transid 60687 /dev/sdd scanned by systemd-udevd (728)
[    8.734798] kernel: BTRFS: device label BTRFS_RAID1_srv devid 10 transid 60687 /dev/sdb scanned by systemd-udevd (740)
[    8.739384] kernel: BTRFS: device label BTRFS_RAID1_srv devid 6 transid 60687 /dev/sde scanned by systemd-udevd (712)
[    8.743825] kernel: BTRFS: device label BTRFS_RAID1_srv devid 4 transid 60687 /dev/sdf scanned by systemd-udevd (704)
[    9.092727] kernel: BTRFS info (device sdf): disk space caching is enabled
[    9.092730] kernel: BTRFS info (device sdf): has skinny extents
[    9.093897] kernel: BTRFS error (device sdf): devid 2 uuid 65c21020-6607-425b-a36b-5455d80ab0d1 is missing
[    9.093902] kernel: BTRFS error (device sdf): failed to read the system array: -2
[    9.165813] kernel: BTRFS error (device sdf): open_ctree failed
[   12.610775] kernel: BTRFS: device label BTRFS_RAID1_srv devid 2 transid 60687 /dev/sdh scanned by systemd-udevd (738)
[   12.611604] kernel: BTRFS: device label BTRFS_RAID1_srv devid 1 transid 60687 /dev/sdg scanned by systemd-udevd (707)
[   13.377043] kernel: BTRFS: device label BTRFS_RAID1_srv devid 3 transid 60687 /dev/sdj scanned by systemd-udevd (707)
[   13.396209] kernel: BTRFS: device label BTRFS_RAID1_srv devid 5 transid 60687 /dev/sdi scanned by systemd-udevd (738)

And:

$ sudo blkid 
/dev/nvme0n1p1: UUID="059b5282-ba55-4352-893b-61fb5d9b0d54" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="5ced97fa-01"
/dev/nvme0n1p2: UUID="rIxtVb-o9de-jXU4-o4WL-yt9H-2Vkb-cvU0U3" TYPE="LVM2_member" PARTUUID="5ced97fa-02"
/dev/mapper/fedora_oxygen-root00: UUID="4a4582ad-c6ff-4c87-a5aa-b8db16dcc3ed" BLOCK_SIZE="4096" TYPE="ext4"
/dev/mapper/fedora_oxygen-swap00: UUID="5a271f7a-1276-4055-ac24-62bbbea287f8" TYPE="swap"
/dev/mapper/fedora_oxygen-home00: UUID="26cd7051-85fa-46d6-b70f-b81b0ff30084" BLOCK_SIZE="4096" TYPE="ext4"
/dev/sda: LABEL="BTRFS_RAID1_srv" UUID="f89f0a16-fc85-4021-9b0f-8356f2bc9c43" UUID_SUB="e17daff1-4c3f-46d3-9a11-785d7179f8c2" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/sdb: LABEL="BTRFS_RAID1_srv" UUID="f89f0a16-fc85-4021-9b0f-8356f2bc9c43" UUID_SUB="9a9b05c9-1817-427d-9ae6-6c8a42c285fe" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/sdc: LABEL="BTRFS_RAID1_srv" UUID="f89f0a16-fc85-4021-9b0f-8356f2bc9c43" UUID_SUB="e6f34072-6b6c-4082-b033-507ade585f7a" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/sdd: LABEL="BTRFS_RAID1_srv" UUID="f89f0a16-fc85-4021-9b0f-8356f2bc9c43" UUID_SUB="22cb2be1-ebb4-4d3f-8768-78f3b72af5d2" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/sde: LABEL="BTRFS_RAID1_srv" UUID="f89f0a16-fc85-4021-9b0f-8356f2bc9c43" UUID_SUB="8eee4f65-8f79-4f64-ba90-c686a187a52d" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/sdf: LABEL="BTRFS_RAID1_srv" UUID="f89f0a16-fc85-4021-9b0f-8356f2bc9c43" UUID_SUB="41dfdcad-aeb4-4495-8049-0604147e53a9" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/sdg: LABEL="BTRFS_RAID1_srv" UUID="f89f0a16-fc85-4021-9b0f-8356f2bc9c43" UUID_SUB="10d6f5ce-830e-4ef9-bb60-cabf80825626" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/sdh: LABEL="BTRFS_RAID1_srv" UUID="f89f0a16-fc85-4021-9b0f-8356f2bc9c43" UUID_SUB="65c21020-6607-425b-a36b-5455d80ab0d1" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/sdi: LABEL="BTRFS_RAID1_srv" UUID="f89f0a16-fc85-4021-9b0f-8356f2bc9c43" UUID_SUB="81ee9f10-9319-49e7-a15f-3882a66abb44" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/sdj: LABEL="BTRFS_RAID1_srv" UUID="f89f0a16-fc85-4021-9b0f-8356f2bc9c43" UUID_SUB="8d71fba7-04f7-4dc2-8432-ec51d8aaf40c" BLOCK_SIZE="4096" TYPE="btrfs"

Notice how the disk sdf it was complaining about missing at boot with UUID 65c21020-6607-425b-a36b-5455d80ab0d1, is sdh once logged in, and we have a different disk for sdf.

Thank you so much for all your work trying to diagnose and cure this problem. :grinning:

Hi @sixpack13

I don’t think that it can be a lack of current supply. This thing has a pretty beefy power supply (it has 24 drive bays) and it currently only has 10 hot drives in it. The same box has also run fine with many more drives than 10. The spin up of the drives is also staggered at startup so that they are not all pulling maximum startup current at the same time.

The problem started when I migrated the systems big storage from a mdadm based RAID6 based array with an XFS filesystem to a BTRFS RAID1 volume (I wanted BTRFS’s bit-rot protection, and flexibility). The BTRFS volume works perfectly once it has been mounted manually, reporting no errors. So this does seem to be something wrong with the way the drives are being detected and the volume mounted at boot.

Cheers, Dan

Better post here: https://paste.opensuse.org/ (not everybody has a goooogle account)

2 Likes

Hi @florian,

You raise a very good point. I actually made a mistake when I created the original link on Google Drive and did not make it accessible to all. I have since corrected this in my original post and am repeating here for clarity:

https://drive.google.com/file/d/1o1-7smQAjg3LKP98EFfeHbb_jZdNz_jT/view?usp=sharing

This link should be accessible to everyone irrespective of whether they have a Google account or not. I tested it on a VM and it seemed to work for me when not logged in to my Google account.

I tried to use https://paste.opensuse.org/ but I got a 404 after I filled out the form and clicked the “Create” button. I tried a couple of times, but had the same problem. I was not logged in when I did this, but that should not matter, or if it does then I should not see the form and “Create” button… :confused:

If people still have problems accessing the file, let me know and I can try something else, maybe put it on an “old school” web server! :wink:

Cheers, Dan

EDIT: Corrected typo.

1 Like

Are any of these Btrfs drives in USB enclosures? I see a number of USB related problems:

[    4.628233] kernel: usb 3-7-port1: attempt power cycle
...
[    5.436201] kernel: usb 3-7.1: Device not responding to setup address.
...
[    5.644094] kernel: usb 3-7.1: device not accepting address 7, error -71

Also, the google drive attachment doesn’t have rd.udev.debug systemd.log_level=debug set. For sure there is a mount attempt happening before all devices have been scanned by udev.

Also note: BTRFS error (device sdf): is not a literal reference to sdf. That’s just the node btrfs is using to refer to the entire file system. Consider it short hand for the fs UUID. So when we see this:

[    9.093897] kernel: BTRFS error (device sdf): devid 2 uuid 65c21020-6607-425b-a36b-5455d80ab0d1 is missing

It doesn’t mean sdf is missing. It means devid 2 is missing which is mapped to sdh. And the reason why it’s missing is udev hasn’t scanned it, that happens later at [ 12.610775] for both devid 2 and 1. So it’s not just one device that’s showing up late to the party. The question still is, why is systemd even attempting the mount when udev hasn’t scanned all these devices? Hopefully rd.udev.debug will reveal why … or maybe someone will find another clue in the journal that I’ve overlooked.

1 Like

None are USB drives. They are SATA drives which are either directly attached to the motherboard or attached via an HBA card plugged in to a PCI slot on the motherboard.

Yes, I have a few things plugged into a USB hub, perhaps that is causing these issues? I will try rebooting with only the keyboard and mouse plugged in and see if that gets rid of these — just to check what is causing them.

I must have messed this up when I did this. I will have another go and post the log tomorrow.

Ah, interesting! I did not know that.

Thank you so much for your help with this @chrismurphy :smiley:

Yes, I have a few things plugged into a USB hub, perhaps that is causing these issues?

The USB errors, yes. The Btrfs mount issue, no. We could ask why so many of the sdX devices aren’t appearing or being scanned by udev until later. I think that matters less than why the udev rule seems to be ignored.

1 Like

Hi @chrismurphy

Here is another log where I have got the debugging turned on correctly — I made a typo in my original attempt.

https://drive.google.com/file/d/1jVHjAQ8CY9vABtM2giPTB6XeZCclm7R-/view?usp=sharing

Incidentally I simplified my connected USB devices which fixed the errors reported in the previous log. I also installed targetcli which fixed another error which was reported in the previous log. So now the only errors reported are due to the failed attempt to mount the BTRFS volume.

Cheers, Dan

OK super. Here’s what’s going on

[   25.862907] systemd[1]: srv.mount: About to execute: /usr/bin/mount /dev/disk/by-uuid/f89f0a16-fc85-4021-9b0f-8356f2bc9c43 /srv -t btrfs -o defaults,x-systemd.requires=/

That means systemd, via systemd-fstab-generator, has read /etc/fstab and is using that information to mount this file system by UUID. Importantly, it’s not something other than systemd trying to mount. The problem?

[   30.923721] kernel: BTRFS: device label BTRFS_RAID1_srv devid 1 transid 60815 /dev/sdg scanned by systemd-udevd (710)
...
[   30.938057] kernel: BTRFS: device label BTRFS_RAID1_srv devid 2 transid 60815 /dev/sdj scanned by systemd-udevd (699)
...
[   31.815008] kernel: BTRFS: device label BTRFS_RAID1_srv devid 3 transid 60815 /dev/sdi scanned by systemd-udevd (699)
...
[   31.832904] kernel: BTRFS: device label BTRFS_RAID1_srv devid 5 transid 60815 /dev/sdh scanned by systemd-udevd (710)

Four devices weren’t scanned by udev, and ostensibly were not ready, until after the mount attempt. This shouldn’t happen due to /usr/lib/udev/rules.d/64-btrfs.rules and I see no reference at all to that udev rule.

Can you confirm you have this file? /usr/lib/udev/rules.d/64-btrfs.rules
And confirm you have systemd-245.8-2.fc32?

I’m going to start a discussion on the upstream systemd-devel list about this because I don’t understand why this udev rule is apparently being ignored.
https://lists.freedesktop.org/archives/systemd-devel/2020-October/045399.html

3 Likes

Yes, I have it:

$ ls -lhZ /usr/lib/udev/rules.d/64-btrfs.rules 
-rw-r--r--. 1 root root system_u:object_r:lib_t:s0 616 Sep 21 08:43 /usr/lib/udev/rules.d/64-btrfs.rules

$ cat /usr/lib/udev/rules.d/64-btrfs.rules 
# do not edit this file, it will be overwritten on update

SUBSYSTEM!="block", GOTO="btrfs_end"
ACTION=="remove", GOTO="btrfs_end"
ENV{ID_FS_TYPE}!="btrfs", GOTO="btrfs_end"
ENV{SYSTEMD_READY}=="0", GOTO="btrfs_end"

# let the kernel know about this btrfs filesystem, and check if it is complete
IMPORT{builtin}="btrfs ready $devnode"

# mark the device as not ready to be used by the system
ENV{ID_BTRFS_READY}=="0", ENV{SYSTEMD_READY}="0"

# reconsider pending devices in case when multidevice volume awaits
ENV{ID_BTRFS_READY}=="1", RUN+="/usr/bin/udevadm trigger -s block -p ID_BTRFS_READY=0"

LABEL="btrfs_end"

Yes, I do:

$ sudo rpm -q systemd
systemd-245.8-2.fc32.x86_64

$ sudo systemctl --version
systemd 245 (v245.8-2.fc32)
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified

Great thank you. I will follow that and try and add any more information that I can when appropriate. Just a FYI I won’t be able to get access to this system for the next few days but will be able to by the end of this week.

Cheers, Dan

1 Like

As you can see in that systemd-devel@ thread, two more bits of info are needed.

  1. Instead of rd.udev.debug use udev.log_priority=debug and reproduce the problem and post that in the systemd-devel@ thread.

  2. Also confirm in that same email to systemd-devel@ the result from:

sudo lsinitrd /boot/initramfs-5.8.13-300.fc33.x86_64.img | grep btrfs

Of course use the actual kernel version you’re using to reproduce the problem. We don’t need full output from that, just need to know whether you have

-rw-r--r--   1 root     root          616 Sep 29 02:12 usr/lib/udev/rules.d/64-btrfs.rules

So we know you have 64-btrfs.rules but the question is whether it’s in the initrd. It is for me on F33, and I suspect it is for you on F32 but we need confirmation. Thanks.

2 Likes

Hello. I do not have access to the computer at the moment. I will get all this information to you on Friday. Thanks, Dan

Hi @chrismurphy

I have posted my reply to the systemd-devel mailing list:

https://lists.freedesktop.org/archives/systemd-devel/2020-October/045433.html

I will add a short summary of what happened for those following along here.

  • When booting with udev.log_priority=debug the btrfs volume mounted correctly at boot time!
  • Booting with the original unmodified grub configuration and the btrfs volume failed to mount at boot time as before.
  • Changing the way the hard drives are connected from 4 attached to the motherboard and 6 to the HBA, to all 10 connected to the HBA caused the btrfs volume to mount correctly at boot time.

So it seems that having the drives of the volume split between the HBA and motherboard is a contributing cause of this issue. And perhaps the “slowing effect” of having udev.log_priority=debug set causes all hard drives to be ready by the time a mount is attempted.

When running:

$ sudo lsinitrd /boot/initramfs-5.8.13-200.fc32.x86_64.img | grep btrfs
$ # No matches for "btrfs"

Where as say:

$ sudo lsinitrd /boot/initramfs-5.8.13-200.fc32.x86_64.img | grep ext4
-rwxr-xr-x   2 root     root       340632 Jan 31  2020 usr/sbin/fsck.ext4

Best wishes,

Dan

3 Likes

Further updates for those following along :smiley: :

So it has been suggested on the systemd-devel mailing list that this may be an issue with dracut.

https://lists.freedesktop.org/archives/systemd-devel/2020-October/045440.html

So I have reported it to dracut here:

Cheers, Dan

3 Likes

OK! I think the debugging is merely slowing startup down enough that all the devices appear in time for the mount option to succeed. And switching hardware around also changes their availability timing.

Ultimately the problem is that the btrfs “wait for all devices to be ready” udev rule is not in your initramfs. The Fedora initramfs is a “host-only” initramfs. It looks at the current configuration and only puts what’s needed in the initramfs. I don’t know the dracut logic for this use case, if it should always include this udev rule or if it should look in fstab for btrfs devices and conditionally add the udev rule, or if it’s expected the user should manually add this udev rule with a dracut configuration option.

I think the 2nd case is probably appropriate: if btrfs is found in either fstab or a native systemd mount unit, then dracut should create an initramfs that includes btrfs udev rules. Your fstab definitely has a btrfs entry. Question: was that entry in fstab already at the time the current kernel was installed? If yes, then dracut probably doesn’t have the necessary logic and we’ll need to take it up with dracut upstream. If no or you don’t remember you can just sudo dracut -f and it will be regenerated based on the current configuration; and now you can lsinitrd the new initramfs and grep for btrfs to see if it’s now included (or not).

Update: OK you figured all of this out and already filed a dracut bug; and I just realized it! So that’s awesome!

3 Likes