Thanks for explaining that subvolume list doesn’t work on the block device. That’s much less useful and seems silly, but I understand.
But, the rest of your statement doesn’t make sense. subvolume list shows only one volume, called ext2_saved, which is NOT what is mounted.
# mount -v
...
/dev/mapper/fedora-root on /mnt type btrfs (rw,relatime,seclabel,space_cache,subvolid=5,subvol=/)
Why isn’t subvolume id 5 (named “/”) reported, too?
I’m not building a flat system, that’s not my goal. My goal is to convert this existing fedora 33 installation from ext4 to btrfs root.
I still don’t know why the system cannot boot off of the converted filesystem. But, it seems like grub2-mkconfig is broken and does not set the correct values, possibly because btrfs is broken and doesn’t report the subvolume I want to be come root.
Well, I even tried to add kernel parameters to help the kernel find root, but that also doesn’t work. I’m completely out of ideas. Something critical is missing and I have no way to find out what. boot fails, but everything I try works fine; no indication of what the problem could be.
I used serial redirect to take a look a the boot messages. The kernel seems quite happy and switch root seemed to succeed. However, journald keeps starting and failing, indefinitely. Here is the first sign of problems:
Starting Switch Root...
[ 12.384465] systemd-journald[192]: Received SIGTERM from PID 1 (systemd).
[ 12.505654] SELinux: Permission watch in class filesystem not defined in policy.
[ 12.506675] SELinux: Permission watch in class file not defined in policy.
[ 12.507576] SELinux: Permission watch_mount in class file not defined in policy.
[ 12.508530] SELinux: Permission watch_sb in class file not defined in policy.
[ 12.509444] SELinux: Permission watch_with_perm in class file not defined in policy.
[ OK ] Reached target Switch Root.
[ OK ] Finished Plymouth switch root service. [884/1554]
Starting Switch Root...
[ OK ] Stopped Hardware RNG Entropy Gatherer Daemon.
[ 12.384465] systemd-journald[192]: Received SIGTERM from PID 1 (systemd).
[ 12.505654] SELinux: Permission watch in class filesystem not defined in policy.
[ 12.506675] SELinux: Permission watch in class file not defined in policy.
...
[ 12.571377] systemd[1]: Successfully loaded SELinux policy in 139.337ms.
...
[ 19.371825] audit: type=1400 audit(1599518934.854:125): avc: denied { search } for pid=560 comm="systemd-modules" name="/" dev="dm-0" ino=
256 scontext=system_u:system_r:systemd_modules_load_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=dir permissive=0
[ 19.376468] audit: type=1400 audit(1599518934.854:126): avc: denied { search } for pid=560 comm="systemd-modules" name="/" dev="dm-0" ino=
256 scontext=system_u:system_r:systemd_modules_load_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=dir permissive=0
[ 19.376472] audit: type=1400 audit(1599518934.854:127): avc: denied { search } for pid=560 comm="systemd-modules" name="/" dev="dm-0" ino=
256 scontext=system_u:system_r:systemd_modules_load_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=dir permissive=0
[ 19.376475] audit: type=1400 audit(1599518934.854:128): avc: denied { search } for pid=560 comm="systemd-modules" name="/" dev="dm-0" ino=
256 scontext=system_u:system_r:systemd_modules_load_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=dir permissive=0
[ 19.376476] audit: type=1400 audit(1599518934.854:129): avc: denied { search } for pid=560 comm="systemd-modules" name="/" dev="dm-0" ino=
...
[DEPEND] Dependency failed for Flush Journal to Persistent[ 33.410490] systemd[1]: Failed to start Load Kernel Modules.
Storage.
[ 40.305310] audit: type=1130 audit(1599518957.995:180): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit
=systemd-modules-load comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
[ 40.305517] systemd[1]: Finished Remount Root and Kernel File Systems.
[FAILED] Failed to start Load Kernel Modules.
...
So, there are a large number of errors starting SELinux, but, ultimately, I think it worked. However, eventually, there are a large number of AVCs when systemd-modules starts. I don’t know if these are normal or not. But, systemd fails to successfully start “Load Kernel Modules”, so I think we can say boot has failed at this point.
In case a new reader missed it, I upgraded to F33 on this system prior to my last 5 attempts.
Could it be that SELinux policy doesn’t work for a converted root? Again, I’m not aware of a reason to expect that, but I’m certainly not an expert in SELinux.
A quick way to check if selinux is the issue would be to set selinux to disabled
(edit /etc/selinux/config and change the line with “SELINUX=enforcing” or “SELINUX=permissive” to “SELINUX=disabled”)
then reboot and see if there is any change.
If it boots then you can troubleshoot the selinux issue, if it still does not boot then I would leave selinux disabled until whatever other issue is resolved before I turned it back on.
Yes, for sure. That is a great suggestion, but I was wondering what I would do if it did work. I figured I would just see if auto-relabel would work…and it did!
I created ‘/.autorelabel’ using the install media and rebooted. Bingo.
$ mount -v | rg btrfs
/dev/mapper/fedora-root on / type btrfs (rw,relatime,seclabel,space_cache,subvolid=5,subvol=/)
Also, I remember now, this was mentioned in one of the conversion walkthroughs, but not one that I bookmarked, so I must not have thought it was very good, otherwise. I just forgot.
So, root cause: SELinux context must be updated after conversion. So add that to my post starting the thread. This can be done automatically on first boot after btrfs conversion (using /.autorelabel). But, I bet you could do it manually from chroot before you reboot.
SELinux relabel
My only lingering question pertains to the root subvolume name. I worry about this strange situation where the btrfs root subvolume is the default one that doesn’t even get reported by the ‘list’ subcommand. Should I try to change this (to /root) as is used by F33 default install?
I do not yet use btrfs so my post above was purely related to your thoughts about “relabel” and one of the means of troubleshooting whether selinix was causing the failure.
“/root” is the root users home directory. “/” is the root directory at the very top of the directory tree. I think you are misunderstanding the install syntax in that / is the root subvolume mount point, not /root.
I find it a bit confusing that they call it the root subvolume although it clearly is similar to the way LVM defines the name of the volume mounted at “/” as “/dev/VGname/root”. I am used to considering partitions and LVMs as volumes and will have to get used to the subvolume terminology used with btrfs.
btrfs certainly makes this nomenclature more confusing. So, I see why you misunderstood my meaning. btrfs gives subvoulumes both names and IDs. Fedora decided to name the subvolume that gets mounted at ‘/’ “root”. (Other distributions do it differently.) So, in btrfs, the subvolume tree will show “root” as leaf on subvolume branch ‘/’, i.e. “/root”. I guess the kernel devs call it “toplevel”, not ‘/’, so maybe that helps to keep subvolumes & filesystems separate. So, in Fedora, that string refers two completely different things, now, unfortunately. Perhaps I should have just said “root”, not “/root”. In any case, there is the default, root subvolume, called ‘/’ or toplevel, and a filesystem that gets mounted at ‘/’, but that is not necessarily the subvolume named ‘/’. On my system, it is, now; but, that’s not typical and not what you will get with a new F33 install. Does that make sense?
So, my question is, should I name the default subvolume “root” so that it looks like a default install. I ask because I didn’t have to fiddle with kernel boot parameters and it looks like F33 on btfs will require new boot params that help the kernel select the correct subvolume. Perhaps mkconfig will do this if I make this change and then re-run it. But, if it doesn’t, then I have to do some kludge and then I never know when it is safe to undo the kludge.
IMO, it is something you just have to get used to; it was confusing to me, too, the first couple of years I used btrfs. Now, I have learned to clearly think and say that I’m referring to the btrfs root subvolume (the top level of a btrfs filesystem) or the Linux root directory, which can be located in a btrfs root subvolume or a lower level subvolume.
Sorry, I can’t answer your main question about what you need to do to prepare for Fedora 33. But here is what I will probably do: backup all my stuff to an off-system drive, then fresh install Fedora 33 and see what it does. Then, restore needed data from the backups.