Btrfs push with Fedora - where does that leave Silverblue?

Hi jakfrost ,

I just went back an read your earlier post where you said:

… Grub2 would boot, because I could select the system and the logo with the spinner would come up, but then it would switch back to the text boot up, and hang there. …

I think if you see the spinner, then you are well past grub and grub is not the problem. So forget what I said earlier about the drivemap file and grub probing. I think that problem must be to do with Fedora, not grub.

  1. At installation start, select installation destination to set up drive(s) to install to.
  2. Select drive(s) to be used
  3. Select custom partitioning
  4. Change filesystem to btrfs instead of LVM
  5. Select auto allocation via the link
  6. Popup will appear within seconds.

Once the popup appears you have the option to debug or quit. If you quit the system gets rebooted. It was on those reboots that the USB was chosen as the boot device and it does get accessed since I see the busy light flash then go dark. But the system tries to boot from the primary boot device after this. I tried two different USB thumb drives, a Sandisk Cruizer 4GB and a Verbatim StoreNGo 8GB, both behave the same and I have to re-write the image to them, even though it is fine when mounted on another system. So I suspected there was something the install was writing to the USB and left it at that to proceed with the install using the advanced menu.

Well, that sucks in this case since I specifically wanted to reap that benefit from btrfs in that configuration, otherwise why give up 240GB of SSD storage? So how do you specify btrfs RAID1 during the installation? I couldn’t get the option to when I selected btrfs prior to RAID1, I must be missing something.

  1. Select drive(s) to be used

2x250G and 1x1T = so three drives selected? There must be more going on because I couldn’t get it to crash, but (a) I’m using a VM and (b) the drives are effectively zero’d. In /tmp/ following an installer error/crash like you describe, there will be an ‘anaconda-tb’ file which contains most everything needed by developers to figure out what went wrong.

Well, that sucks in this case since I specifically wanted to reap that benefit from btrfs in that configuration, otherwise why give up 240GB of SSD storage?

Let’s double check this. What do you get for:
sudo btrfs filesystem usage /
cat /proc/mdstat

So how do you specify btrfs RAID1 during the installation? I couldn’t get the option to when I selected btrfs prior to RAID1, I must be missing something.

You have to select Btrfs Volume as the Device Type. Check the two devices you want to use. Then select raid1 as the Raid Level. See 1st screenshot.

Based on your description “specify the software RAID setup in advance of selecting the filesystem” I think you created this:

If you’re confused, it’s because it’s confusing. Designing a user interface for complex things is hard.

2 Likes

output of btrfs command filesystem usage for / is
Overall:
Device size: 223.44GiB
Device allocated: 8.02GiB
Device unallocated: 215.42GiB
Device missing: 0.00B
Used: 4.73GiB
Free (estimated): 217.98GiB (min: 217.98GiB)
Data ratio: 1.00
Metadata ratio: 1.00
Global reserve: 15.25MiB (used: 0.00B)
Multiple profiles: no

Data,single: Size:7.01GiB, Used:4.46GiB (63.58%)
   /dev/md127	   7.01GiB

Metadata,single: Size:1.01GiB, Used:280.88MiB (27.22%)
   /dev/md127	   1.01GiB

System,single: Size:4.00MiB, Used:16.00KiB (0.39%)
   /dev/md127	   4.00MiB

Unallocated:
   /dev/md127	 215.42GiB

… and cat of /proc/mdstat

Personalities : [raid1] 
md127 : active raid1 sdb1[0] sdc1[1]
      234297344 blocks super 1.2 [2/2] [UU]
      bitmap: 0/2 pages [0KB], 65536KB chunk

unused devices: <none>

Ah yes I did follow the second path with the SW raid setup.

Prior to getting to the point of the popup errors when trying to allocate storage, I had successfully completed an install of Silverblue twice, my setup was the same drives, just in different order. SATA0=/dev/sda=SSD1(mnt /) SATA1=/dev/sdb=SDD2 (no mnt) SATA2=/dev/sdc=HDD (mnt /var). The spinning disk has the BiosBoot and /boot (ext4) partitions with /var as btrfs. After reboot from installs with this physical layout, I would get to the spinning logo screen prior to login then the screen would go back to the previous text display before the logo and the system would hang there.

I won’t argue with you there. I spent a good number of years designing and implementing UI’s on machines/lines/processes that were expected to be readily understandable and usable by operators who very likely would never get formal training on what it was they were making let alone on how to use the equipment to make it. So for those operators and the maintenance staff who had to keep things going, it was in my best interest to make a UI that was both functional and understandable. So yes, making a good UI is difficult.
I would like to try to get the SSD’s setup as intended originally and don’t mind doing another install session to try and get the bug to happen. I’ll document my steps.

2 Likes
Data,single: Size:7.01GiB, Used:4.46GiB (63.58%)
   /dev/md127	   7.01GiB

Metadata,single: Size:1.01GiB, Used:280.88MiB (27.22%)
   /dev/md127	   1.01GiB

System,single: Size:4.00MiB, Used:16.00KiB (0.39%)
   /dev/md127	   4.00MiB

Yes. On the one hand, df is too little information for btrfs raid, and btrfs filesystem usage is a firehose of information with lots of jargon.

‘single’ is a reference to Btrfs profiles: single, dup, raid0, raid1, raid10, raid5, raid6, raic1c3, raid1c4. And single profile means “single copy”. So there is one copy of data and metadata. It should say raid1 for all three. And there should be two device nodes per chunk type. Chunk type is: data, metadata, system. The convention used for Btrfs is definitely unusual. Btrfs raid1 is not a mirroring of the entire block device, but in chunks. Typically this is 1G chunks for data, and 256M chunks for metadata.

More concise: btrfs filesystem df

1 Like

I think Silverblue can be seen as doing a bit more than openSUSE’s BTRFS-based transactions. In particular, Silverblue is aiming at being immutable; I can have a rootfs that’s relatively clean, without a ton of concern for it getting overly messy and polluted over time. Rebasing is insanely easy, and I don’t have to be as concerned about weird package conflicts. BTRFS transactions can probably accomplish this with enough tooling, but I’d argue SB is the cleaner way.

As for Stratis, I think it’ll evolve to look a bit different from BTRFS. XFS’s structure and performance characteristics aren’t really the same, and Stratis might give better performance for a server workload. Now, for personal use? We’ll have to see, personally I’m a huge XFS fan so I’m still looking towards Stratis rather than BTRFS to see what comes up.

1 Like

I think it’s the same as what openSUSE microOS does, if I understand correctly, but microOS takes advantage of the Btrfs features, the OS base is a snapshot:https://en.opensuse.org/Portal:MicroOS/Design
Could Silverblue do the same thing with Btrfs?
I am curious to know an opinion on both methods.

1 Like

Hello @cmurf,
I decided to try to get RAID1 going on the two ssd’s as you had suggested. One thing I tried was the failure of autoallocating as btrfs while in custom, this time I received and error message from the installer stating the desired feature wasn’t working and to use the Blivet advanced custom menu. So I proceeded to the Blivet GUI and did the partitioning as originally thought of …
2 ssd’s in raid1 with a subvol mounted as /; 1 HDD with a bootbios part, a ext4 part mounted as /boot @ 1.1gb, and a swap of 64GB then a btrfs volume with a subvol mounted as /var.
The install proceeds as expected , a bit longer than usual, I get to reboot, and the reboot loops from the Fedora Logo loading with spinner then back to the tty view, then repeat continuously AFAICT. I suspect this is a bug. Any suggestions on how to get out of this looping would be appreciated. I can access the Grub menu and edit, goto command prompt, as expected. I think the install has indeed been done and it is not pointing to the ostree deploy.

this time I received and error message from the installer stating the desired feature wasn’t working

The installer goes to great lengths to preserve existing data, and that means parsing every single nuance on the disk, to make sure it understands the exact state before getting started. So it’s definitely possible for it to start out in a confused state, not easily reproduced. Pretty much my tactic with the installer is if it crashes, stops, spits out an error or otherwise doesn’t do what I expect - I just stop what I’m doing, gather all logs, and file a bug. Because there’s maybe only a 50/50 chance it gets fixed without a bug report.

the reboot loops from the Fedora Logo loading with spinner then back to the tty view

Sounds like the earlier mentioned bug, rootflags=subvol=root is missing from the kernel command line.

1 Like

I also noticed that it’s possible /etc/fstab is created in a way that will cause confusion. In my case things still boot but if the drive nodes were to ever be assigned differently during boot - which can be pseudo-random on some computers - then boot will intermittently fail.

Bug 1860602 - sometimes fstab references a /dev node instead of fs UUID

1 Like

I’ve been using F32 Silverblue with the default Btrfs partitioning for about a week now and so far so good. I went ahead and turned on all the testing repos before I did my first update, deleted all the preinstalled flatpaks and then reinstalled everything from the fedora-testing repo. Turned on the RPM Fusion free repos as well for ffmpeg and libdvdcss.

34 flatpaks from the fedora-testing repo and another 15 from the flathub repo to round out. Basically Gnome core apps / Fedora defaults plus things like LibreOffice, Steam, GIMP, etc. I installed the fedora-testing Firefox flatpak too - what I’m using now. It all works really well except the FF flatpak doesn’t seem to recognize any of my installed codecs. I’ve probably used FF, LibreOffice, and Steam the most. Played some Mass Effect and Half-Life.

I’m definitely excited now to see what Fedora can put together for the next release and going forward. Maybe we can help make it even better than ZFS.

I’ve heard…mixed results from microOS users, I don’t recall the exact trade-offs, but I remember Silverblue did do better in some ways (I know that’s not super helpful but maybe would give you a direction if you wanted to look into it on your own?).

2 Likes

OSTree is intentionally filesystem independent; all it relies on is plain old Unix hardlinks. It will use reflinks if available for /etc, but reflinks are also available with XFS and are enabled by default - this is the default situation with Fedora CoreOS, which uses XFS and (rpm-)ostree.

We might switch at some point to using reflinks for ostree’s “deployments” because I think they are more efficient than link() on BTRFS, but I need to check.

I don’t think maintaining multiple “backends” the same way the container stack is doing is worth it.

So basically OSTree on top of BTRFS is the same as OSTree with xfs/ext4, no advantages or disadvantages.

2 Likes

There is an article on Phoronix about the performance of the new Samsung 870 QVO SSD using F2FS, XFS, Btrfs and EXT4 on Ubuntu.
The conclusion is that in last place half the time was Btrfs.

Those tests are questionable. Have you read the comments about them following the article? Pretty much sum up the quality of the test metrics.

Hard links on Btrfs are quite efficient, it’s just two items in a leaf, DIR ITEM and DIR INDEX ,and an extra name in the inode. So it’s pretty cheap even compared to a reflink copy, which is a completely separate file with its own inode, and shared extents.

Cheaper than either would be to snapshot the entire tree, that’s maybe 180KiB total, and then update its contents with the changed files. Btrfs snapshots are read write by default which is kinda confusing at first. But whether hard links, reflinks, snapshots - they’re all way way way cheaper than full file copies. I don’t off hand see anything rpm-ostree needs to do different or special for Btrfs.

The Phoronix benchmarks are definitely curious and suspicious, as in, nearly two orders magnitude slower app launch times is just not OK if it were true. It’s not something I’ve seen even on a busy file system, but how busy does it have to get to trigger this behavior? I don’t know. But we’re looking into it. Maybe it’s just an oddity. Or maybe it could lead to some other strange problem that we do in fact care about.

2 Likes

Fedora-Silverblue-ostree-x86_64-Rawhide-20200731.n.0.iso contains the fix for bug 1753485

3 Likes

Hello @cmurf,
I did the install again with the noted Silverblue rawhide iso. Thanks to all BTW in getting this together.
As far as the install went, it worked, but I now had an issue in advanced blivet gui where if I deleted the btrfs subvolume and volume for the two 240GB SSD’s that were previously set up as a btrfs volume raid1 array, I couldn’t afterwards create any filesystem on the drives. I left the drives as they were and set their subvolume’s mount point to / and was able to continue the installation. This is my output from btrfs filesystem df /

Data, RAID1: total=6.00GiB, used=5.50GiB
System, RAID1: total=8.00MiB, used=16.00KiB
Metadata, RAID1: total=1.00GiB, used=284.00MiB
GlobalReserve, single: total=14.52MiB, used=0.00B

This seems to indicate the raid1 array is as I set it up previous to this install, although I couldn’t “see” the raid array in the installer just the devices, volume, and subvolume showing with no indication of raid1 anywhere.

As well, I tried the custom installer and had it autoallocate the drive layout. This resulted in the popup about an error and the options to report a bug or quit, or use the terminal to debug. After quitting, the USB media won’t boot at next or subsequent reboots on this system, while a different media burnt with the same ISO will. If you would like I can try to replicate with a VM and submit a bug report, I also still have that media intact for review. I did this after I had already installed with the advanced blivet gui used for partitioning, so when it rebooted due to selecting quit, even though I chose the USB HDD to boot from in the boot menu, my system booted into the install on my SSD’s.

In Custom partitioning, the installer will not allow a Btrfs file system to be destroyed while it has subvolumes on it. Each one has to be removed first. This is intentional.

I’m not certain of the behavior in Advanced-Custom. This UI is super layered, dense, and permits all kinds of things not allowed in the other interfaces. But if you’ve deleted everything, and still can’t create file systems - it might be confusing UI (file a bug as RFE: which means request for enhancement) or it might be a straight up bug (it should allow it but doesn’t).

Basically with Rawhide, all paths lead to bug reports. The best place to talk about that is #fedora-qa on IRC, and the test@ list.

1 Like

Why is it a problem that the “Advanced” option allows things that the other
interfaces don’t? Isn’t that the entire purpose of it?