Silverblue upgrade woes

rant

#1

So, since the release is soon and since no one seemed to have horror stories on the upgrade, I decided to upgrade my laptop to F29. Spoiler: it doesn’t end well and would likely just negate all the good work done by Fedora improvement to get back mind share.

So, my laptop is a Lenovo T470s, paid by my employer. As people know, I had some issues to get partitioning as I wanted, I did faced SELinux issues in the past because the policy wasn’t ostree aware, etc. But in the end, I did managed to get what I wanted, and after some efforts, it was working kinda okayish. So I figured “Fedora upgrade usually work quite well, so I guess Silverblue would be the same”. Turn out that I am not always right, and when I am wrong, I am very wrong.

So my motivation to upgrade was mostly to get fwupd running. Every day, I do get the same notification on “there is a firmware upgrade”, and since that is my fix the latest version of Meltdown-like CPU fix, I kinda prefer to upgrade before someone find a way to abuse that from javascript. This bug was reported: https://github.com/hughsie/fwupd/issues/665

So after waiting for a upgrade on F28 that seems to never come, I decided to jump on F29, release is in 2 weeks.

So first issue, the upgrade doc was bad. I did send a patch for fixing that: https://docs.fedoraproject.org/en-US/fedora-silverblue/upgrades/

But surprise, the doc I sent is also wrong, because we renamed the branch between F28 and F29, and as far as I know, this wasn’t written anywhere I looked. And since this got merged anyway, that also mean that our docs verification is next to non existent, and/or no one is aware. The whole point of review is to review.

So I did the slow upgrade (because the mirror issue is still not fully fixed and I forgot to enable the mirror, hoping that maybe, someone will start to care about us non US folks by default), and ended with a rather old version of F29. The only way I could figure something was going on was the date at the reboot, which was somewhere middle of august. And so, after a bit of ostree mang pages reading, yes, the ref got changed to “silverblue”. Nice pointless breakage, but well, ok, let’s do again the slow upgrade and reboot.

Then it fill my hard drive. The fact that a basic desktop is able to fill a 25G of hard drive is a bit puzzling, but I guess all the component duplication of flatpak, of docker, of podman is taking a toll. So I do some cleanup, again from the commandline to get the upgrade running (because last time it went out of space, stuff broke in interesting way)

So it reboot fine, and so I start to fiddle around. So I start to get around, go on the web, watch videos, etc.

And I go see a training on tensorflow, and decide to just pop a container for that. Ctrl - R, look for my podman command:

$ podman run -it --privileged   --rm fedora:28 /bin/bash
ERRO[0001] could not find slirp4netns, the network namespace won't be configured: exec: "slirp4netns": executable file not found in $PATH 
[root@15dadde61f5f /]# 

“network not configured” is a interesting error, because this was working fine before, and as much as I appreciate the extra layer of security this give me, the container is close to useless with this.

The solution is now to either layer “slirp4netns” (but then, why isn’t it done by default ?), or use --network host. See https://github.com/containers/libpod/issues/1234 for more informations.

I would have reported it if I had a idea on where to do that, because there is again no doc to guide people for reporting and this whole project (Silverblue) is a messy hotchpotch that is just about to get messier.

So with the workaround found, I start to look again at my disk space, because that is still a issue. A few df later, the problem is quite easy, there is lots of space used in /var/lib/docker, and in /var/lib/flatpak, and /var/lib/containers. So I do remove some flatpak (because older SDK did tend to accumulate here, but I guess nowadays, people just have disk to spare), remove the duplicate Fedora containers from podman (because for some reasons, I did had some from docker hub, some from fedora registry), and start to look at cleaning the docker images.

Surprise, docker is no longer installed:

[misc@windgrace ~]$ rpm -qa | grep docker
[misc@windgrace ~]$ 

Great, so the upgrade would have broke my workflow if I was using docker, and left me with a 3G of blobby data I can’t read or manipulate easily. As my patience was running out low, I just nuked that. I already didn’t trust Docker for anything so I placed nothing important there anyway, but maybe “potential data lose” should be higher on the list of stuff we want to avoid.

Good, so my space is back, i can go back on my training. I download the zip file, open it, see that all files are world writable, see that Nautilus didn’t see that as a issue, and keep a note to investigate that later. And I start to double click to open a file.

Surprise, Gedit got removed. Double surprise, the UX to get it back is rather suboptimal. After double clicking on the file, I get a popup that say “do you want to search”, I say “yes”, I get a notification in the taskbar that say nothing was found, and again the same search popup in loop until I say “no”. Not really believing my eyes, I start to look for gedit, and see that indeed, this is not installed. Exalm on IRC point to https://discussion.fedoraproject.org/t/changes-in-fedora-silverblue-29/ , which started back in August. I did remember the discussion, but given the breakage, I would have think someone would have reverted and fixed the change, cause you know, we care about users. Seems not.

But no, there is no Gedit, and the workflow to install it is broken. Ok so I install, I do read my python source code, start my video.

Then I get a Telegram message from a friend, and wanting to see what time is in their timezone, I see that I lost the clock in the calendar widget, which is the straw that broke the camel’s back and what prompted me to write this rant. Again, the root cause is likely the fact that Gnome Weather got removed like Gedit.

But Silverblue is supposed to be the preferred version of Fedora Workstation, Fedora Workstation who is supposed to be aiming for devs. As someone part of the target market, I kinda feel the current state to be a joke.

Fedora Workstation website show for example “Built-in support for Docker” (https://getfedora.org/en/workstation/), but that’s wrong, since it got removed.

The website say “support developper”, but that is also wrong, since Gedit isn’t even installed, and seems no one tested the basic part of “running a python software”, otherwise the rootless bug I stumbled on would have been found.

I am kinda sure I am doing nothing special as a user this time, and yet, it took me just 1 day of using to see that basic features are broken since a few weeks, and that no one seems to fix them. Either no one is using it, or people just work around them. Even more shocking, the test days did had lots of participants: http://testdays.fedorainfracloud.org/events/47 , and yet, basic stuff went likely untested.

And the issue with upgrading my firmware, the one that prompted me to do the upgrade in the first place ?

Guess what, that’s still broken. I suspect I will finally have to report it somewhere, once I will find where to do it, and what broke in the first place, as I am sure this is related to Silverblue.

I am not against being the guinea pig, but part of the social contract of helping tests is also to get bug fixes, or at least, be able to workaround without much trouble. And Silverblue is not that, and this create more problems than it solve. So yeah, maybe I should just go back on regular Fedora, who offer almost all that Silverblue provides, without the endless stream of problems.


Not a symbolic link: boot/loader
#2

The ref got deleted but somehow got re-created: https://pagure.io/releng/issue/7761#comment-536273

This is actually a bit complicatied. slirp4netns is recommended by podman but it’s not included in the workstation comps group so it doesn’t get installed. We should really just build Silverblue from the Everything repo IMHO: PR for this for rawhide: https://pagure.io/pungi-fedora/pull-request/665

We have to have enough people that care about silverblue first. I myself am just a volunteer and I don’t even run silverblue yet (have been trying to get over there, but I’m still running f27 here so that let’s you know how good I am at finding time for moving my setup over). We need someone to lead the effort and track down and fix issues. Any help is welcome.


#3

Silverblue has made great strides in the F29 cycle. I love the concept and will most likely keep my Silverblue partition when it goes stable. I’ve been testing it with my normal workflow and there have been very few actual bugs and none of them weren’t also in Fedora Workstation.

The critical piece as far as I’m concerned is documentation. A moderately experienced Linux workstation user can jump between Fedora, Ubuntu, Antergos, Linux Mint, etc. with little mental gear-shifting. That’s not true with Silverblue. Once you get beyond what’s in the GNOME user experience and drop to the command line, it’s a whole different world.


#4

It certainly is at that. The concept is great, it certainly seems logical to want to update the system as separate from the applications used on it, and visa versa. Currently there are caveat’s though. and as was mentioned above regarding slirp4net, regarding what constitutes the base ostree at a bare minimum is not pinned down as of yet I think.

I too find the documentation trail a bit frustrating at times. Although I use my “Linux Box” as my daily primarily personal use computer, I also create user manuals for my customers on it, using TeXstudio, and do some Java development on it. Occasionally I get to fix some electronics, at which time I will use my MSO on it to trouble shoot the devices I am fixing. For me, all of those activities, including my personal use, are an integral part of my computing ecosystem. I do understand the “living at the near bleeding edge” life of using leading edge technologies, can sometimes lead to unforeseen consequences. I think Silverblue is in need of a “Champion” to lead the charge, and there have been some in the community who take the lead on pretty much any given topic surrounding it, which is heartening.


#5

I was going to wait for F29 GA to start mirroring F29 content on my mirror, but that doesn’t seem like the right idea. So I’ve enabled mirroring the fedora/29/x86_64/silverblue ref at https://faw.piratemirror.party/. I know it is a bit late for your troubles, but hopefully will be useful going forward.

It may be too late for F29, but we can expand the set of test cases to include things you’ve mentioned here.

Your concerns about the documentation problems, especially given the changes that users will face going from F28 -> F29, definitely need to be addressed. We can expand that upgrade doc to include a section specific to F29, since there are some radical changes (i.e. no docker) that users will need to be aware of. I’ve filed the following ticket to track this - https://pagure.io/teamsilverblue/issue/50

It sucks that you had such a bad experience, but this is valuable feedback to teach us that the project needs to do better.


#6

I think Silverblue 29 was one of the bigger changes we’re likely to make in terms of removing things by default.

Anyways I submitted:

https://pagure.io/fedora-docs/silverblue/pull-request/13


#7

FWIW, it’s slow on this side of the pond as well :). It unfortunately doesn’t help that we’re publishing composes every day and so don’t have any good static delta targets as a result. (Or if we really want daily updates for some reason, regen static deltas for the last 14 days or something?)

The following is totally not acceptable to ask of users and is provided purely for informational purposes but: if you have a local server, you could set up a systemd timer there to pull the latest and on your workstation, use the OSTree mirrorlist + contenturl support to make it try to fetch content objects (but not metadata & signatures) from your local server first and otherwise fall back to upstream.


#8

This is done in rawhide and in f29 now. slirp4netns should be in the next f29 SB compose.


#9

I currently get this:

> sudo rpm-ostree upgrade
error: No such branch 'fedora/29/x86_64/silverblue' in repository summary

Any ideas?

// Seems to be resolved now.


#10

fyi you don’t need sudo for rpm-ostree


#11
> ostree --repo=/ostree/repo pull atomic:fedora/29/x86_64/workstation 
error: Cannot write to repository: Permission denied

and

> sudo ostree --repo=/ostree/repo pull atomic:fedora/29/x86_64/workstation
error: No such branch 'fedora/29/x86_64/workstation' in repository summary

Today, right now.

I followed the Dual-Boot instruction in https://docs.fedoraproject.org/en-US/fedora-silverblue/installation-dual-boot/

  • it could be more explicit about 28/29/… replace with $version
  • must update to “silverblue” instead of “workstation” in ... pull atomic:fedora/...
  • Should rename Silverblue to Atomic-Workstation, everywhere :slight_smile:

I submitted a Pull Request to update the page (not to go back to Atomic-Workstation).


#12

Thanks for the PR!

Fedora 29 Silverblue introduced a lot of changes and our docs aren’t that great to begin with, so we need more folks like yourself who are willing to make PRs for these kinds of problems.


#13

Thanks,
Now I am stuck with

> sudo ostree admin os-init fedora                                       
error: Not a symbolic link: boot/loader

I have a UEFI enabled system. I did not found much reference to similar problem ??? https://www.reddit.com/r/Fedora/comments/9hv2ik/trying_to_convert_to_atomic_workstation_but/

I am using Grub2

> grub2-install --version
grub2-install (GRUB) 2.03

I have no /boot/grub2/grub.cfg file (just UEFI).


#14

What do you get if you run file /boot/loader and ls /boot/loader?


#15

Please follow up here Not a symbolic link: boot/loader

> file /boot/loader
/boot/loader: directory
> ls -l /boot/loader
total 4
drwx------. 2 root root 4096  4 oct.  23:18 entries

And if that can help …

ls -lah  /boot/ 
total 176M
dr-xr-xr-x.  8 root root 4,0K  9 nov.  19:50 .
dr-xr-xr-x. 21 root root  285  9 nov.  14:27 ..
-rw-r--r--.  1 root root 194K 21 oct.  02:07 config-4.18.16-200.fc28.x86_64
-rw-r--r--.  1 root root 194K 21 oct.  01:37 config-4.18.16-300.fc29.x86_64
-rw-r--r--.  1 root root 194K  5 nov.  19:16 config-4.18.17-300.fc29.x86_64
drwx------.  7 root root 4,0K  1 janv.  1970 efi
drwxr-xr-x.  2 root root 4,0K  8 mai    2018 extlinux
drwx------.  3 root root 4,0K  8 nov.  21:45 grub2
-rw-------.  1 root root  68M 26 oct.   2017 initramfs-0-rescue-3fd5fe8db6e34af78b9c6619e8503b3f.img
-rw-------.  1 root root  21M 24 oct.  10:17 initramfs-4.18.16-200.fc28.x86_64.img
-rw-------.  1 root root  21M  8 nov.  21:49 initramfs-4.18.16-300.fc29.x86_64.img
-rw-------.  1 root root  22M  9 nov.  19:50 initramfs-4.18.17-300.fc29.x86_64.img
drwxr-xr-x.  3 root root 4,0K  8 mai    2018 loader
drwx------.  2 root root  16K 26 oct.   2017 lost+found
-rw-------.  1 root root 4,0M 21 oct.  02:07 System.map-4.18.16-200.fc28.x86_64
-rw-------.  1 root root 4,0M 21 oct.  01:37 System.map-4.18.16-300.fc29.x86_64
-rw-------.  1 root root 4,0M  5 nov.  19:16 System.map-4.18.17-300.fc29.x86_64
-rwxr-xr-x.  1 root root 7,9M 26 oct.   2017 vmlinuz-0-rescue-3fd5fe8db6e34af78b9c6619e8503b3f
-rwxr-xr-x.  1 root root 8,3M 21 oct.  02:07 vmlinuz-4.18.16-200.fc28.x86_64
-rw-r--r--.  1 root root  168 21 oct.  02:04 .vmlinuz-4.18.16-200.fc28.x86_64.hmac
-rwxr-xr-x.  1 root root 8,3M 21 oct.  01:38 vmlinuz-4.18.16-300.fc29.x86_64
-rw-r--r--.  1 root root  168 21 oct.  01:34 .vmlinuz-4.18.16-300.fc29.x86_64.hmac
-rwxr-xr-x.  1 root root 8,3M  5 nov.  19:17 vmlinuz-4.18.17-300.fc29.x86_64
-rw-r--r--.  1 root root  168  5 nov.  19:12 .vmlinuz-4.18.17-300.fc29.x86_64.hmac
 ~  ls -lah  /boot/loader 
total 12K
drwxr-xr-x. 3 root root 4,0K  8 mai    2018 .
dr-xr-xr-x. 8 root root 4,0K  9 nov.  19:50 ..
drwx------. 2 root root 4,0K  4 oct.  23:18 entries
 ~  sudo ls -lah  /boot/loader/entries
total 8,0K
drwx------. 2 root root 4,0K  4 oct.  23:18 .
drwxr-xr-x. 3 root root 4,0K  8 mai    2018 ..