Toolbox broken again (crun update in 31.20191112.0)

Toolbox in Silverblue version 31.20191111.0 (2019-11-11T00:35:35Z) 37c730f97d9c8c2bcb43f513c3aba72ce88b511d13a5f8ef86798f2788d28d5f works for me.

Upgrading to 31.20191112.0 (2019-11-12T00:37:19Z) e26beb4354049bd5b361d1a9c3fd047c92d51a04e16b4e3c2b925df46136f285 pulls in a new crun (and kernel) and toolbox no longer can run or create containers… again.

crun was upgraded from crun-0.10.2-1.fc31.x86_64 to crun-0.10.5-2.fc31; something in the commit breaks toolbox here.

Using Silverblue is generally great. What’s not great is the random breakage of toolbox and everything it depends upon (podman, crun/runc, etc.). This happened during pre-beta and beta versions of Fedora (which is annoying), but, sadly, it’s also happening post-release too.

Could we have some sort of automated testing which verifies that toolbox can create (and run) new containers and also run existing ones before the Silverblue version is bumped publicly please?

So many people using Silverblue depend on Flatpak and toolbox working. Randomly being without one or the other really throws a wrench in things. (Thankfully, I haven’t had an issue with Flatpak on Silverblue yet. Whew.)

3 Likes

There’s an issue filed for toolbox https://github.com/containers/toolbox/issues/330 (but it should probably be at crun’s issue tracker?)

This is not a good experience for new people who are trying out Silverblue. Even I, who have been using Silverblue for 3 years, don’t want to rollback every other week. These changes need to spend more time in testing before they are pushed to the tree.

5 Likes

There’s going to be a fix in crun. (It’s not in bodhi yet.)

Meanwhile, while we wait for the proper fix, there are three workarounds:

  1. Don’t update if you haven’t yet
  2. If you have updated, roll back
    Example: rpm-ostree rollback 37c730f97d9c8c2bcb43f513c3aba72ce88b511d13a5f8ef86798f2788d28d5f
  3. Apply the hotfix in a local copy of toolbox. (It has to be local, as Silverblue has a read-only filesystem.)
2 Likes

Thank you for your workarounds.

I was also bitten by this. My whole workflow depends an toolbox being available, so things like these render my computer unusable for me. I agree with you, random breakage of such an important core component should not happen.

But in defense of Silverblue: Applying a rollback until the fix is out was easy due to the innovative architecture.

1 Like

Yes this bug got me as as well. It is rather disruptive considering the amount of dependency i have on toolbox in silverblue.

thanks for the workarounds, will try those now

As far as I can tell rpm-ostree rollback only rolls you back to the previous update (if you are two updates further then where the problem first happened this will not fix it)
to actulally get to the BaseCommit: 37c730f97d9c8c2bcb43f513c3aba72ce88b511d13a5f8ef86798f2788d28d5f one has to execute the command:
rpm-ostree deploy 37c730f97d9c8c2bcb43f513c3aba72ce88b511d13a5f8ef86798f2788d28d5f

1 Like

However, how do you even know that the issue is the update to silverblue? I just get an error from toolbox “Is a directory: OCI runtime error.” Which makes me think the problem is in toolbox.

I switched to silverblue for f31 and I am a pretty experienced user and find the experience generally frustrating. Trying to pull some blog posts together outlining my experience and the countless workarounds.

/me goes to attempt a rollback and pray to get back the last 2+ hours head hitting wall some how.

(Please note, I actually am having this on “31.20191112.0 (2019-11-12T00:37:19Z)”)

1 Like

If you have the same issue, then the problem started when the package crun was updated, on which toolbox relies as mentioned here:

I encountered this bug too, right after upgrading from SB30 to SB31 and afterwards getting all updates. Unfortunately I couldn’t get back anymore as the two grub entries showed SB31 already do due updates. Then tried to start toolbox containers, which failed. Thought it had to do with the fact they were F30 base images and got some migrate hints. Set me off completely, eventually by trying to come across it all I messed up my existing F30 toolbox containers.

So glad I found this post! A lot of what I found and read in other topics related had to do with beta stage and all said to be fixed, however not a single toolbox would run on my system. This post clarifies the reason, finally (tried different things for hours and hours).

I just wanted to get back to you all to let you know that downgrading crun is the sole thing that I needed to get toolbox working under SB31 with latest updates (31.20191115.0 (2019-11-15T01:59:08Z).

Thank you @garret! Steps I performed on two SB31 hosts which made toolbox work for now (as work-around) until this gets fixed in crun (?):

This only applies if going back is no longer an option and all SB31 updates are already installed.

  1. Download crun-0.10.4-1.fc31 (https://kojipkgs.fedoraproject.org//packages/crun/0.10.4/1.fc31/x86_64/crun-0.10.4-1.fc31.x86_64.rpm)
  2. rpm-ostree override replace ~/Downloads/crun-0.10.4-1.fc31.x86_64.rpm
  3. systemctl reboot

Maybe this helps someone who’s in the position I was and uses the system as main desktop, cannot get any work done without toolbox.

4 Likes

Dear Stephan,
Thank you for your post. I was in the same situation and your solution worked perfectely for me.
Ale

Thanks @stephan your workaround fixed the issue for me too.

It would be really helpful if someone with who follows these things closely could post here when it’s safe to upgrade again (i.e. when an updated crun package hits the right repo?). I’ve just blindly updated with rpm-ostree upgrade until now because I assumed the base image would be well tested. Don’t want to sound crass since I love Silverblue (and rpm-ostree rollback worked beautifully), but to my workflow a broken crun is almost as bad as a broken OS.

There’s a new version of crun and it’s now in bodhi @ https://bodhi.fedoraproject.org/updates/FEDORA-2019-4b4957bbc6 — so hopefully this will land soon and we’ll have working containers in Silverblue once more (without workarounds).

2 Likes

Will the override automatically dissolve once the same version enters ‘updates’/‘testing-updates’? Or should we manually undo the override once the fixed package lands in the base image?

It looks like the new crun (0.10.6-1) is in stable and in the latest Silverblue ( 31.20191119.0 / 8c8a2da1c1071a2780abd37934c3884d09599357daa40c311510b5fea4a7cffa ).

When I did an rpm-ostree update, rpm-ostree seemed to figure out that the override is no longer needed and used the baked-in version of crun. (At least it looks that way in rpm-ostree status as there is no longer a ReplacedBasePackages section in my new deployment.)

Quick summary: rpm-ostree update and your system should be fine again without any workarounds. :tada:

Huge thanks to everyone involved! :+1:

3 Likes

Thanks so much for informing the rest of us! Updating now. :+1:

Thanks @garrett, much appreciated.

I have the same experience. The override simply dissolved. Everything’s working fine again :slight_smile:

Just to let you know, to me simply updating through GNOME Software didn’t replaced crun, but rpm-ostree override reset crun did it.