#9

If you have the same issue, then the problem started when the package crun was updated, on which toolbox relies as mentioned here:

#10

I encountered this bug too, right after upgrading from SB30 to SB31 and afterwards getting all updates. Unfortunately I couldn’t get back anymore as the two grub entries showed SB31 already do due updates. Then tried to start toolbox containers, which failed. Thought it had to do with the fact they were F30 base images and got some migrate hints. Set me off completely, eventually by trying to come across it all I messed up my existing F30 toolbox containers.

So glad I found this post! A lot of what I found and read in other topics related had to do with beta stage and all said to be fixed, however not a single toolbox would run on my system. This post clarifies the reason, finally (tried different things for hours and hours).

I just wanted to get back to you all to let you know that downgrading crun is the sole thing that I needed to get toolbox working under SB31 with latest updates (31.20191115.0 (2019-11-15T01:59:08Z).

Thank you @garret! Steps I performed on two SB31 hosts which made toolbox work for now (as work-around) until this gets fixed in crun (?):

This only applies if going back is no longer an option and all SB31 updates are already installed.

  1. Download crun-0.10.4-1.fc31 (https://kojipkgs.fedoraproject.org//packages/crun/0.10.4/1.fc31/x86_64/crun-0.10.4-1.fc31.x86_64.rpm)
  2. rpm-ostree override replace ~/Downloads/crun-0.10.4-1.fc31.x86_64.rpm
  3. systemctl reboot

Maybe this helps someone who’s in the position I was and uses the system as main desktop, cannot get any work done without toolbox.

4 Likes
#11

Dear Stephan,
Thank you for your post. I was in the same situation and your solution worked perfectely for me.
Ale

#12

Thanks @stephan your workaround fixed the issue for me too.

#13

It would be really helpful if someone with who follows these things closely could post here when it’s safe to upgrade again (i.e. when an updated crun package hits the right repo?). I’ve just blindly updated with rpm-ostree upgrade until now because I assumed the base image would be well tested. Don’t want to sound crass since I love Silverblue (and rpm-ostree rollback worked beautifully), but to my workflow a broken crun is almost as bad as a broken OS.

#14

There’s a new version of crun and it’s now in bodhi @ https://bodhi.fedoraproject.org/updates/FEDORA-2019-4b4957bbc6 — so hopefully this will land soon and we’ll have working containers in Silverblue once more (without workarounds).

2 Likes
Why do you use Silverblue?
#15

Will the override automatically dissolve once the same version enters ‘updates’/‘testing-updates’? Or should we manually undo the override once the fixed package lands in the base image?

#16

It looks like the new crun (0.10.6-1) is in stable and in the latest Silverblue ( 31.20191119.0 / 8c8a2da1c1071a2780abd37934c3884d09599357daa40c311510b5fea4a7cffa ).

When I did an rpm-ostree update, rpm-ostree seemed to figure out that the override is no longer needed and used the baked-in version of crun. (At least it looks that way in rpm-ostree status as there is no longer a ReplacedBasePackages section in my new deployment.)

Quick summary: rpm-ostree update and your system should be fine again without any workarounds. :tada:

Huge thanks to everyone involved! :+1:

3 Likes
#17

Thanks so much for informing the rest of us! Updating now. :+1:

#18

Thanks @garrett, much appreciated.

#19

I have the same experience. The override simply dissolved. Everything’s working fine again :slight_smile:

#20

Just to let you know, to me simply updating through GNOME Software didn’t replaced crun, but rpm-ostree override reset crun did it.

#21

Hmm, yeah. From first impression, it seems that rpm-ostree concludes that the manually upgraded crun-override is now ineffective. Consequently, one cannot reset the override. However, it seems that with a next upgrade, it might pin crun to the overridden version again.
So the reset needs to be performed upon the next upgrade. That makes it a bit nasty, as you’ll need to remember to do that then.

#22

Known bug, if you mention the full package name on the reset rather than the package name, it should reset.

1 Like
#23

Thanks @garrett everything is working fine after the update.

#24

I’m still having the same problem across all containers on F31 crun-0.10.6.

#25

@cryobry, is your F31 upgraded from F30? If so check this https://fedoraproject.org/wiki/Common_F31_bugs#Podman_fails_to_run_containers_on_upgraded_systems_.28due_to_use_of_runc_runtime_with_cgroups_v2.29

#26

I too had to perform rpm-ostree override reset crun like @blaster9678 mentioned before the latest crun package would be used instead of the earlier override. Latest crun package indeed fixes the issue.

#27

Yes, I’m using an upgraded F30->F31. podman info suggests I’m already using crun.

$ podman info
host:
  BuildahVersion: 1.11.3
  CgroupVersion: v2
  Conmon:
    package: conmon-2.0.2-1.fc31.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.2, commit: 186a550ba0866ce799d74006dab97969a2107979'
  Distribution:
    distribution: fedora
    version: "31"
  IDMappings:
    gidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  MemFree: 2333261824
  MemTotal: 16659312640
  OCIRuntime:
    name: crun
    package: crun-0.10.6-1.fc31.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.10.6
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  SwapFree: 17179865088
  SwapTotal: 17179865088
  arch: amd64
  cpus: 8
  eventlogger: journald
  hostname: fedora-laptop
  kernel: 5.3.11-300.fc31.x86_64
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: slirp4netns-0.4.0-20.1.dev.gitbbd6f25.fc31.x86_64
    Version: |-
      slirp4netns version 0.4.0-beta.3+dev
      commit: bbd6f25c70d5db2a1cd3bfb0416a8db99a75ed7e
  uptime: 11h 52m 9.73s (Approximately 0.46 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - quay.io
store:
  ConfigFile: /home/bryan/.config/containers/storage.conf
  ContainerStore:
    number: 5
  GraphDriverName: overlay
  GraphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-0.7-1.fc31.x86_64
      Version: |-
        fusermount3 version: 3.6.2
        fuse-overlayfs: version 0.7
        FUSE library version 3.6.2
        using FUSE kernel interface version 7.29
  GraphRoot: /home/bryan/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 21
  RunRoot: /run/user/1000
  VolumePath: /home/bryan/.local/share/containers/storage/volumes

And libpod.conf (It had already been regenerated as part of my troubleshooting):

$ cat ~/.config/containers/libpod.conf 
volume_path = "/home/bryan/.local/share/containers/storage/volumes"
image_default_transport = "docker://"
runtime = "crun"
runtime_supports_json = ["crun", "runc"]
runtime_supports_nocgroups = ["crun"]
conmon_path = ["/usr/libexec/podman/conmon", "/usr/local/libexec/podman/conmon", "/usr/local/lib/podman/conmon", "/usr/bin/conmon", "/usr/sbin/conmon", "/usr/local/bin/conmon", "/usr/local/sbin/conmon", "/run/current-system/sw/bin/conmon"]
conmon_env_vars = ["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"]
cgroup_manager = "systemd"
init_path = ""
static_dir = "/home/bryan/.local/share/containers/storage/libpod"
tmp_dir = "/run/user/1000/libpod/tmp"
max_log_size = -1
no_pivot_root = false
cni_config_dir = "/etc/cni/net.d/"
cni_plugin_dir = ["/usr/libexec/cni", "/usr/lib/cni", "/usr/local/lib/cni", "/opt/cni/bin"]
infra_image = "k8s.gcr.io/pause:3.1"
infra_command = "/pause"
enable_port_reservation = true
label = true
network_cmd_path = ""
num_locks = 2048
lock_type = "shm"
events_logger = "journald"
events_logfile_path = ""
detach_keys = "ctrl-p,ctrl-q"
SDNotify = false
cgroup_check = true
[runtimes]
crun = ["/usr/bin/crun", "/usr/local/bin/crun"]
runc = ["/usr/bin/runc", "/usr/sbin/runc", "/usr/local/bin/runc", "/usr/local/sbin/runc", "/sbin/runc", "/bin/runc", "/usr/lib/cri-o-runc/sbin/runc", "/run/current-system/sw/bin/runc"]
#28

I still had this issue with most recent FSB31 (crun v0.10.6-1) and my container was created in F31. I had to delete whole ~/.local/share/containers and re-create. Two questions:

  • Can we specify ostree to retain more than two images?
  • Are both old and new toolbox containers now included in CI, that would be really nice!
1 Like