Rpm-ostree update breaks toolbox (Fedora 40)

I have noticed for the last couple of days that rpm-ostree update has broken all my toolboxes (podman can still start containers).

This happend on Jun 12, and I had to roll back to get my system working again. Today (June 14th) I retried with the same result.

I get the following error:
Error: can only create exec sessions on running containers: container state improper

Rollback and all starts normally.

Has anyone else encountered this? If I retry, what logs can I provide to further troubleshoot this?

Iā€™m also having issues with toolbox on Fedora Silverblue 40. Iā€™m unable to enter containers, and I canā€™t create new ones either.

I get the following error message when I try to enter a container:

Error: failed to inspect entry point of container <name of container>

And this error message when I try to create a new one:

Error: failed to create container <name of container>

journalctl -p info,err | grep toolbox provide the output of this here in </> preformatted text.

Before you roll back again, or maybe fish through these logs you could :
journalctl -xe


If you wanted to get container specific logs you could :

List your running containers
podman ps -a

Inspect the containers for the container id :
podman inspect <container_id>

Then get those logs :
podman logs <container_id>

ok Iā€™ll come back with some of those logs. In the mean time maybe this can help. Please note this is not a huge problem for me as I can create a new toolbox and it works. But this is the second time in the last couple of months that this has happenedā€¦

bkelly@x1:~  $ toolbox -vv enter tools
DEBU Running as real user ID 1000                 
DEBU Resolved absolute path to the executable as /usr/bin/toolbox 
DEBU Running on a cgroups v2 host                 
DEBU Looking up sub-GID and sub-UID ranges for user bkelly 
DEBU TOOLBOX_PATH is /usr/bin/toolbox             
DEBU Migrating to newer Podman                    
DEBU Toolbox config directory is /var/home/bkelly/.config/toolbox 
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called version.PersistentPreRunE(podman --log-level debug version --format json) 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
INFO[0000] Using boltdb as database backend             
DEBU[0000] Initializing boltdb state at /var/home/bkelly/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/bkelly/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /var/home/bkelly/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/bkelly/.local/share/containers/storage/volumes 
DEBU[0000] Using transient store: false                 
DEBU[0000] Not configuring container store              
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime crun-vm initialization failed: no valid executable found for OCI runtime crun-vm: invalid argument 
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 37             
DEBU[0000] Called version.PersistentPostRunE(podman --log-level debug version --format json) 
DEBU[0000] Shutting down engines                        
DEBU Current Podman version is 5.1.0              
DEBU Creating runtime directory /run/user/1000/toolbox 
DEBU Old Podman version is 5.1.0                  
DEBU Migration not needed: Podman version 5.1.0 is unchanged 
DEBU Setting up configuration                     
DEBU Setting up configuration: file /var/home/bkelly/.config/containers/toolbox.conf not found 
DEBU Resolving container and image names          
DEBU Container: ''                                
DEBU Distribution (CLI): ''                       
DEBU Image (CLI): ''                              
DEBU Release (CLI): ''                            
DEBU Resolved container and image names           
DEBU Container: 'fedora-toolbox-40'               
DEBU Image: 'fedora-toolbox:40'                   
DEBU Release: '40'                                
DEBU Resolving container and image names          
DEBU Container: 'tools'                           
DEBU Distribution (CLI): ''                       
DEBU Image (CLI): ''                              
DEBU Release (CLI): ''                            
DEBU Resolved container and image names           
DEBU Container: 'tools'                           
DEBU Image: 'fedora-toolbox:40'                   
DEBU Release: '40'                                
DEBU Checking if container tools exists           
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called exists.PersistentPreRunE(podman --log-level debug container exists tools) 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
INFO[0000] Using boltdb as database backend             
DEBU[0000] Initializing boltdb state at /var/home/bkelly/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/bkelly/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /var/home/bkelly/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/bkelly/.local/share/containers/storage/volumes 
DEBU[0000] Using transient store: false                 
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: storage already configured with a mount-program 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that metacopy is not being used 
DEBU[0000] Cached value indicated that native-diff is usable 
DEBU[0000] backingFs=btrfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime crun-vm initialization failed: no valid executable found for OCI runtime crun-vm: invalid argument 
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument 
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument 
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 37             
DEBU[0000] Called exists.PersistentPostRunE(podman --log-level debug container exists tools) 
DEBU[0000] Shutting down engines                        
DEBU Inspecting mounts of container tools         
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called inspect.PersistentPreRunE(podman --log-level debug inspect --format json --type container tools) 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
INFO[0000] Using boltdb as database backend             
DEBU[0000] Initializing boltdb state at /var/home/bkelly/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/bkelly/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /var/home/bkelly/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/bkelly/.local/share/containers/storage/volumes 
DEBU[0000] Using transient store: false                 
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: storage already configured with a mount-program 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that metacopy is not being used 
DEBU[0000] Cached value indicated that native-diff is usable 
DEBU[0000] backingFs=btrfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument 
DEBU[0000] Configured OCI runtime crun-vm initialization failed: no valid executable found for OCI runtime crun-vm: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument 
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 37             
DEBU[0000] Looking up image "9aef520cae87299b9613159354f48936c203a9d3a0f4aee86645df53ab35e2bb" in local containers storage 
DEBU[0000] Trying "9aef520cae87299b9613159354f48936c203a9d3a0f4aee86645df53ab35e2bb" ... 
DEBU[0000] parsed reference into "[overlay@/var/home/bkelly/.local/share/containers/storage+/run/user/1000/containers]@9aef520cae87299b9613159354f48936c203a9d3a0f4aee86645df53ab35e2bb" 
DEBU[0000] Found image "9aef520cae87299b9613159354f48936c203a9d3a0f4aee86645df53ab35e2bb" as "9aef520cae87299b9613159354f48936c203a9d3a0f4aee86645df53ab35e2bb" in local containers storage 
DEBU[0000] Found image "9aef520cae87299b9613159354f48936c203a9d3a0f4aee86645df53ab35e2bb" as "9aef520cae87299b9613159354f48936c203a9d3a0f4aee86645df53ab35e2bb" in local containers storage ([overlay@/var/home/bkelly/.local/share/containers/storage+/run/user/1000/containers]@9aef520cae87299b9613159354f48936c203a9d3a0f4aee86645df53ab35e2bb) 
DEBU[0000] Called inspect.PersistentPostRunE(podman --log-level debug inspect --format json --type container tools) 
DEBU[0000] Shutting down engines                        
DEBU Starting container tools                     
Error: failed to start container tools

1 Like

I did PR#526: common: Keep fuse-overlayfs for compatibility - workstation-ostree-config - Pagure.io yesterday.

Can you try again with the latest update?

1 Like

Yep that works! But does that suggest that there is some legacy setting or something in my toolbox?

It looks like your rootless podman configuration uses fuse-overlayfs and that may going away in Fedora 41. See: Dependency on `fuse-overlayfs` in `containers-common` moved to `Suggests` Ā· Issue #1749 Ā· coreos/fedora-coreos-tracker Ā· GitHub

You can look at your current config with:

$ podman system info

Resetting all user podman settings should set things back to the new defaults:

# Warning: This will remove all containers
$ podman system reset

Thanks for taking care of this, @siosm ! I made an attempt to upstream this:

ā€¦ because I keep coming across reports of this in the wild. I suppose the hairy part is to actually detect the situation in Toolbx and offer users a migration path.

1 Like

Upstream Toolbx issue: Old containers don't start when fuse-overlayfs is missing Ā· Issue #1512 Ā· containers/toolbox Ā· GitHub

1 Like