Cannot upgrade Silverblue

Sorry, I misunderstood. I thought you were referring to ostree admin pin [OPTION…] INDEX. You’re welcome for any and all help I may be able to provide.

1 Like

Infra issue: Issue #9634: revert libsolv for rpm-ostree based systems - fedora-infrastructure - Pagure.io
Upstream tracker: libsolv-0.7.17-1.fc33 failure tracker · Issue #2548 · coreos/rpm-ostree · GitHub

Had the same issue as mentioned here. I was hesitant with trying the fix suggested, because I would need to review commands I haven’t used before and understand if they had any impact on the system. Luckily I had an idea that worked.

Another quick fix is to use:

$ rpm-ostree rollback

Since you can’t upgrade after you hit this bug I m guessing that most if not all will rollback to a version that is not affected. After rebooting you can try upgrading, since in my case I didn’t see the offending package in the packages to be updated, I m guessing the offending package has been pulled off.

1 Like
$ rpm-ostree rollback

I did that, reverting my Silverblue to 33.20210202.0 (2021-02-02T02:52:11Z)
with libsolv-0.7.15-1.fc33 and rpm-ostree-2021.1-2.fc33 but rpm-ostreed was still crashing.
Then I tried usroverlay + rpm-ostreed restart with rpm-ostree-2021.1-3, still the same:

# rpm-ostree upgrade
2 metadata, 0 content objects fetched; 788 B transferred in 1 seconds; 0 bytes content written
Checking out tree 655b209... done
error: Bus owner changed, aborting. This likely means the daemon crashed; check logs with `journalctl -xe`.

In journal:
rpm-ostreed.service: Main process exited, code=killed, status=11/SEGV
interesting part of stack trace is
Stack trace of thread 45644:
#0 0x00007f8bf9237a7f _ostree_loose_path (libostree-1.so.1 + 0x2fa7f)
#1 0x00007f8bf9241f7f load_metadata_internal.isra.0 (libostree-1.so.1 + 0x39f7f)
#2 0x00007f8bf924a375 ostree_repo_load_commit (libostree-1.so.1 + 0x42375)
#3 0x000055d345ad00f4 rpmostree_pkgcache_find_pkg_header (rpm-ostree + 0x690f4)
#4 0x000055d345b13f93 rpmostree_sysroot_upgrader_prep_layering (rpm-ostree + 0xacf93)

Is this worth reporting in rhbz or already known issue?

1 Like

I haven’t seen this issue encountered before. I don’t see anything saying that this would be related to the libsolv issue, but I also can’t say I’m knowledgeable enough to say that for certain. Likewise, I would open a bug on RHBZ or upstream.

nm, already filed as 1925584 – [abrt] rpm-ostree: _ostree_loose_path(): rpm-ostree killed by SIGSEGV

1 Like

https://bodhi.fedoraproject.org/updates/FEDORA-2021-9091468793

WOOHOOOOOO!!! :partying_face:

1 Like

I’m running into this issue as well…

output from rpm-ostree status
$ sudo rpm-ostree status
State: idle
Deployments:
● ostree://fedora:fedora/33/x86_64/silverblue
                   Version: 33.20210204.0 (2021-02-04T01:41:19Z)
                BaseCommit: 39ac11c939acaa8cc7bb53635bd946b7c76ac74a805272ed1720b615b75dfcb4
              GPGSignature: Valid signature by 963A2BEB02009608FE67EA4249FD77499570FF31
           LayeredPackages: fedora-workstation-repositories fish foo2xqx golang
                            gstreamer1-plugin-openh264 gstreamer1-plugins-ugly-free hledger
                            myrepos ripgrep xclip
$ sudo rpm-ostree upgrade
2 metadata, 0 content objects fetched; 788 B transferred in 1 seconds; 0 bytes content written
Checking out tree 26b3558... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora updates-archive
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2020-08-25T19:10:34Z
rpm-md repo 'updates' (cached); generated: 2021-02-11T01:29:27Z
rpm-md repo 'fedora' (cached); generated: 2020-10-19T23:27:19Z
rpm-md repo 'updates-archive' (cached); generated: 2021-02-11T02:56:55Z
Importing rpm-md... done
Resolving dependencies... done
Checking out packages... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
error: Sanity-checking final rpmdb: Didn't find package 'gstreamer1-plugin-openh264-1.16.2-2.fc33.x86_64'

… but I apparently have incredibly bad luck, and discovered the sudo rpm-ostree cleanup -b command after version 33.20210204.0 broke my system, but before I knew there was a problem

… so when I try to roll back…

$ sudo rpm-ostree rollback
error: No rollback deployment found

… and when I try to deploy the version from the day before (i.e.: so I can upgrade from there to a working version), it fails the sanity check, similar to what happens when I sudo rpm-ostree upgrade from where I am currently…

output from rpm-ostree deploy 33.20210203.0
$ rpm-ostree deploy 33.20210203.0
Resolving version '33.20210203.0'
1 metadata, 0 content objects fetched; 592 B transferred in 3 seconds; 0 bytes content written
⠒ Receiving objects: 98% (89/90) 1.4 MB/s 89.8 MB 
26 metadata, 64 content objects fetched; 88689 KiB transferred in 69 seconds; 12Receiving objects: 98% (89/90) 1.4 MB/s 89.8 MB... done
Checking out tree 8776ab1... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora updates-archive
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2020-08-25T19:10:34Z
rpm-md repo 'updates' (cached); generated: 2021-02-11T01:29:27Z
rpm-md repo 'fedora' (cached); generated: 2020-10-19T23:27:19Z
rpm-md repo 'updates-archive' (cached); generated: 2021-02-11T02:56:55Z
Importing rpm-md... done
Resolving dependencies... done
Checking out packages... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
error: Sanity-checking final rpmdb: Didn't find package 'gstreamer1-plugin-openh264-1.16.2-2.fc33.x86_64'

… is there a way I can rescue myself from this situation, i.e.: is there a way to deploy a version without performing a sanity check?


Reading some of the previously-linked articles, I found the following comment

The nuclear fix for this is to:

  1. wait until https://bodhi.fedoraproject.org/updates/FEDORA-2021-9091468793 is in stable and in a compose
  2. drop all layerings (rpm-ostree reset)
  3. upgrade && reboot; this should be a pure OSTree operation, so no RPM should be involved
  4. re-add any wanted layerings/overrides

… but its not clear to me from the documentation for rpm-ostree whether rpm-ostree reset will just remove the LayeredPackages (i.e.: and leave my home directory alone), or reset everything (i.e.: including my home directory).

cd Downloads
wget https://kojipkgs.fedoraproject.org//packages/rpm-ostree/2021.1/3.fc33/x86_64/rpm-ostree-2021.1-3.fc33.x86_64.rpm && wget https://kojipkgs.fedoraproject.org//packages/rpm-ostree/2021.1/3.fc33/x86_64/rpm-ostree-libs-2021.1-3.fc33.x86_64.rpm && wget https://kojipkgs.fedoraproject.org//packages/libsolv/0.7.15/1.fc33/x86_64/libsolv-0.7.15-1.fc33.x86_64.rpm
sudo rpm-ostree usroverlay && sudo rpm -Uvh rpm-ostree-{,libs-}*.rpm
systemctl restart rpm-ostreed.service
rpm-ostree upgrade

Should still work

No, as far as I know, rpm-ostree reset will not touch your home folder neither your etc folder. The command just drops all changes you made to base image. That is layered packages and overrides.

1 Like

Hey there! I am a huge fan of Silverblue, that for i want it to be used and enjoyed, so i helped install it on many machines not only in my family. Now the system stoped updating. I would like to ask whether this is likely to be fixed without user interaction? If not, i think there should be information for all regular users.
Thanks everybody!! :upside_down_face:

I booted in to an old version and upgraded from it and then booted up in the new one.

1 Like

Another way, aside what was already mentioned by @wilburns , is to use rpm-ostree rollback to rollback to previous commit then upgrade from there after reboot. This preserves whatever state your layering was at during that rollback commit. It has no affect on /var/home/ or /etc.

1 Like

I have just started getting this issue with my main silverblue installation, having had it with a raspberry pi before. I tried a rollback, but that didn’t work. I tried ostree admin fsck which reported that I had a corrupt file with an unexpected checksum. I had just opened a port to quickly test a development django site, and think I may have been hacked.
Is it possible to completely reinstall the rpm-ostree tree? There is a command ‘ostree admin ostree-init’ that looks like it can do just that, but I haven’t been able to find a tutorial or post about using it, just the man page.
Is there any way to see if this issue is the libsolv one?
My journalctl -xe give me this:

rpm-ostreed.service: Failed with result 'signal'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The unit rpm-ostreed.service has entered the 'failed' state with result 'signal'.
Feb 14 16:06:53 localhost.localdomain systemd[1]: rpm-ostreed.service: Consumed 20.284s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The unit rpm-ostreed.service completed and consumed the indicated resources.

I am going to open a new post for this issue as I think it may be different to the libsolv issue.

Thanks for the quick responses! :slight_smile: If the error is only fixable by user interaction, then maybe there should be a post on social media to inform? Or is work ongoing for a fix without user interaction?

1 Like

@jakfrost I was thinking something similar but completely forgot to post. Would a “mini”. magazine article be appropriate in this situation? People using silverblue should know it is not “complete” software, but it would still be nice to make solutions like this easier to access in this specific scenario.

1 Like

Update on this…

Recap: I was on 33.20210204.0 and couldn’t update.

I saved the output from rpm-ostree status to a file, then backed everything up (Déjà Dup is pretty handy!), and ran sudo rpm-ostree reset and rebooted.

After the reboot, it had deleted all my LayeredPackages; but left my home folder untouched (as @aitvaras said). I then ran rpm-ostree upgrade and rebooted again when it was done.

After the reboot, I am now at 33.20210214.0 (i.e.: today’s release, i.e.: yay, I can update again!).

I rpm-ostree installed all the LayeredPackages I had before, and rebooted a third time, and everything’s back to normal; except I’m updated properly again and not getting any “Sanity-checking final rpmdb” errors, so I daresay it’s fixed.

Thanks everyone for your help!

2 Likes

Well, Silverblue should still be considered a testing OS since it isn’t an official version of Fedora yet. This is truly the leading edge in Fedora for container based workflow usage, along with Fedora CoreOS. The idea behind both is the atomic system core, which includes the kernel and system configuration as well as other details important to the proper operation. This atomic part is immutable and therefore quite secure and stable, plus with SB and CoreOS you can layer packages on top of this atomic base of the OS to tailor it to your specific use case needs. And there are an ever growing field of applications being offered in flatpak format. And since they run isolated from the system bit’s in their own personal sandbox, again very secure. But truly, the concept also encompasses the constant push/pull relation ship of distro/application updates on your system, and mitigates it mostly.
So, from the POV of “How to use SB effectively?” automatic rollback when an error occurs with updating may be acceptable for some, but is certainly not for users like me. I want to know what failed and why so it can be fixed/mitigated/avoided in the future. Certainly a two word command at the command line rpm-ostree rollback is an easy work around for an issue that prevents me from updating.

1 Like

The rpm-ostree rollback method worked for me without any issues. This particular problem would have been very hard to detect even with automated testing because it cannot be discovered until the next release is available and then it fails the update. I guess changes to packages that are critical to upgrade process should get careful review and impact consideration before being released to the tree. These critical packages should always be staged separately from any pending security fixes. Anyway, at least we can always rollback and try again later which is what I love about Silverblue.

1 Like

Thank you very much for the detailed answers! In my enthusiasm I had somewhat overlooked that SB is still in an early phase. I think I was rather surprised, since I had no problems at all so far and even now it worked for me with rpm-ostree rollback quite wonderfully simple! Thank you so much! :slight_smile: