#21

I’m running into this issue as well…

output from rpm-ostree status 
$ sudo rpm-ostree status
State: idle
Deployments:
● ostree://fedora:fedora/33/x86_64/silverblue
                   Version: 33.20210204.0 (2021-02-04T01:41:19Z)
                BaseCommit: 39ac11c939acaa8cc7bb53635bd946b7c76ac74a805272ed1720b615b75dfcb4
              GPGSignature: Valid signature by 963A2BEB02009608FE67EA4249FD77499570FF31
           LayeredPackages: fedora-workstation-repositories fish foo2xqx golang
                            gstreamer1-plugin-openh264 gstreamer1-plugins-ugly-free hledger
                            myrepos ripgrep xclip
$ sudo rpm-ostree upgrade
2 metadata, 0 content objects fetched; 788 B transferred in 1 seconds; 0 bytes content written
Checking out tree 26b3558... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora updates-archive
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2020-08-25T19:10:34Z
rpm-md repo 'updates' (cached); generated: 2021-02-11T01:29:27Z
rpm-md repo 'fedora' (cached); generated: 2020-10-19T23:27:19Z
rpm-md repo 'updates-archive' (cached); generated: 2021-02-11T02:56:55Z
Importing rpm-md... done
Resolving dependencies... done
Checking out packages... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
error: Sanity-checking final rpmdb: Didn't find package 'gstreamer1-plugin-openh264-1.16.2-2.fc33.x86_64'

… but I apparently have incredibly bad luck, and discovered the sudo rpm-ostree cleanup -b command after version 33.20210204.0 broke my system, but before I knew there was a problem

… so when I try to roll back…

$ sudo rpm-ostree rollback
error: No rollback deployment found

… and when I try to deploy the version from the day before (i.e.: so I can upgrade from there to a working version), it fails the sanity check, similar to what happens when I sudo rpm-ostree upgrade from where I am currently…

output from rpm-ostree deploy 33.20210203.0 
$ rpm-ostree deploy 33.20210203.0
Resolving version '33.20210203.0'
1 metadata, 0 content objects fetched; 592 B transferred in 3 seconds; 0 bytes content written
⠒ Receiving objects: 98% (89/90) 1.4 MB/s 89.8 MB 
26 metadata, 64 content objects fetched; 88689 KiB transferred in 69 seconds; 12Receiving objects: 98% (89/90) 1.4 MB/s 89.8 MB... done
Checking out tree 8776ab1... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora updates-archive
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2020-08-25T19:10:34Z
rpm-md repo 'updates' (cached); generated: 2021-02-11T01:29:27Z
rpm-md repo 'fedora' (cached); generated: 2020-10-19T23:27:19Z
rpm-md repo 'updates-archive' (cached); generated: 2021-02-11T02:56:55Z
Importing rpm-md... done
Resolving dependencies... done
Checking out packages... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
error: Sanity-checking final rpmdb: Didn't find package 'gstreamer1-plugin-openh264-1.16.2-2.fc33.x86_64'

… is there a way I can rescue myself from this situation, i.e.: is there a way to deploy a version without performing a sanity check?

Reading some of the previously-linked articles, I found the following comment…

The nuclear fix for this is to:

  1. wait until https://bodhi.fedoraproject.org/updates/FEDORA-2021-9091468793 is in stable and in a compose
  2. drop all layerings (rpm-ostree reset)
  3. upgrade && reboot; this should be a pure OSTree operation, so no RPM should be involved
  4. re-add any wanted layerings/overrides

… but its not clear to me from the documentation for rpm-ostree whether rpm-ostree reset will just remove the LayeredPackages (i.e.: and leave my home directory alone), or reset everything (i.e.: including my home directory).

#22 
cd Downloads
wget https://kojipkgs.fedoraproject.org//packages/rpm-ostree/2021.1/3.fc33/x86_64/rpm-ostree-2021.1-3.fc33.x86_64.rpm && wget https://kojipkgs.fedoraproject.org//packages/rpm-ostree/2021.1/3.fc33/x86_64/rpm-ostree-libs-2021.1-3.fc33.x86_64.rpm && wget https://kojipkgs.fedoraproject.org//packages/libsolv/0.7.15/1.fc33/x86_64/libsolv-0.7.15-1.fc33.x86_64.rpm
sudo rpm-ostree usroverlay && sudo rpm -Uvh rpm-ostree-{,libs-}*.rpm
systemctl restart rpm-ostreed.service
rpm-ostree upgrade

Should still work

#23

No, as far as I know, rpm-ostree reset will not touch your home folder neither your etc folder. The command just drops all changes you made to base image. That is layered packages and overrides.

1 Like
#24

Hey there! I am a huge fan of Silverblue, that for i want it to be used and enjoyed, so i helped install it on many machines not only in my family. Now the system stoped updating. I would like to ask whether this is likely to be fixed without user interaction? If not, i think there should be information for all regular users.
Thanks everybody!! :upside_down_face:

#25

I booted in to an old version and upgraded from it and then booted up in the new one.

1 Like
#26

Another way, aside what was already mentioned by @wilburns , is to use rpm-ostree rollback to rollback to previous commit then upgrade from there after reboot. This preserves whatever state your layering was at during that rollback commit. It has no affect on /var/home/ or /etc.

1 Like
#27

I have just started getting this issue with my main silverblue installation, having had it with a raspberry pi before. I tried a rollback, but that didn’t work. I tried ostree admin fsck which reported that I had a corrupt file with an unexpected checksum. I had just opened a port to quickly test a development django site, and think I may have been hacked.
Is it possible to completely reinstall the rpm-ostree tree? There is a command ‘ostree admin ostree-init’ that looks like it can do just that, but I haven’t been able to find a tutorial or post about using it, just the man page.
Is there any way to see if this issue is the libsolv one?
My journalctl -xe give me this:

rpm-ostreed.service: Failed with result 'signal'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The unit rpm-ostreed.service has entered the 'failed' state with result 'signal'.
Feb 14 16:06:53 localhost.localdomain systemd[1]: rpm-ostreed.service: Consumed 20.284s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The unit rpm-ostreed.service completed and consumed the indicated resources.

I am going to open a new post for this issue as I think it may be different to the libsolv issue.

#28

Thanks for the quick responses! :slight_smile: If the error is only fixable by user interaction, then maybe there should be a post on social media to inform? Or is work ongoing for a fix without user interaction?

1 Like
#29

@jakfrost I was thinking something similar but completely forgot to post. Would a “mini”. magazine article be appropriate in this situation? People using silverblue should know it is not “complete” software, but it would still be nice to make solutions like this easier to access in this specific scenario.

1 Like
#30

Update on this…

Recap: I was on 33.20210204.0 and couldn’t update.

I saved the output from rpm-ostree status to a file, then backed everything up (Déjà Dup is pretty handy!), and ran sudo rpm-ostree reset and rebooted.

After the reboot, it had deleted all my LayeredPackages; but left my home folder untouched (as @aitvaras said). I then ran rpm-ostree upgrade and rebooted again when it was done.

After the reboot, I am now at 33.20210214.0 (i.e.: today’s release, i.e.: yay, I can update again!).

I rpm-ostree installed all the LayeredPackages I had before, and rebooted a third time, and everything’s back to normal; except I’m updated properly again and not getting any “Sanity-checking final rpmdb” errors, so I daresay it’s fixed.

Thanks everyone for your help!

2 Likes
#31

Well, Silverblue should still be considered a testing OS since it isn’t an official version of Fedora yet. This is truly the leading edge in Fedora for container based workflow usage, along with Fedora CoreOS. The idea behind both is the atomic system core, which includes the kernel and system configuration as well as other details important to the proper operation. This atomic part is immutable and therefore quite secure and stable, plus with SB and CoreOS you can layer packages on top of this atomic base of the OS to tailor it to your specific use case needs. And there are an ever growing field of applications being offered in flatpak format. And since they run isolated from the system bit’s in their own personal sandbox, again very secure. But truly, the concept also encompasses the constant push/pull relation ship of distro/application updates on your system, and mitigates it mostly.
So, from the POV of “How to use SB effectively?” automatic rollback when an error occurs with updating may be acceptable for some, but is certainly not for users like me. I want to know what failed and why so it can be fixed/mitigated/avoided in the future. Certainly a two word command at the command line rpm-ostree rollback is an easy work around for an issue that prevents me from updating.

1 Like
#32

The rpm-ostree rollback method worked for me without any issues. This particular problem would have been very hard to detect even with automated testing because it cannot be discovered until the next release is available and then it fails the update. I guess changes to packages that are critical to upgrade process should get careful review and impact consideration before being released to the tree. These critical packages should always be staged separately from any pending security fixes. Anyway, at least we can always rollback and try again later which is what I love about Silverblue.

1 Like
#33

Thank you very much for the detailed answers! In my enthusiasm I had somewhat overlooked that SB is still in an early phase. I think I was rather surprised, since I had no problems at all so far and even now it worked for me with rpm-ostree rollback quite wonderfully simple! Thank you so much! :slight_smile: