I have been running Silverblue for ~3 months now and it has been a mostly flawless experience, however I have run into my first major issue when trying to upgrade today:
To anyone with layered packages sitting on a deployment with libsolv v0.7.17: once this errata is pushed to stable and we have a new compose with the new rpm-ostree, you’ll need to be running either from a deployment without libsolv v0.7.17 or one with libsolv v0.7.17 and this rpm-ostree.
Concretely, this means that you either need to rollback to an older deployment if you still have one, or you can just rpm-ostree usroverlay && rpm -Uvh rpm-ostree-{,libs}-...rpm . And then rpm-ostree upgrade (and make sure it shows this errata in the diff before rebooting into it).
Finally, figured out a temporary, hacky, solution. libsolv is the culprit, and it or rpm-ostree will probably need fixes upstream. Here is the full set of commands that got me to a working deployment:
I have layered pkg’s and also was using libsolv 0.7.15 prior to doing an update yesterday which went well and resulted in libsolv 0.7.17 being installed. It appears the issue you are experiencing is potentially related to a kernel regression, which should see a fix imminently as there was a solution already for rawhide in the comments I read.
Currently using it now. I noticed it was replaced in my summary after the update completed but before reboot. Version 0.7.15 > 0.7.17, Sorry didn’t take a screenshot. I think if everyone was having this problem it would be a much more heavily viewed topic. I would hazard a guess it is somehow kernel related but to specific hardware potentially. From reading the links and subsequent ones you referenced above, I saw that it was something related to non-completion of a task during the rpm-ostree rebuild stage (where your layers get added) and further comments by the dev’s note the OOM killer and combined with the release notes on some AMD chips having memory leaks on F33 release, again kernel issue here. Of course, I haven’t dug deeply into it beyond that and am only offering an opinion at this point, not a solution.
Since you have overrode it now, it should stay that way through updates until you reset the override. So if everything is working for you now, then pinning would be redundant IMO. I would hazard a guess that this will be fixed ASAP.
Sorry, I misunderstood. I thought you were referring to ostree admin pin [OPTION…] INDEX. You’re welcome for any and all help I may be able to provide.
Had the same issue as mentioned here. I was hesitant with trying the fix suggested, because I would need to review commands I haven’t used before and understand if they had any impact on the system. Luckily I had an idea that worked.
Another quick fix is to use:
$ rpm-ostree rollback
Since you can’t upgrade after you hit this bug I m guessing that most if not all will rollback to a version that is not affected. After rebooting you can try upgrading, since in my case I didn’t see the offending package in the packages to be updated, I m guessing the offending package has been pulled off.
I did that, reverting my Silverblue to 33.20210202.0 (2021-02-02T02:52:11Z)
with libsolv-0.7.15-1.fc33 and rpm-ostree-2021.1-2.fc33 but rpm-ostreed was still crashing.
Then I tried usroverlay + rpm-ostreed restart with rpm-ostree-2021.1-3, still the same:
# rpm-ostree upgrade
2 metadata, 0 content objects fetched; 788 B transferred in 1 seconds; 0 bytes content written
Checking out tree 655b209... done
error: Bus owner changed, aborting. This likely means the daemon crashed; check logs with `journalctl -xe`.
In journal:
rpm-ostreed.service: Main process exited, code=killed, status=11/SEGV
interesting part of stack trace is
Stack trace of thread 45644: #0 0x00007f8bf9237a7f _ostree_loose_path (libostree-1.so.1 + 0x2fa7f) #1 0x00007f8bf9241f7f load_metadata_internal.isra.0 (libostree-1.so.1 + 0x39f7f) #2 0x00007f8bf924a375 ostree_repo_load_commit (libostree-1.so.1 + 0x42375) #3 0x000055d345ad00f4 rpmostree_pkgcache_find_pkg_header (rpm-ostree + 0x690f4) #4 0x000055d345b13f93 rpmostree_sysroot_upgrader_prep_layering (rpm-ostree + 0xacf93)
…
Is this worth reporting in rhbz or already known issue?
I haven’t seen this issue encountered before. I don’t see anything saying that this would be related to the libsolv issue, but I also can’t say I’m knowledgeable enough to say that for certain. Likewise, I would open a bug on RHBZ or upstream.