I’ve not been having a great time of it lately with Silverblue. First, I noticed that firewalld was silently failing to start because of selinux-policy, so I opened this issue.
As you can see from the comments there, I ran restorecon -Rv / as advised, and now I can’t boot into either of my ostree options at startup. A bunch of stuff got relabeled (including lots of stuff in /sysroot), so I guess this could have something to do with it?
I also can’t seem to get into the emergency shell (with systemd.unit=emergency.target) – I get prompted for the root password for it alright, but then there are some error messages that show up for less than a second (some start with "sulogin: ", and then it says something about “cannot execute shell”, and then seems to give up and try to fall back to the default target (which doesn’t work).
Does anyone have any suggestions? If I can’t get it fixed by tomorrow evening I’ll need to reinstall in time for Monday morning since it’s my work machine.
Thanks for the tip @refi64! That indeed let me boot successfully, and then if I sudo setenforce 1 everything still works (so far). If I reboot again though, I hit the same problem as before, unless I add enforcing=0 again.
This means that I don’t need to reinstall for now, so huge thanks! Any idea what I should do next to fix the underlying issues? I’ll make note of this progress on the original bug also, in case the selinux people have any suggestions.
To be entirely honestly, I’m not 100% sure, maybe some things to try:
Given that enforcing works once booted, it seems that something very low level is what’s running into SELinux denials. Try running journalctl -b to view the logs for this boot, then search for audit (/audit → Enter) and go forward/backwards (n / N) until you find some sort of notable-looking SELinux denial, likely from very early boot.
If nothing comes up, maybe try a different version of the “big hammer” command (restorecon) from above: boot with autorelabel=1.
@walters thanks for the info. I wasn’t aware of that limitation (and I guess the selinux developer who told me to do it might not have been either).
I followed the instructions in that issue comment, and I can now boot without having to set enforcing=0 or anything like that. As mentioned there, all of my layered packages are now gone, but I’m having trouble installing them again – for any package I get an error similar to this:
I think I’ve gotten slightly further: rpm-ostree upgrade worked today, so I’ve now got a slightly different message when trying to install layered packages:
The reason I think this is further is because this doesn’t seem to be an error with the first dependency package of the package I’m actually trying to install, it’s the krb5-workstation package itself (and it’s about “Relabeling” rather than “Checking out packages” (although when I now look at this output and the output from last night, it seems “Checking out packages” hasn’t happened yet).
I figured (possibly erroneously, idk) that these failures are occurring because for some reason it expects those “objects” to exist, when they don’t, and that maybe it thinks that because they’re referenced from a different commit. Does that sound possibly correct?
To test this, I found an rpm with minimal dependencies that I’ve never installed before, to try to remove the possibility that it fails because some object doesn’t exist, and it worked!
$ rpm-ostree install deepin-wallpapers
Checking out tree 9477661... done
Enabled rpm-md repositories: updates fedora
rpm-md repo 'updates' (cached); generated: 2019-06-20T02:47:04Z
rpm-md repo 'fedora' (cached); generated: 2019-04-25T23:49:41Z
Importing rpm-md... done
Resolving dependencies... done
Will download: 1 package (64.5 MB)
Downloading from 'fedora'... done
Importing packages... done
Checking out packages... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
Freed: 374.2 MB (pkgcache branches: 4)
Added:
deepin-wallpapers-1.7.6-5.fc30.noarch
Run "systemctl reboot" to start a reboot
Now, I’ve still got the latest commit that have my layered packages “pinned”, in case I need to use it in a hurry before I can figure out how to layer on this new fixed commit. If I remove that, would that somehow fix this current issue, or is there something else I should do to fix?
As you said, the easiest path is probably to just prune all the other deployments; rpm-ostree will automatically delete all the cached pkgs. (You can check this by verifying whether ostree refs still shows rpmostree/ refs).
If you’re booted on the deployment without layered packages already, you can do: rpm-ostree cleanup -pr.
