I can't boot into either my latest or previous ostree after running restorecon

I’ve not been having a great time of it lately with Silverblue. First, I noticed that firewalld was silently failing to start because of selinux-policy, so I opened this issue.

As you can see from the comments there, I ran restorecon -Rv / as advised, and now I can’t boot into either of my ostree options at startup. A bunch of stuff got relabeled (including lots of stuff in /sysroot), so I guess this could have something to do with it?

I also can’t seem to get into the emergency shell (with systemd.unit=emergency.target) – I get prompted for the root password for it alright, but then there are some error messages that show up for less than a second (some start with "sulogin: ", and then it says something about “cannot execute shell”, and then seems to give up and try to fall back to the default target (which doesn’t work).

Does anyone have any suggestions? If I can’t get it fixed by tomorrow evening I’ll need to reinstall in time for Monday morning since it’s my work machine.

(unrelated, but relevant for silverblue users, toolbox also seems to be broken by the latest podman update).

Thanks for any help you can provide,
Gerard.

You can try editing your boot options to add enforcing=0 as described here?

I believe the toolbox issue has already been reported upstream.

1 Like

Thanks for the tip @refi64! That indeed let me boot successfully, and then if I sudo setenforce 1 everything still works (so far). If I reboot again though, I hit the same problem as before, unless I add enforcing=0 again.

This means that I don’t need to reinstall for now, so huge thanks! Any idea what I should do next to fix the underlying issues? I’ll make note of this progress on the original bug also, in case the selinux people have any suggestions.

To be entirely honestly, I’m not 100% sure, maybe some things to try:

  • Given that enforcing works once booted, it seems that something very low level is what’s running into SELinux denials. Try running journalctl -b to view the logs for this boot, then search for audit (/audit -> Enter) and go forward/backwards (n / N) until you find some sort of notable-looking SELinux denial, likely from very early boot.
  • If nothing comes up, maybe try a different version of the “big hammer” command (restorecon) from above: boot with autorelabel=1.
  • Boot with
2 Likes

I ran restorecon -Rv /

Don’t ever do that on an ostree-managed system. See https://bugzilla.redhat.com/show_bug.cgi?id=1259018

As the bug said we’re eventually going to do some work to fix this, it’s just tricky.

It’s possible to re-pull all of the objects, see https://github.com/ostreedev/ostree/pull/345#issuecomment-262263824

1 Like

More useful direct link: https://github.com/ostreedev/ostree/issues/1265#issuecomment-484557615

1 Like

@walters thanks for the info. I wasn’t aware of that limitation (and I guess the selinux developer who told me to do it might not have been either).

I followed the instructions in that issue comment, and I can now boot without having to set enforcing=0 or anything like that. As mentioned there, all of my layered packages are now gone, but I’m having trouble installing them again – for any package I get an error similar to this:

$ rpm-ostree install krb5-workstation
Checking out tree 73634a1... done
Enabled rpm-md repositories: updates fedora
rpm-md repo 'updates' (cached); generated: 2019-06-19T01:37:55Z
rpm-md repo 'fedora' (cached); generated: 2019-04-25T23:49:41Z
Importing rpm-md... done
Resolving dependencies... done
Checking out packages... done
error: Checkout libkadm5-1.17-14.fc30.x86_64: Opening content object 9d52277ba88a0c429f4c3ed3fb33dfb20d369974ee0749dcee03828f1736c86a: Couldn't find file object '9d52277ba88a0c429f4c3ed3fb33dfb20d369974ee0749dcee03828f1736c86a'

Any idea what I’ve missed or other suggestions? Thanks again!

I think I’ve gotten slightly further: rpm-ostree upgrade worked today, so I’ve now got a slightly different message when trying to install layered packages:

$ rpm-ostree install krb5-workstation
Checking out tree 9477661... done
Enabled rpm-md repositories: updates fedora
rpm-md repo 'updates' (cached); generated: 2019-06-20T02:47:04Z
rpm-md repo 'fedora' (cached); generated: 2019-04-25T23:49:41Z
Importing rpm-md... done
Resolving dependencies... done
Relabeling... done
error: Relabeling krb5-workstation-1.17-14.fc30.x86_64: Opening content object a5fcb24c16122bc1ea87b7dbfc335446557f0bc9feca0249ce46a0c2b013f059: Couldn't find file object 'a5fcb24c16122bc1ea87b7dbfc335446557f0bc9feca0249ce46a0c2b013f059'

The reason I think this is further is because this doesn’t seem to be an error with the first dependency package of the package I’m actually trying to install, it’s the krb5-workstation package itself (and it’s about “Relabeling” rather than “Checking out packages” (although when I now look at this output and the output from last night, it seems “Checking out packages” hasn’t happened yet).

Further progress!

I figured (possibly erroneously, idk) that these failures are occurring because for some reason it expects those “objects” to exist, when they don’t, and that maybe it thinks that because they’re referenced from a different commit. Does that sound possibly correct?

To test this, I found an rpm with minimal dependencies that I’ve never installed before, to try to remove the possibility that it fails because some object doesn’t exist, and it worked!

$ rpm-ostree install deepin-wallpapers
Checking out tree 9477661... done
Enabled rpm-md repositories: updates fedora
rpm-md repo 'updates' (cached); generated: 2019-06-20T02:47:04Z
rpm-md repo 'fedora' (cached); generated: 2019-04-25T23:49:41Z
Importing rpm-md... done
Resolving dependencies... done
Will download: 1 package (64.5 MB)
Downloading from 'fedora'... done
Importing packages... done
Checking out packages... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
Freed: 374.2 MB (pkgcache branches: 4)
Added:
  deepin-wallpapers-1.7.6-5.fc30.noarch
Run "systemctl reboot" to start a reboot

Now, I’ve still got the latest commit that have my layered packages “pinned”, in case I need to use it in a hurry before I can figure out how to layer on this new fixed commit. If I remove that, would that somehow fix this current issue, or is there something else I should do to fix?

Yup, that’s likely what’s going on here. See: https://github.com/projectatomic/rpm-ostree/pull/1860.

As you said, the easiest path is probably to just prune all the other deployments; rpm-ostree will automatically delete all the cached pkgs. (You can check this by verifying whether ostree refs still shows rpmostree/ refs).

If you’re booted on the deployment without layered packages already, you can do: rpm-ostree cleanup -pr.

1 Like

Thanks a lot @jlebon, that worked and I think I’m back to a normal state for the first time in a while!

Now I just need to go back to the issue I was trying to get sorted in the first place! :smiley: