Boot partition of Silverblue is without space

felipehw · 2018-12-06T13:55:09.980Z

My boot partition runs out of space. I don’t know what files i must erase for retrieve space. In vanilla Fedora this never happens before.

[dread@localhost ~]$ df -h
Sist. Arq.               Tam. Usado Disp. Uso% Montado em
devtmpfs                 1,5G     0  1,5G   0% /dev
tmpfs                    1,5G     0  1,5G   0% /dev/shm
tmpfs                    1,5G  1,4M  1,5G   1% /run
tmpfs                    1,5G     0  1,5G   0% /sys/fs/cgroup
/dev/mapper/fedora-root   24G   15G  7,7G  66% /sysroot
tmpfs                    1,5G   92K  1,5G   1% /tmp
/dev/sda1                976M  942M     0 100% /boot
tmpfs                    301M  4,6M  297M   2% /run/user/1000
[dread@localhost ~]$ sudo du -hs /boot/*
13M	/boot/grub2
0	/boot/loader
24K	/boot/loader.0
16K	/boot/lost+found
927M	/boot/ostree
[dread@localhost ~]$ sudo du -hs /boot/ostree/*
59M	/boot/ostree/fedora-workstation-0694ccf3bd13323f79a771ab9fb356e81dcffd09512fc22649cbc49cdda7ceb8
59M	/boot/ostree/fedora-workstation-07e2fcf43d85b9518452ac52b393073189f34265d2d162cde6887ed1ec3f08f9
59M	/boot/ostree/fedora-workstation-1c6885c4a4abbef0fbf6fff765f1638bf01cd38f1b86ceffde3f4cb00cdd677e
59M	/boot/ostree/fedora-workstation-3f4d9a60a740a3d44af75e2630afa80ff1d25343673d9efbbd8254403d6aebf9
59M	/boot/ostree/fedora-workstation-5ca962965e58225ba2696b57a531596dc793ecd5f4a9a239c5ba4a4adf0de06b
57M	/boot/ostree/fedora-workstation-80c5ffdb201d3ee9b6f15359c51e70becfb8655e90f720920aa4e6a2170bcc35
58M	/boot/ostree/fedora-workstation-812ad5ed714b0b011829ed77e2430a933f654451ee563be0b7f09de271c292e6
57M	/boot/ostree/fedora-workstation-a1d9d48bc5e16ade2624d938fd0fc52f0bce41c562278701d70976091c50d4de
57M	/boot/ostree/fedora-workstation-b0d57a28ce40ed6599deb78e3f58121e74fc9645638f222f2ed6b478d843a0fc
58M	/boot/ostree/fedora-workstation-b1fb5c5b63e40094eeb0c989c3c7f88263f35d1cdf9b64fbe63e1e851a0706cc
60M	/boot/ostree/fedora-workstation-b7bf80ee65bd9170fe1e6cd03c9cd022ba1cd4465e1a8b208c74112e6da32dc9
57M	/boot/ostree/fedora-workstation-bbf30f8ea88d7f46cee22d6c7a9de098e6f47af387603b1b56401632ead8cfaa
59M	/boot/ostree/fedora-workstation-d897872924e8a29d84896601805c61a09c532c1e63a43e03ab952a7fe4faf531
58M	/boot/ostree/fedora-workstation-edd311aebc1fee1929e617530ca7f46cb57218bf1f761b5b7369d8228b8652ef
59M	/boot/ostree/fedora-workstation-f1caf60fe2e9fff5113c38ff2ed48811dd51d3d5fc5d2457e793101c54b3b384
58M	/boot/ostree/fedora-workstation-f876fd5eda2acb30c13946732c21513e6d1ff1c56a8d7f786e6421f05cb1fd52
[dread@localhost ~]$ rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
● ostree://fedora-workstation:fedora/rawhide/x86_64/silverblue
                   Version: Rawhide.20181205.n.0 (2018-12-05T07:07:18Z)
                    Commit: 32ebe212fe6bb4a624c4005d4df6731fad6361119e08ed5c3c135fa9b73453c5
              GPGSignature: Valid signature by F1D8EC98F241AAF20DF69420EF3C111FCFC659B9

  ostree://fedora-workstation:fedora/rawhide/x86_64/silverblue
                   Version: Rawhide.20181204.n.0 (2018-12-04T07:16:17Z)
                    Commit: 28aa16e487040fac06742e361e657526ee8157944d131c445ecc78587c1ea3a7
              GPGSignature: Valid signature by F1D8EC98F241AAF20DF69420EF3C111FCFC659B9
[dread@localhost ~]$

walters · 2018-12-06T14:30:38.953Z

Hmm. That is strange. What does rpm-ostree cleanup say?

Are there any errors in journalctl -b -u rpm-ostreed? What’s your /boot partition setup (cat /etc/fstab) ?

miabbott · 2018-12-06T14:31:22.432Z

Do you have a number of deployments pinned?

I have three deployments on my host and three entries under /boot/ostree. They correspond to entries in /boot/loader.1/entries:

# rpm-ostree status
State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.service: last run failed
Deployments:
● ostree://fedora-workstation:fedora/29/x86_64/silverblue
                   Version: 29.20181130.0 (2018-11-30T00:57:38Z)
                BaseCommit: c97eb640ffbe5053f2c8be93e2440deda9bb60aac9bf9b977b2e0092ac738b19
              GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4
           LayeredPackages: chrome-gnome-shell compat-ffmpeg28 ffmpeg-libs krb5-workstation libselinux-python libvirt-client origin-clients tmux vagrant-libvirt vim-enhanced virt-install virt-manager

  ostree://fedora-workstation:fedora/29/x86_64/silverblue
                   Version: 29.20181119.0 (2018-11-19T00:52:11Z)
                BaseCommit: 8b984a62c6367e805c8ca37fe9184fc682260a156be888986db89787db81f156
              GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4
           LayeredPackages: chrome-gnome-shell compat-ffmpeg28 ffmpeg-libs krb5-workstation libselinux-python libvirt-client origin-clients tmux vagrant-libvirt vim-enhanced virt-install virt-manager

  ostree://fedora-workstation:fedora/28/x86_64/workstation
                   Version: 28.20181020.0 (2018-10-20T23:39:45Z)
                BaseCommit: eadacb28fb227b2aaf2438959110c7c5a75d8b743587ca3db1a55c8c8daa06c7
              GPGSignature: Valid signature by 128CF232A9371991C8A65695E08E7E629DB62FB1
           LayeredPackages: chrome-gnome-shell compat-ffmpeg28 ffmpeg-libs krb5-workstation libselinux-python libvirt-client origin-clients tmux vagrant-libvirt vim-enhanced virt-install virt-manager
                    Pinned: yes

# ls /boot/ostree/
fedora-workstation-2752ba690e9c41f9345dcf08b66c95fa83ecea315ff885d42b6f9a8a19603400  fedora-workstation-d1e29c95fcddb948b6e49be3bd2b4884d935ec18a8bb18fb43b06d0cee7c125b
fedora-workstation-69eb80898af9f89253e9ff33dfa3a175de7108dd6bb6b1cf4b73fd6672510ee3

# ls /boot/ostree/ | xargs -I {} grep {} /boot/loader/entries/*
/boot/loader/entries/ostree-2-fedora-workstation.conf:initrd /ostree/fedora-workstation-2752ba690e9c41f9345dcf08b66c95fa83ecea315ff885d42b6f9a8a19603400/initramfs-4.19.2-300.fc29.x86_64.img
/boot/loader/entries/ostree-2-fedora-workstation.conf:linux /ostree/fedora-workstation-2752ba690e9c41f9345dcf08b66c95fa83ecea315ff885d42b6f9a8a19603400/vmlinuz-4.19.2-300.fc29.x86_64
/boot/loader/entries/ostree-3-fedora-workstation.conf:initrd /ostree/fedora-workstation-69eb80898af9f89253e9ff33dfa3a175de7108dd6bb6b1cf4b73fd6672510ee3/initramfs-4.19.4-300.fc29.x86_64.img
/boot/loader/entries/ostree-3-fedora-workstation.conf:linux /ostree/fedora-workstation-69eb80898af9f89253e9ff33dfa3a175de7108dd6bb6b1cf4b73fd6672510ee3/vmlinuz-4.19.4-300.fc29.x86_64
/boot/loader/entries/ostree-1-fedora-workstation.conf:initrd /ostree/fedora-workstation-d1e29c95fcddb948b6e49be3bd2b4884d935ec18a8bb18fb43b06d0cee7c125b/initramfs-4.18.14-200.fc28.x86_64.img
/boot/loader/entries/ostree-1-fedora-workstation.conf:linux /ostree/fedora-workstation-d1e29c95fcddb948b6e49be3bd2b4884d935ec18a8bb18fb43b06d0cee7c125b/vmlinuz-4.18.14-200.fc28.x86_64

felipehw · 2018-12-06T16:42:09.174Z

My partition setup (is a default installation):

[dread@localhost ~]$ cat /etc/fstab 

#
# /etc/fstab
# Created by anaconda on Sat Oct 20 11:01:56 2018
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
/dev/mapper/fedora-root /                       ext4    defaults        1 1
UUID=26e1b093-fbbd-49f5-a363-5f332328bf17 /boot                   ext4    defaults        1 2
/dev/mapper/fedora-swap swap                    swap    defaults        0 0

felipehw · 2018-12-06T16:46:12.393Z

I try the rpm-ostree cleanup commands now. They don’t resolve the problem but show to me the same error of rpm-ostree upgrade (i forget of speaking about this errormsg). I believe the problem is here.
Let’s see:

read@localhost ~]$ sudo rpm-ostree cleanup -b
error: syscore cleanup: Cleaning deployments: unlinkat: Operation not permitted
dread@localhost ~]$ sudo rpm-ostree cleanup -p
Deployments unchanged.
[dread@localhost ~]$ sudo rpm-ostree cleanup -r
Transaction complete; bootconfig swap: yes; deployment count change: -1
error: syscore cleanup: Cleaning deployments: unlinkat: Operation not permitted
[dread@localhost ~]$ sudo rpm-ostree cleanup -m
[dread@localhost ~]$

How you can see, the problem is this message: unlinkat: Operation not permitted.

felipehw · 2018-12-06T16:51:11.219Z

I have before rpm-ostree cleanup only 2 deployments, and now i have only 1.
But there are many more entries under /boot/ostree.
I believe the problem is related to the unlinkat: Operation not permitted error msg when i use the rpm-ostree commands.

Some data:

[dread@localhost ~]$ rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
● ostree://fedora-workstation:fedora/rawhide/x86_64/silverblue
                   Version: Rawhide.20181205.n.0 (2018-12-05T07:07:18Z)
                    Commit: 32ebe212fe6bb4a624c4005d4df6731fad6361119e08ed5c3c135fa9b73453c5
              GPGSignature: Valid signature by F1D8EC98F241AAF20DF69420EF3C111FCFC659B9

[dread@localhost ~]$ sudo ls /boot/ostree/
fedora-workstation-0694ccf3bd13323f79a771ab9fb356e81dcffd09512fc22649cbc49cdda7ceb8
fedora-workstation-07e2fcf43d85b9518452ac52b393073189f34265d2d162cde6887ed1ec3f08f9
fedora-workstation-1c6885c4a4abbef0fbf6fff765f1638bf01cd38f1b86ceffde3f4cb00cdd677e
fedora-workstation-3f4d9a60a740a3d44af75e2630afa80ff1d25343673d9efbbd8254403d6aebf9
fedora-workstation-5ca962965e58225ba2696b57a531596dc793ecd5f4a9a239c5ba4a4adf0de06b
fedora-workstation-80c5ffdb201d3ee9b6f15359c51e70becfb8655e90f720920aa4e6a2170bcc35
fedora-workstation-812ad5ed714b0b011829ed77e2430a933f654451ee563be0b7f09de271c292e6
fedora-workstation-a1d9d48bc5e16ade2624d938fd0fc52f0bce41c562278701d70976091c50d4de
fedora-workstation-b0d57a28ce40ed6599deb78e3f58121e74fc9645638f222f2ed6b478d843a0fc
fedora-workstation-b1fb5c5b63e40094eeb0c989c3c7f88263f35d1cdf9b64fbe63e1e851a0706cc
fedora-workstation-b7bf80ee65bd9170fe1e6cd03c9cd022ba1cd4465e1a8b208c74112e6da32dc9
fedora-workstation-bbf30f8ea88d7f46cee22d6c7a9de098e6f47af387603b1b56401632ead8cfaa
fedora-workstation-d897872924e8a29d84896601805c61a09c532c1e63a43e03ab952a7fe4faf531
fedora-workstation-edd311aebc1fee1929e617530ca7f46cb57218bf1f761b5b7369d8228b8652ef
fedora-workstation-f1caf60fe2e9fff5113c38ff2ed48811dd51d3d5fc5d2457e793101c54b3b384
fedora-workstation-f876fd5eda2acb30c13946732c21513e6d1ff1c56a8d7f786e6421f05cb1fd52

[dread@localhost ~]$ sudo ls /boot/ostree/ | xargs -I {} grep {} /boot/loader/entries/*
initrd /ostree/fedora-workstation-b7bf80ee65bd9170fe1e6cd03c9cd022ba1cd4465e1a8b208c74112e6da32dc9/initramfs-4.20.0-0.rc5.git1.1.fc30.x86_64.img
linux /ostree/fedora-workstation-b7bf80ee65bd9170fe1e6cd03c9cd022ba1cd4465e1a8b208c74112e6da32dc9/vmlinuz-4.20.0-0.rc5.git1.1.fc30.x86_64

felipehw · 2018-12-06T16:52:46.378Z

More data:

[dread@localhost ~]$ cat /etc/fstab journalctl -b -u rpm-ostreed
-- Logs begin at Sat 2018-10-20 11:06:53 -03, end at Thu 2018-12-06 14:35:58 -02. --
dez 06 14:25:08 localhost.localdomain systemd[1]: Starting RPM-OSTree System Management Daemon...
dez 06 14:25:08 localhost.localdomain rpm-ostree[2036]: Reading config file '/etc/rpm-ostreed.conf'
dez 06 14:25:08 localhost.localdomain rpm-ostree[2036]: In idle state; will auto-exit in 60 seconds
dez 06 14:25:08 localhost.localdomain systemd[1]: Started RPM-OSTree System Management Daemon.
dez 06 14:25:08 localhost.localdomain rpm-ostree[2036]: Allowing active client :1.314 (uid 1000)
dez 06 14:25:08 localhost.localdomain rpm-ostree[2036]: client(id:gnome-software dbus:1.314 unit:session-2.scope uid:1000) added; new total=1
dez 06 14:30:36 localhost.localdomain rpm-ostree[2036]: client(id:cli dbus:1.396 unit:gnome-terminal-server.service uid:0) added; new total=2
dez 06 14:30:37 localhost.localdomain rpm-ostree[2036]: client(id:cli dbus:1.396 unit:gnome-terminal-server.service uid:0) vanished; remaining=1
dez 06 14:30:43 localhost.localdomain rpm-ostree[2036]: client(id:cli dbus:1.399 unit:gnome-terminal-server.service uid:0) added; new total=2
dez 06 14:30:43 localhost.localdomain rpm-ostree[2036]: client(id:cli dbus:1.399 unit:gnome-terminal-server.service uid:0) vanished; remaining=1
dez 06 14:31:09 localhost.localdomain rpm-ostree[2036]: client(id:cli dbus:1.404 unit:gnome-terminal-server.service uid:0) added; new total=2
dez 06 14:31:09 localhost.localdomain rpm-ostree[2036]: Initiated txn Cleanup for client(id:cli dbus:1.404 unit:gnome-terminal-server.service uid:0): /org/projectatomic/rpmostree1/fedora_workstation
dez 06 14:31:09 localhost.localdomain rpm-ostree[2036]: Txn Cleanup on /org/projectatomic/rpmostree1/fedora_workstation failed: syscore cleanup: Cleaning deployments: unlinkat: Operation not permitted
dez 06 14:31:09 localhost.localdomain rpm-ostree[2036]: client(id:cli dbus:1.404 unit:gnome-terminal-server.service uid:0) vanished; remaining=1
dez 06 14:31:49 localhost.localdomain rpm-ostree[2036]: client(id:cli dbus:1.412 unit:gnome-terminal-server.service uid:0) added; new total=2
dez 06 14:31:49 localhost.localdomain rpm-ostree[2036]: Initiated txn Cleanup for client(id:cli dbus:1.412 unit:gnome-terminal-server.service uid:0): /org/projectatomic/rpmostree1/fedora_workstation
dez 06 14:31:49 localhost.localdomain rpm-ostree[2036]: Txn Cleanup on /org/projectatomic/rpmostree1/fedora_workstation successful
dez 06 14:31:49 localhost.localdomain rpm-ostree[2036]: client(id:cli dbus:1.412 unit:gnome-terminal-server.service uid:0) vanished; remaining=1
dez 06 14:31:54 localhost.localdomain rpm-ostree[2036]: client(id:cli dbus:1.415 unit:gnome-terminal-server.service uid:0) added; new total=2
dez 06 14:31:54 localhost.localdomain rpm-ostree[2036]: Initiated txn Cleanup for client(id:cli dbus:1.415 unit:gnome-terminal-server.service uid:0): /org/projectatomic/rpmostree1/fedora_workstation
dez 06 14:31:58 localhost.localdomain root[3300]: os-prober: debug: running /usr/libexec/os-probes/mounted/05efi on mounted /dev/sda1
dez 06 14:31:58 localhost.localdomain root[3300]: 05efi: debug: Not on UEFI platform
dez 06 14:31:58 localhost.localdomain root[3300]: os-prober: debug: running /usr/libexec/os-probes/mounted/10freedos on mounted /dev/sda1
dez 06 14:31:58 localhost.localdomain root[3300]: 10freedos: debug: /dev/sda1 is not a FAT partition: exiting
dez 06 14:31:58 localhost.localdomain root[3300]: os-prober: debug: running /usr/libexec/os-probes/mounted/10qnx on mounted /dev/sda1
dez 06 14:31:58 localhost.localdomain root[3300]: 10qnx: debug: /dev/sda1 is not a QNX4 partition: exiting
dez 06 14:31:58 localhost.localdomain root[3300]: os-prober: debug: running /usr/libexec/os-probes/mounted/20macosx on mounted /dev/sda1
dez 06 14:31:58 localhost.localdomain macosx-prober[3351]: debug: /dev/sda1 is not an HFS+ partition: exiting
dez 06 14:31:58 localhost.localdomain root[3300]: os-prober: debug: running /usr/libexec/os-probes/mounted/20microsoft on mounted /dev/sda1
dez 06 14:31:58 localhost.localdomain root[3300]: 20microsoft: debug: /dev/sda1 is not a MS partition: exiting
dez 06 14:31:58 localhost.localdomain root[3300]: os-prober: debug: running /usr/libexec/os-probes/mounted/30utility on mounted /dev/sda1
dez 06 14:31:58 localhost.localdomain root[3300]: 30utility: debug: /dev/sda1 is not a FAT partition: exiting
dez 06 14:31:58 localhost.localdomain root[3300]: os-prober: debug: running /usr/libexec/os-probes/mounted/40lsb on mounted /dev/sda1
dez 06 14:31:58 localhost.localdomain root[3300]: os-prober: debug: running /usr/libexec/os-probes/mounted/70hurd on mounted /dev/sda1
dez 06 14:31:58 localhost.localdomain root[3300]: os-prober: debug: running /usr/libexec/os-probes/mounted/80minix on mounted /dev/sda1
dez 06 14:31:58 localhost.localdomain root[3300]: os-prober: debug: running /usr/libexec/os-probes/mounted/83haiku on mounted /dev/sda1
dez 06 14:31:58 localhost.localdomain root[3300]: 83haiku: debug: /dev/sda1 is not a BeFS partition: exiting
dez 06 14:31:58 localhost.localdomain root[3300]: os-prober: debug: running /usr/libexec/os-probes/mounted/90linux-distro on mounted /dev/sda1
dez 06 14:31:58 localhost.localdomain root[3300]: os-prober: debug: running /usr/libexec/os-probes/mounted/90solaris on mounted /dev/sda1
dez 06 14:31:58 localhost.localdomain root[3300]: os-prober: debug: running /usr/libexec/os-probes/50mounted-tests on /dev/sda2
dez 06 14:31:58 localhost.localdomain root[3300]: 50mounted-tests: debug: /dev/sda2 is an LVM member; skipping
dez 06 14:31:58 localhost.localdomain root[3300]: os-prober: debug: /dev/mapper/fedora-swap: is active swap
dez 06 14:31:59 localhost.localdomain rpm-ostree[2036]: Transaction complete; bootconfig swap: yes; deployment count change: -1
dez 06 14:32:07 localhost.localdomain rpm-ostree[2036]: Txn Cleanup on /org/projectatomic/rpmostree1/fedora_workstation failed: syscore cleanup: Cleaning deployments: unlinkat: Operation not permitted
dez 06 14:32:07 localhost.localdomain rpm-ostree[2036]: client(id:cli dbus:1.415 unit:gnome-terminal-server.service uid:0) vanished; remaining=1
dez 06 14:32:25 localhost.localdomain rpm-ostree[2036]: Allowing active client :1.417 (uid 1000)
dez 06 14:32:25 localhost.localdomain rpm-ostree[2036]: client(id:cli dbus:1.417 unit:gnome-terminal-server.service uid:1000) added; new total=2
dez 06 14:32:25 localhost.localdomain rpm-ostree[2036]: client(id:cli dbus:1.417 unit:gnome-terminal-server.service uid:1000) vanished; remaining=1
dez 06 14:34:07 localhost.localdomain rpm-ostree[2036]: client(id:cli dbus:1.424 unit:gnome-terminal-server.service uid:0) added; new total=2
dez 06 14:34:08 localhost.localdomain rpm-ostree[2036]: Initiated txn Cleanup for client(id:cli dbus:1.424 unit:gnome-terminal-server.service uid:0): /org/projectatomic/rpmostree1/fedora_workstation
dez 06 14:34:08 localhost.localdomain rpm-ostree[2036]: Txn Cleanup on /org/projectatomic/rpmostree1/fedora_workstation successful
dez 06 14:34:08 localhost.localdomain rpm-ostree[2036]: client(id:cli dbus:1.424 unit:gnome-terminal-server.service uid:0) vanished; remaining=1

walters · 2018-12-06T19:27:27.141Z

Do you have any SELinux denials? grep avc.*denied /var/log/audit/audit.log?

If so try: restorecon -Rv /boot

felipehw · 2018-12-06T19:38:17.834Z

Yes, i i have SELinux denials. But i run restorecon -Rv /boot and didn’t solve the problem. I still get unlinkat: Operation not permitted when i run rpm-ostree cleanup -b.

[dread@localhost ~]$ sudo grep avc.*denied /var/log/audit/audit.log
[sudo] senha para dread: 
type=AVC msg=audit(1540075313.931:246): avc:  denied  { execute } for  pid=2702 comm="geoclue" path=2F746D702F666669626762765841202864656C6574656429 dev="tmpfs" ino=45809 scontext=system_u:system_r:geoclue_t:s0 tcontext=system_u:object_r:geoclue_tmp_t:s0 tclass=file permissive=0
type=AVC msg=audit(1540075313.931:247): avc:  denied  { execute } for  pid=2702 comm="geoclue" path=2F7661722F746D702F666669683776546457202864656C6574656429 dev="dm-0" ino=920513 scontext=system_u:system_r:geoclue_t:s0 tcontext=system_u:object_r:geoclue_tmp_t:s0 tclass=file permissive=0
type=AVC msg=audit(1540075313.931:248): avc:  denied  { write } for  pid=2702 comm="geoclue" name="/" dev="tmpfs" ino=18578 scontext=system_u:system_r:geoclue_t:s0 tcontext=system_u:object_r:tmpfs_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1540079179.206:297): avc:  denied  { unlink } for  pid=6538 comm="systemd-user-ru" name=".flatpak-cache" dev="tmpfs" ino=39709 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=lnk_file permissive=0
type=AVC msg=audit(1540092355.835:277): avc:  denied  { unlink } for  pid=5392 comm="systemd-user-ru" name=".flatpak-cache" dev="tmpfs" ino=48953 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=lnk_file permissive=0
type=AVC msg=audit(1540247913.669:280): avc:  denied  { unlink } for  pid=3689 comm="systemd-user-ru" name=".flatpak-cache" dev="tmpfs" ino=38813 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=lnk_file permissive=0
type=AVC msg=audit(1540320217.103:265): avc:  denied  { unlink } for  pid=3413 comm="systemd-user-ru" name=".flatpak-cache" dev="tmpfs" ino=46051 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=lnk_file permissive=0
type=AVC msg=audit(1540413475.598:262): avc:  denied  { unlink } for  pid=4057 comm="systemd-user-ru" name=".flatpak-cache" dev="tmpfs" ino=47882 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=lnk_file permissive=0
type=AVC msg=audit(1540508560.415:276): avc:  denied  { unlink } for  pid=3797 comm="systemd-user-ru" name=".flatpak-cache" dev="tmpfs" ino=43606 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=lnk_file permissive=0
type=AVC msg=audit(1540508835.511:267): avc:  denied  { unlink } for  pid=3096 comm="systemd-user-ru" name=".flatpak-cache" dev="tmpfs" ino=45749 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=lnk_file permissive=0
type=AVC msg=audit(1540561297.116:283): avc:  denied  { unlink } for  pid=3692 comm="systemd-user-ru" name=".flatpak-cache" dev="tmpfs" ino=49677 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=lnk_file permissive=0
type=AVC msg=audit(1540686679.276:280): avc:  denied  { unlink } for  pid=3605 comm="systemd-user-ru" name=".flatpak-cache" dev="tmpfs" ino=50509 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=lnk_file permissive=0
type=AVC msg=audit(1540756614.813:273): avc:  denied  { unlink } for  pid=3307 comm="systemd-user-ru" name=".flatpak-cache" dev="tmpfs" ino=43709 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=lnk_file permissive=0
type=AVC msg=audit(1540854949.617:323): avc:  denied  { unlink } for  pid=3297 comm="systemd-user-ru" name=".flatpak-cache" dev="tmpfs" ino=50248 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=lnk_file permissive=0
type=AVC msg=audit(1540926865.674:303): avc:  denied  { unlink } for  pid=3842 comm="systemd-user-ru" name=".flatpak-cache" dev="tmpfs" ino=42919 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=lnk_file permissive=0
type=AVC msg=audit(1541016153.202:278): avc:  denied  { unlink } for  pid=3613 comm="systemd-user-ru" name=".flatpak-cache" dev="tmpfs" ino=49384 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=lnk_file permissive=0
type=AVC msg=audit(1541124142.521:261): avc:  denied  { unlink } for  pid=3286 comm="systemd-user-ru" name=".flatpak-cache" dev="tmpfs" ino=47307 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=lnk_file permissive=0
type=AVC msg=audit(1541331244.097:266): avc:  denied  { unlink } for  pid=3555 comm="systemd-user-ru" name=".flatpak-cache" dev="tmpfs" ino=46634 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=lnk_file permissive=0
type=AVC msg=audit(1541331348.842:240): avc:  denied  { read } for  pid=2535 comm="systemd-user-ru" name="dbus-1" dev="tmpfs" ino=26293 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:session_dbusd_tmp_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1541331429.693:267): avc:  denied  { read } for  pid=2685 comm="systemd-user-ru" name="dbus-1" dev="tmpfs" ino=35928 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:session_dbusd_tmp_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1541360369.706:230): avc:  denied  { read } for  pid=2478 comm="systemd-user-ru" name="dbus-1" dev="tmpfs" ino=23311 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:session_dbusd_tmp_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1541360403.438:256): avc:  denied  { read } for  pid=2657 comm="systemd-user-ru" name="dbus-1" dev="tmpfs" ino=33894 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:session_dbusd_tmp_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1541360457.830:235): avc:  denied  { read } for  pid=2609 comm="systemd-user-ru" name="dbus-1" dev="tmpfs" ino=27783 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:session_dbusd_tmp_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1541360484.899:271): avc:  denied  { read } for  pid=2811 comm="systemd-user-ru" name="dbus-1" dev="tmpfs" ino=32484 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:session_dbusd_tmp_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1541470106.799:239): avc:  denied  { read } for  pid=2567 comm="systemd-user-ru" name="dbus-1" dev="tmpfs" ino=25385 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:session_dbusd_tmp_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1541471015.049:275): avc:  denied  { unlink } for  pid=3898 comm="systemd-user-ru" name=".flatpak-cache" dev="tmpfs" ino=47783 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=lnk_file permissive=0
type=AVC msg=audit(1541471015.049:276): avc:  denied  { read } for  pid=3898 comm="systemd-user-ru" name="dbus-1" dev="tmpfs" ino=30603 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:session_dbusd_tmp_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1541471105.597:235): avc:  denied  { read } for  pid=2512 comm="systemd-user-ru" name="dbus-1" dev="tmpfs" ino=29760 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:session_dbusd_tmp_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1541471417.958:262): avc:  denied  { unlink } for  pid=3083 comm="systemd-user-ru" name=".flatpak-cache" dev="tmpfs" ino=46666 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=lnk_file permissive=0
type=AVC msg=audit(1541471417.959:263): avc:  denied  { read } for  pid=3083 comm="systemd-user-ru" name="dbus-1" dev="tmpfs" ino=22507 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:session_dbusd_tmp_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1541606419.824:239): avc:  denied  { read } for  pid=2596 comm="systemd-user-ru" name="dbus-1" dev="tmpfs" ino=24274 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:session_dbusd_tmp_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1541606789.992:268): avc:  denied  { unlink } for  pid=4041 comm="systemd-user-ru" name=".flatpak-cache" dev="tmpfs" ino=47659 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=lnk_file permissive=0
type=AVC msg=audit(1541606789.993:269): avc:  denied  { read } for  pid=4041 comm="systemd-user-ru" name="dbus-1" dev="tmpfs" ino=30960 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:session_dbusd_tmp_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1541607251.706:238): avc:  denied  { read } for  pid=2575 comm="systemd-user-ru" name="dbus-1" dev="tmpfs" ino=23335 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:session_dbusd_tmp_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1541607271.257:262): avc:  denied  { read } for  pid=2817 comm="systemd-user-ru" name="dbus-1" dev="tmpfs" ino=33222 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:session_dbusd_tmp_t:s0 tclass=dir permissive=0

walters · 2018-12-06T22:30:51.257Z

OK I wrote https://gitlab.gnome.org/GNOME/libglnx/merge_requests/4 which would help debug this in the future. I could provide a test build with that, but…can you try this command as root?

# capsh --drop=cap_dac_override -- -c 'find /boot/ "!" -writable'

Explainer: You’d think a simple find /boot "!" -writable would find unwritable files, but root processes by default have CAP_DAC_OVERRIDE - we do want to see directories/files which are normally unwritable.

This command will also turn up files with chattr +i which is I suspect what happened for you at some point.

felipehw · 2018-12-06T23:08:00.296Z

I try the command but the problem remains.

[root@localhost ~]# capsh --drop=cap_dac_override -- -c 'find /boot/ "!" -writable'
[root@localhost ~]#

[dread@localhost ~]$ sudo rpm-ostree cleanup  -b
error: syscore cleanup: Cleaning deployments: unlinkat: Operation not permitted
[dread@localhost ~]$ sudo rpm-ostree update
997 metadata, 3649 content objects fetched; 192735 KiB transferred in 260 seconds                                                                                                                                  
Staging deployment... error: Cleaning deployments: unlinkat: Operation not permitted
[dread@localhost ~]$ rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
  ostree://fedora-workstation:fedora/rawhide/x86_64/silverblue
                   Version: Rawhide.20181206.n.0 (2018-12-06T07:05:27Z)
                    Commit: afce15ba4c1b5c2f10db967f2a839218821b00639285afaa44413b754ec23bfc
              GPGSignature: Valid signature by F1D8EC98F241AAF20DF69420EF3C111FCFC659B9

● ostree://fedora-workstation:fedora/rawhide/x86_64/silverblue
                   Version: Rawhide.20181205.n.0 (2018-12-05T07:07:18Z)
                    Commit: 32ebe212fe6bb4a624c4005d4df6731fad6361119e08ed5c3c135fa9b73453c5
              GPGSignature: Valid signature by F1D8EC98F241AAF20DF69420EF3C111FCFC659B9
[dread@localhost ~]$

walters · 2018-12-07T14:30:50.107Z

Ah wait, that Cleaning deployments: part is a useful hint that I missed earlier. It’s not your /boot directory; the un-unlink()able file is something in your /ostree/deploy/ subdirectory. Hmm. Try this command:

find /ostree/deploy/*/deploy/ \( -type d -o -type f \) -a "!" -writable

felipehw · 2018-12-07T21:01:32.211Z

I tried this new command but without success.
I’ll proceed to a new installation. But i think that maybe is better wait the next official release to try again

walters · 2019-01-03T20:03:18.943Z

We found the likely culprit for this:

https://src.fedoraproject.org/rpms/nfs-utils/c/684c60c247dd9eb846698decc9386853a7defac3

Which was part of https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/F3SLKANTCMSR25HGSJJ6H4FHPZDP7TFW/

I filed a PR here: https://src.fedoraproject.org/rpms/nfs-utils/pull-request/7

walters · 2019-01-03T20:19:15.109Z

And just to be explicit, for anyone affected by this, just:

chattr -i /etc/sysconfig/nfs

felipehw · 2019-01-17T16:02:48.792Z

The problem remains:

[root@localhost ~]# rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
● ostree://fedora-workstation:fedora/rawhide/x86_64/silverblue
                   Version: Rawhide.20190108.n.0 (2019-01-08T07:12:59Z)
                    Commit: 46c8a469662d193ae28570120c2000f9b79856f344129ae520c05cc5c80c90dc
              GPGSignature: Valid signature by F1D8EC98F241AAF20DF69420EF3C111FCFC659B9

  ostree://fedora-workstation:fedora/rawhide/x86_64/silverblue
                   Version: Rawhide.20190107.n.0 (2019-01-07T07:03:05Z)
                    Commit: 33c51f5852a7a87ead5a36e4a3c76e633922af258c78e2ecbcde10f25a2a6dbb
              GPGSignature: Valid signature by F1D8EC98F241AAF20DF69420EF3C111FCFC659B9
[root@localhost ~]# chattr -i /etc/sysconfig/nfs
[root@localhost ~]# rpm-ostree update
⠤ Receiving objects: 99% (6822/6841) 904,7 kB/s 290,4 MB 
Receiving objects: 99% (6822/6841) 904,7 kB/s 290,4 MB... done
Staging deployment... done
error: Cleaning deployments: unlinkat: Operation not permitted
[root@localhost ~]# rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
  ostree://fedora-workstation:fedora/rawhide/x86_64/silverblue
                   Version: Rawhide.20190117.n.0 (2019-01-17T07:22:44Z)
                    Commit: 74a7eacd7eba30f20841928f01dce4936f8cfd80391718c49a0bafe98768e593
              GPGSignature: Valid signature by F1D8EC98F241AAF20DF69420EF3C111FCFC659B9

● ostree://fedora-workstation:fedora/rawhide/x86_64/silverblue
                   Version: Rawhide.20190108.n.0 (2019-01-08T07:12:59Z)
                    Commit: 46c8a469662d193ae28570120c2000f9b79856f344129ae520c05cc5c80c90dc
              GPGSignature: Valid signature by F1D8EC98F241AAF20DF69420EF3C111FCFC659B9

  ostree://fedora-workstation:fedora/rawhide/x86_64/silverblue
                   Version: Rawhide.20190107.n.0 (2019-01-07T07:03:05Z)
                    Commit: 33c51f5852a7a87ead5a36e4a3c76e633922af258c78e2ecbcde10f25a2a6dbb
              GPGSignature: Valid signature by F1D8EC98F241AAF20DF69420EF3C111FCFC659B9
[root@localhost ~]#