Silverblue fsck-detected corruption

I am testing Silverblue 40 on an older laptop, everything seems to be working except nvidia drivers. However, ostree fcsk -a reports 3 partial commits and fsck corruption.

adminm@fedora ~ [1]> sudo ostree fsck -a
Validating refs...
Validating refs in collections...
Enumerating commits...
Verifying content integrity of 225 commit objects...
fsck objects (133467/133467) [=============] 100%
3 partial commits not verified
error: 3 partial commits from fsck-detected corruption

Checking the partial commit ids from ostree/repo, one of them (9ffc..) is the base commit for the current deployment, which seems to be booting normally except nvidia drivers, so maybe one of driver files got corrupted some-how or this is something unrelated. Would love any help on how to fix this.

adminm@fedora ~ [1]> sudo find /ostree/repo/state/ -type f -name "*.commitpartial"
/ostree/repo/state/9ffc6c50ecebd3ead2e17af302d7ff1cf1241a975ea6d7a664f0e89bffa30963.commitpartial
/ostree/repo/state/d2d11e9c93fb96e7f8aa04e05f4aed6da903a2b5cfb721e2cb52f2482bb70a1c.commitpartial
/ostree/repo/state/b5a05860c9efb0235d34128915bf1d76181ddb73abd3c4d0360e7fbc78c92f65.commitpartial

adminm@fedora ~> rpm-ostree status -v
State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: no runs since boot
Deployments:
● ostree-image-signed:docker://ghcr.io/ublue-os/silverblue-main:latest (index: 0)
                   Digest: sha256:1c469cf7296a7ad0e1127b24d73553439d40a1b995d68c5df487526c232b5c00
                  Version: 40.20240708.0 (2024-07-08T03:12:23Z)
               BaseCommit: 9ffc6c50ecebd3ead2e17af302d7ff1cf1241a975ea6d7a664f0e89bffa30963
                   Commit: c1bdacaae9ec3335d9c6aba858a616fbd83e0c154993cc8451ad1a72d7f5cdef
                           β”œβ”€ copr:copr.fedorainfracloud.org:phracek:PyCharm (2024-03-18T11:54:41Z)
                           β”œβ”€ copr:copr.fedorainfracloud.org:ublue-os:akmods (2024-06-10T03:51:26Z)
                           β”œβ”€ fedora (2024-04-14T18:51:11Z)
                           β”œβ”€ fedora-cisco-openh264 (2024-03-12T11:45:42Z)
                           β”œβ”€ google-chrome (2024-06-27T18:50:17Z)
                           β”œβ”€ rpmfusion-free (2024-04-20T12:11:51Z)
                           β”œβ”€ rpmfusion-free-updates (2024-07-05T09:05:10Z)
                           β”œβ”€ rpmfusion-free-updates-testing (2024-07-05T09:05:30Z)
                           β”œβ”€ rpmfusion-nonfree (2024-04-20T12:18:23Z)
                           β”œβ”€ rpmfusion-nonfree-nvidia-driver (2024-07-05T08:40:39Z)
                           β”œβ”€ rpmfusion-nonfree-steam (2024-07-05T08:54:24Z)
                           β”œβ”€ rpmfusion-nonfree-updates (2024-07-05T09:21:18Z)
                           β”œβ”€ rpmfusion-nonfree-updates-testing (2024-07-05T09:21:28Z)
                           β”œβ”€ updates (2024-07-08T01:52:36Z)
                           └─ updates-archive (2024-06-30T05:31:17Z)
                   Staged: no
                StateRoot: fedora
          LayeredPackages: broadcom-wl fish kmod-nvidia-470xx xorg-x11-drv-nvidia-470xx-cuda
                   Pinned: yes

  fedora:fedora/40/x86_64/silverblue (index: 1)
                  Version: 40.20240706.0 (2024-07-06T00:45:38Z)
               BaseCommit: 9964b37082d1fcb3f0fb26ea4447c838e606947ba3dfa238c7c32954df25a8a7
                           β”œβ”€ repo-0 (2024-04-14T18:51:11Z)
                           β”œβ”€ repo-1 (2024-07-06T00:17:34Z)
                           └─ repo-2 (2024-07-06T00:26:08Z)
                   Commit: b4e83a2328f025a0e50fc41e2d4345279b861c9699b5a2dbbfae0e48120330a8
                StateRoot: fedora
             GPGSignature: 1 signature
                           Signature made Fri 05 Jul 2024 08:54:31 PM EDT using RSA key ID 0727707EA15B79CC
                           Good signature from "Fedora <fedora-40-primary@fedoraproject.org>"
          LayeredPackages: broadcom-wl fish kmod-nvidia-470xx rpmfusion-free-release rpmfusion-nonfree-release tmux xorg-x11-drv-nvidia-470xx-cuda
                   Pinned: yes

  fedora:fedora/40/x86_64/silverblue (index: 2)
                  Version: 40.20240706.0 (2024-07-06T00:45:38Z)
               BaseCommit: 9964b37082d1fcb3f0fb26ea4447c838e606947ba3dfa238c7c32954df25a8a7
                           β”œβ”€ repo-0 (2024-04-14T18:51:11Z)
                           β”œβ”€ repo-1 (2024-07-06T00:17:34Z)
                           └─ repo-2 (2024-07-06T00:26:08Z)
                   Commit: cd9845518162c017fbae365bf7ee563b56e19f33c7a8a6b2e063ce9a41093114
                           β”œβ”€ fedora (2024-04-14T18:51:11Z)
                           β”œβ”€ fedora-cisco-openh264 (2024-03-12T11:45:42Z)
                           β”œβ”€ rpmfusion-free (2024-04-20T12:11:51Z)
                           β”œβ”€ rpmfusion-free-updates (2024-07-05T09:05:10Z)
                           β”œβ”€ rpmfusion-nonfree (2024-04-20T12:18:23Z)
                           β”œβ”€ rpmfusion-nonfree-updates (2024-07-05T09:21:18Z)
                           β”œβ”€ updates (2024-07-06T07:03:40Z)
                           └─ updates-archive (2024-06-30T05:31:17Z)
                StateRoot: fedora
             GPGSignature: 1 signature
                           Signature made Fri 05 Jul 2024 08:54:31 PM EDT using RSA key ID 0727707EA15B79CC
                           Good signature from "Fedora <fedora-40-primary@fedoraproject.org>"
          LayeredPackages: broadcom-wl fish rpmfusion-free-release rpmfusion-nonfree-release tmux
                   Pinned: yes

Silverblue troubleshooting ([1]) suggests to re-pull the commit again, however ostree pull ostree-image-signed:docker://ghcr.io/ublue-os/silverblue-main:<commit_id> is erroring out with invalid refspec.

Added ostree, rpmfusion, ublue-unsupported

This is a uBlue issue, please use their forum

1 Like

You should still be able to do POC: fsck repair by krnowak Β· Pull Request #345 Β· ostreedev/ostree Β· GitHub to mark the β€œbroken” commits/file as partially fetched. Then I would reboot on a previous version, cleanup the new one and re-pull it as a normal update, which should re-add everything.

2 Likes

Removed ublue-unsupported

Removed rpmfusion

Thanks, updating from older-version re-pulled the 3 partial commits. ostree fsck now reports only one corrupted file.

adminm@fedora ~> sudo ostree fsck
Validating refs...
Validating refs in collections...
Enumerating commits...
Verifying content integrity of 244 commit objects...
fsck objects (94541/134338) [=========    ]  70%
error: In commits d5770d00749797b4a7e4dfbe9a24dcfbf7aa74b61bd90b2f9c3b6ca1b945f491: fsck content object e2b13cdbe0ef6facc26c5938f51bcb18d5fc293d65ab846d19e3ce8c63df38f5: Corrupted file object; checksum expected='e2b13cdbe0ef6facc26c5938f51bcb18d5fc293d65ab846d19e3ce8c63df38f5' actual='933b8b10b69525ffe6c613f3a30e71b2a610432b40f29df5f9c60db06900b02b'

Here’s ostree show for the corrupt objects fwiw

adminm@fedora ~> ostree show e2b13cdbe0ef6facc26c5938f51bcb18d5fc293d65ab846d19e3ce8c63df38f5
Object: e2b13cdbe0ef6facc26c5938f51bcb18d5fc293d65ab846d19e3ce8c63df38f5
Type: file
File Type: regular
Size: 0
Mode: 0100777
Uid: 0
Gid: 0
Extended Attributes: { [(b'security.selinux', b'system_u:object_r:root_t:s0')] }
adminm@fedora ~> ostree show 933b8b10b69525ffe6c613f3a30e71b2a610432b40f29df5f9c60db06900b02b
error: Opening content object 933b8b10b69525ffe6c613f3a30e71b2a610432b40f29df5f9c60db06900b02b: Couldn't find file object '933b8b10b69525ffe6c613f3a30e71b2a610432b40f29df5f9c60db06900b02b'