Fs-verity `FILE CORRUPTED!` errors on ostree file despite `btrfs scrub` showing no errors

[gucci-on-fleek (Max Chernoff)]

My Fedora IoT 42 server recently started misbehaving, and checking the system journal showed lots of fs-verity: FILE CORRUPTED! errors. I first noticed my system misbehaving (services failing, unable to log in) around 18 hours after it had last been updated and rebooted, but searching the system journal showed that the errors first started immediately after it had rebooted for an update. Rebooting the system and running btrfs scrub had no effect; the fs-verity: FILE CORRUPTED! errors kept showing up.

Shell transcript while debugging

$ sudo btrfs scrub start -B -d /dev/vda4
Starting scrub on devid 1

Scrub device /dev/vda4 (id 1) done
Scrub started:    Fri Sep 26 00:42:26 2025
Status:           finished
Duration:         0:00:22
Total to scrub:   113.66GiB
Rate:             5.17GiB/s
Error summary:    no errors found

$ sudo cat /sysroot/ostree/repo/config
[core]
repo_version=1
mode=bare

[sysroot]
readonly=true
bootloader=none

[ex-integrity]
composefs=yes

$ sudo ostree fsck --all --verbose
OT: using fuse: 0
OT: using fuse: 0
Validating refs...
Validating refs in collections...
Enumerating commits...
Verifying content integrity of 88 commit objects...
OT: Traversing commit 5c244a5b03513483fb3dd03b521fa6a9dbf716627d9c1b1ba59c9308565ecc7f
[...]
OT: Traversing commit 0a34ad1b46737209a1d0d8baea456075893a4e9aa4a66f5981623112247d515f
OT: Found dirtree object 7861de45bf7811062f6cf341c7999a7733d8cc7564abdc03bfc14cab434cd290
OT: Found dirmeta object 446a0ef11b7cc167f3b603e585c7eeeeb675faa412d5ec73f62988eb0b6c5488
OT: Traversing dirtree 7861de45bf7811062f6cf341c7999a7733d8cc7564abdc03bfc14cab434cd290
fish: Job 1, 'sudo ostree fsck --all --verbose' terminated by signal SIGBUS (Misaligned address error)

$ sudo btrfs inspect-internal inode-resolve 10909667 /sysroot/  # From dmesg
/sysroot//ostree/repo/objects/78/61de45bf7811062f6cf341c7999a7733d8cc7564abdc03bfc14cab434cd290.dirtree

$ sudo cat /sysroot//ostree/repo/objects/78/61de45bf7811062f6cf341c7999a7733d8cc7564abdc03bfc14cab434cd290.dirtree
cat: /sysroot//ostree/repo/objects/78/61de45bf7811062f6cf341c7999a7733d8cc7564abdc03bfc14cab434cd290.dirtree: Input/output error

$ cat /etc/os-release
NAME="Fedora Linux"
VERSION="42 (IoT Edition)"
RELEASE_TYPE=stable
ID=fedora
VERSION_ID=42
VERSION_CODENAME=""
PLATFORM_ID="platform:f42"
PRETTY_NAME="Fedora Linux 42 (IoT Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:42"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f42/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=42
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=42
SUPPORT_END=2026-05-13
VARIANT="IoT Edition"
VARIANT_ID=iot

$ rpm --query --queryformat='%{NAME}\t%{VERSION}\n' --all | grep -Ev 'perl|python' sort | column --table
aardvark-dns                     1.16.0
acl                              2.3.2
alternatives                     1.33
audit-libs                       4.1.1
audit-rules                      4.1.1
authselect                       1.5.1
authselect-libs                  1.5.1
basesystem                       11
bash                             5.2.37
bees                             0.10
boost-system                     1.83.0
boost-thread                     1.83.0
bootc                            1.8.0
borgbackup                       1.4.1
btrfs-progs                      6.16.1
btrfsmaintenance                 0.5.2
bubblewrap                       0.10.0
busybox                          1.37.0
bzip2-libs                       1.0.8
ca-certificates                  2025.2.80_v9.0.304
catatonit                        0.2.1
checkpolicy                      3.8
chrony                           4.8
composefs                        1.0.8
composefs-libs                   1.0.8
conmon                           2.1.13
container-selinux                2.242.0
containers-common                0.64.2
containers-common-extra          0.64.2
coreutils                        9.6
coreutils-common                 9.6
cpio                             2.15
cracklib                         2.9.11
crun                             1.24
crypto-policies                  20250707
cryptsetup-libs                  2.8.1
curl                             8.11.1
cyrus-sasl-lib                   2.1.28
dbus                             1.16.0
dbus-broker                      36
dbus-common                      1.16.0
dbus-libs                        1.16.0
device-mapper                    1.02.204
device-mapper-libs               1.02.204
diffutils                        3.12
dosfstools                       4.2
dracut                           107
dracut-config-generic            107
e2fsprogs-libs                   1.47.2
efi-filesystem                   6
efibootmgr                       18
efivar-libs                      39
elfutils-libelf                  0.193
expat                            2.7.2
fedora-gpg-keys                  42
fedora-iot-config                1
fedora-release-common            42
fedora-release-identity-iot      42
fedora-release-iot               42
fedora-repos                     42
file                             5.46
file-libs                        5.46
filesystem                       3.18
findutils                        4.10.0
firewalld                        2.3.1
firewalld-filesystem             2.3.1
fish                             4.0.2
fuse                             2.9.9
fuse-common                      3.16.2
fuse-libs                        2.9.9
fuse3                            3.16.2
fuse3-libs                       3.16.2
fwupd-efi                        1.6
gawk                             5.3.1
gdbm                             1.23
gdbm-libs                        1.23
gettext-envsubst                 0.23.1
gettext-libs                     0.23.1
gettext-runtime                  0.23.1
git-core                         2.51.0
glib2                            2.84.4
glibc                            2.41
glibc-common                     2.41
glibc-langpack-en                2.41
glibc-minimal-langpack           2.41
gmp                              6.3.0
gnupg2                           2.4.7
gnutls                           3.8.10
gobject-introspection            1.84.0
gpgme                            1.24.3
gpm-libs                         1.20.7
greenboot                        0.15.8
greenboot-default-health-checks  0.15.8
grep                             3.11
groff-base                       1.23.0
grub2-common                     2.12
grub2-efi-x64                    2.12
grub2-tools                      2.12
grub2-tools-minimal              2.12
gzip                             1.13
htop                             3.4.1
hwloc-libs                       2.12.0
info                             7.2
iproute                          6.12.0
iptables-libs                    1.8.11
iputils                          20250605
jansson                          2.14
jq                               1.7.1
json-c                           0.18
json-glib                        1.10.6
kbd                              2.7.1
kbd-legacy                       2.7.1
kbd-misc                         2.7.1
kernel                           6.16.8
kernel-core                      6.16.8
kernel-modules                   6.16.8
kernel-modules-core              6.16.8
keyutils-libs                    1.6.3
kmod                             33
kmod-libs                        33
krb5-libs                        1.21.3
less                             679
libacl                           2.3.2
libarchive                       3.8.1
libassuan                        2.5.7
libattr                          2.5.2
libb2                            0.98.1
libblkid                         2.40.4
libbpf                           1.5.0
libbtrfs                         6.16.1
libbtrfsutil                     6.16.1
libcap                           2.73
libcap-ng                        0.8.5
libcbor                          0.11.0
libcom_err                       1.47.2
libcurl-minimal                  8.11.1
libeconf                         0.7.6
libedit                          3.1
libevent                         2.1.12
libfdisk                         2.40.4
libffi                           3.4.6
libfido2                         1.15.0
libgcc                           15.2.1
libgcrypt                        1.11.0
libgomp                          15.2.1
libgpg-error                     1.51
libidn2                          2.3.8
libkcapi                         1.5.0
libkcapi-hasher                  1.5.0
libkcapi-hmaccalc                1.5.0
libksba                          1.6.7
libmicrohttpd                    1.0.2
libmnl                           1.0.5
libmodulemd                      2.15.2
libmount                         2.40.4
libnetfilter_conntrack           1.0.9
libnfnetlink                     1.0.1
libnftnl                         1.2.8
libnghttp2                       1.64.0
libnsl2                          2.0.1
libpipeline                      1.5.8
libpkgconf                       2.3.0
libpwquality                     1.4.5
librepo                          1.20.0
libseccomp                       2.5.5
libselinux                       3.8
libselinux-utils                 3.8
libsemanage                      3.8.1
libsepol                         3.8
libsmartcols                     2.40.4
libsodium                        1.0.20
libsolv                          0.7.34
libstdc++                        15.2.1
libtasn1                         4.20.0
libtextstyle                     0.23.1
libtirpc                         1.3.7
libtool-ltdl                     2.5.4
libunistring                     1.1
liburing                         2.9
libusb1                          1.0.29
libutempter                      1.2.1
libuuid                          2.40.4
libverto                         0.3.2
libxcrypt                        4.4.38
libxml2                          2.12.10
libxslt                          1.1.43
libyaml                          0.2.5
libzstd                          1.5.7
lua-libs                         5.4.8
lz4-libs                         1.10.0
lzo                              2.10
mailcap                          2.1.54
man-db                           2.13.0
microcode_ctl                    2.1
mpdecimal                        4.0.1
mpfr                             4.2.2
ncurses                          6.5
ncurses-base                     6.5
ncurses-libs                     6.5
ncurses-term                     6.5
netavark                         1.16.1
nettle                           3.10.1
nftables                         1.1.1
node-exporter                    1.9.1
npth                             1.8
nss-altfiles                     2.23.0
numactl-libs                     2.0.19
oniguruma                        6.9.10
openldap                         2.6.10
openssh                          9.9p1
openssh-clients                  9.9p1
openssh-server                   9.9p1
openssl-libs                     3.2.4
os-prober                        1.81
ostree                           2025.6
ostree-libs                      2025.6
p11-kit                          0.25.8
p11-kit-trust                    0.25.8
pam                              1.7.0
pam-libs                         1.7.0
passt                            0^20250919.g623dbf6
passt-selinux                    0^20250919.g623dbf6
pcre2                            10.45
pcre2-syntax                     10.45
pcre2-utf32                      10.45
pkgconf                          2.3.0
pkgconf-m4                       2.3.0
pkgconf-pkg-config               2.3.0
podman                           5.6.1
policycoreutils                  3.8
polkit-libs                      126
popt                             1.19
procps-ng                        4.0.4
prometheus-podman-exporter       1.18.1
psmisc                           23.7
qemu-guest-agent                 9.2.4
readline                         8.2
ripgrep                          14.1.1
rpm                              4.20.1
rpm-libs                         4.20.1
rpm-ostree                       2025.10
rpm-ostree-libs                  2025.10
rpm-plugin-selinux               4.20.1
rpm-sequoia                      1.7.0
sed                              4.9
selinux-policy                   42.9
selinux-policy-targeted          42.9
setools-console                  4.5.1
setup                            2.15.0
shadow-utils                     4.17.4
shadow-utils-subid               4.17.4
skopeo                           1.20.0
snapper                          0.11.0
snapper-libs                     0.11.0
sqlite-libs                      3.47.2
sudo                             1.9.17
systemd                          257.9
systemd-container                257.9
systemd-journal-remote           257.9
systemd-libs                     257.9
systemd-networkd                 257.9
systemd-pam                      257.9
systemd-resolved                 257.9
systemd-shared                   257.9
systemd-sysusers                 257.9
systemd-udev                     257.9
tar                              1.35
tmux                             3.5a
tpm2-tss                         4.1.3
traceroute                       2.1.6
tzdata                           2025b
udica                            0.2.8
util-linux                       2.40.4
util-linux-core                  2.40.4
vim-common                       9.1.1775
vim-data                         9.1.1775
vim-enhanced                     9.1.1775
vim-filesystem                   9.1.1775
which                            2.23
xxd                              9.1.1775
xxhash-libs                      0.8.3
xz                               5.8.1
xz-libs                          5.8.1
yajl                             2.1.0
zchunk-libs                      1.5.1
zlib-ng-compat                   2.2.5
zram-generator                   1.2.1
zram-generator-defaults          1.2.1

$ mount
composefs on / type overlay (ro,relatime,seclabel,lowerdir+=/run/ostree/.private/cfsroot-lower,datadir+=/sysroot/ostree/repo/objects,redirect_dir=on,metacopy=on)
/dev/vda4 on /etc type btrfs (rw,relatime,seclabel,discard=async,space_cache=v2,subvolid=5,subvol=/)
/dev/vda4 on /sysroot type btrfs (ro,relatime,seclabel,discard=async,space_cache=v2,subvolid=5,subvol=/)
/dev/vda4 on /sysroot/ostree/deploy/fedora-iot/var type btrfs (rw,relatime,seclabel,discard=async,space_cache=v2,subvolid=5,subvol=/)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=4031740k,nr_inodes=1007935,mode=755,inode64)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel,nsdelegate,memory_recursiveprot)
none on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime,seclabel)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,size=1622432k,nr_inodes=819200,mode=755,inode64)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,nosuid,noexec,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=85,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=4614)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime,seclabel)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,nosuid,nodev,relatime,seclabel,pagesize=2M)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime,seclabel)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime,seclabel)
tmpfs on /run/credentials/systemd-journald.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,seclabel,size=1024k,nr_inodes=1024,mode=700,inode64,noswap)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,seclabel,nr_inodes=1048576,inode64)
/dev/vda4 on /var type btrfs (rw,relatime,seclabel,discard=async,space_cache=v2,subvolid=5,subvol=/)
/dev/vda4 on /var/home type btrfs (rw,noatime,seclabel,discard=async,space_cache=v2,subvolid=256,subvol=/00)
/dev/vda2 on /boot type ext4 (rw,relatime,seclabel)
/dev/vda4 on /var/home/.snapshots type btrfs (rw,noatime,seclabel,discard=async,space_cache=v2,subvolid=257,subvol=/00/.snapshots)
/dev/vda1 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=winnt,errors=remount-ro)
tmpfs on /run/credentials/systemd-resolved.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,seclabel,size=1024k,nr_inodes=1024,mode=700,inode64,noswap)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=811216k,nr_inodes=202804,mode=700,uid=1000,gid=1000,inode64)
tmpfs on /run/user/1001 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=811216k,nr_inodes=202804,mode=700,uid=1001,gid=1001,inode64)
tmpfs on /run/user/1002 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=811216k,nr_inodes=202804,mode=700,uid=1002,gid=1002,inode64)
tmpfs on /run/user/1006 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=811216k,nr_inodes=202804,mode=700,uid=1006,gid=1006,inode64)
tmpfs on /run/user/1003 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=811216k,nr_inodes=202804,mode=700,uid=1003,gid=1003,inode64)
tmpfs on /run/credentials/systemd-networkd.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,seclabel,size=1024k,nr_inodes=1024,mode=700,inode64,noswap)
tmpfs on /run/credentials/getty@tty1.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,seclabel,size=1024k,nr_inodes=1024,mode=700,inode64,noswap)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/user/1004 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=811216k,nr_inodes=202804,mode=700,uid=1004,gid=1004,inode64)

System journal while running the above commands

Sep 26 00:42:26 kernel: BTRFS info (device vda4): scrub: started on devid 1
Sep 26 00:42:48 kernel: BTRFS info (device vda4): scrub: finished on devid 1 with status: 0
Sep 26 00:43:21 kernel: fs-verity (vda4, inode 10909667): FILE CORRUPTED! pos=0, level=0, want_hash=sha256:dc33e0d22f2755a8e3081dea3393ea59aeb2bc5640e59b85a5220fa8b3abdd86, real_hash=sha256:ad7facb2586fc6e966c004d7d1d16b024f5805ff7cb47c7a85dabd8b48892ca7
Sep 26 00:43:21 kernel: fs-verity (vda4, inode 10909667): FILE CORRUPTED! pos=0, level=0, want_hash=sha256:dc33e0d22f2755a8e3081dea3393ea59aeb2bc5640e59b85a5220fa8b3abdd86, real_hash=sha256:ad7facb2586fc6e966c004d7d1d16b024f5805ff7cb47c7a85dabd8b48892ca7
Sep 26 00:43:21 kernel: fs-verity (vda4, inode 10909667): FILE CORRUPTED! pos=0, level=0, want_hash=sha256:dc33e0d22f2755a8e3081dea3393ea59aeb2bc5640e59b85a5220fa8b3abdd86, real_hash=sha256:ad7facb2586fc6e966c004d7d1d16b024f5805ff7cb47c7a85dabd8b48892ca7
Sep 26 00:43:21 kernel: fs-verity (vda4, inode 10909667): FILE CORRUPTED! pos=0, level=0, want_hash=sha256:dc33e0d22f2755a8e3081dea3393ea59aeb2bc5640e59b85a5220fa8b3abdd86, real_hash=sha256:ad7facb2586fc6e966c004d7d1d16b024f5805ff7cb47c7a85dabd8b48892ca7
Sep 26 00:43:21 systemd-coredump[5752]: Process 5751 (ostree) of user 0 terminated abnormally with signal 7/BUS, processing...
Sep 26 00:43:21 systemd[1]: Started systemd-coredump@1-5752-0.service - Process Core Dump (PID 5752/UID 0).
Sep 26 00:43:21 systemd-coredump[5753]: Resource limits disable core dumping for process 5751 (ostree).
Sep 26 00:43:21 systemd-coredump[5753]: Process 5751 (ostree) of user 0 terminated abnormally without generating a coredump.
Sep 26 00:43:21 systemd[1]: systemd-coredump@1-5752-0.service: Deactivated successfully.
Sep 26 00:43:54 kernel: fs-verity (vda4, inode 10909667): FILE CORRUPTED! pos=0, level=0, want_hash=sha256:dc33e0d22f2755a8e3081dea3393ea59aeb2bc5640e59b85a5220fa8b3abdd86, real_hash=sha256:ad7facb2586fc6e966c004d7d1d16b024f5805ff7cb47c7a85dabd8b48892ca7

All system logs in the past 90 days containing fs-verity with severity ≥error

Resolved paths of the above inodes

$ echo -n 10397939 10909667 10935659 10985929 9242655 | xargs --delimiter=' ' --replace='@' sudo btrfs inspect-internal inode-resolve @ /sysroot/
/sysroot//ostree/repo/objects/2d/d2a33a0a43236f1b1e2307fd52f08a89578145d67b96c38bd4ad8d75000e5a.file
/sysroot//ostree/deploy/fedora-iot/deploy/8faf3409a66b94d560eb20b6184b08a482d2b65138e4b6a359bd6f53e7536160.0/usr/lib64/libresolv.so.2
/sysroot//ostree/deploy/fedora-iot/deploy/cd5e58f02e387ea4faaaa09542b453602fbec43f4b8a35876a0463608d8e7eac.0/usr/lib64/libresolv.so.2
/sysroot//ostree/repo/objects/78/61de45bf7811062f6cf341c7999a7733d8cc7564abdc03bfc14cab434cd290.dirtree
/sysroot//ostree/repo/objects/88/425fd763f63cb030370dc0a39420ca1ee0121fc0dc592a61315b337d1f278b.dirtree
/sysroot//ostree/deploy/fedora-iot/deploy/cd5e58f02e387ea4faaaa09542b453602fbec43f4b8a35876a0463608d8e7eac.0/.ostree.cfs
/sysroot//ostree/repo/objects/42/873224adf940e85ee19e09a04afab671435bf238e805283253f0ed99579524.file
/sysroot//ostree/deploy/fedora-iot/deploy/8faf3409a66b94d560eb20b6184b08a482d2b65138e4b6a359bd6f53e7536160.0/usr/lib64/libssl.so.3.2.4
/sysroot//ostree/deploy/fedora-iot/deploy/cd5e58f02e387ea4faaaa09542b453602fbec43f4b8a35876a0463608d8e7eac.0/usr/lib64/libssl.so.3.2.4

The fs-verity errors caused new services to stop launching and prevented me from logging in, so I definitely need to fix this somehow. However, I have zero idea what to do here. Any suggestions? I have no problem building packages from source if you want me to test something out.

[gucci-on-fleek (Max Chernoff)]

Some more details:

Boot logs

Sep 26 03:33:59 kernel: Linux version 6.16.8-200.fc42.x86_64 (mockbuild@1b8d6d49127b4222833704415d76d564) (gcc (GCC) 15.2.1 20250808 (Red Hat 15.2.1-1), GNU ld version 2.44-6.fc42) #1 SMP PREEMPT_DYNAMIC Fri Sep 19 17:47:18 UTC 2025
Sep 26 03:33:59 kernel: Command line: BOOT_IMAGE=(hd0,gpt2)/ostree/fedora-iot-c7bf734d902a0ceffb45bd4d10c98cfbc2776cb49bbb675d048756f7ffc3e7ad/vmlinuz-6.16.8-200.fc42.x86_64 resume=UUID=25c85fb7-eadd-414c-9c56-475bf2a401f4 root=UUID=54b902ba-941a-495f-ae19-770ee22681c4 rw ostree=/ostree/boot.1/fedora-iot/c7bf734d902a0ceffb45bd4d10c98cfbc2776cb49bbb675d048756f7ffc3e7ad/0 verity=off
[...]
Sep 26 03:33:59 kernel: Kernel command line: BOOT_IMAGE=(hd0,gpt2)/ostree/fedora-iot-c7bf734d902a0ceffb45bd4d10c98cfbc2776cb49bbb675d048756f7ffc3e7ad/vmlinuz-6.16.8-200.fc42.x86_64 resume=UUID=25c85fb7-eadd-414c-9c56-475bf2a401f4 root=UUID=54b902ba-941a-495f-ae19-770ee22681c4 rw ostree=/ostree/boot.1/fedora-iot/c7bf734d902a0ceffb45bd4d10c98cfbc2776cb49bbb675d048756f7ffc3e7ad/0 verity=off
Sep 26 03:33:59 kernel: Unknown kernel command line parameters "BOOT_IMAGE=(hd0,gpt2)/ostree/fedora-iot-c7bf734d902a0ceffb45bd4d10c98cfbc2776cb49bbb675d048756f7ffc3e7ad/vmlinuz-6.16.8-200.fc42.x86_64 ostree=/ostree/boot.1/fedora-iot/c7bf734d902a0ceffb45bd4d10c98cfbc2776cb49bbb675d048756f7ffc3e7ad/0 verity=off", will be passed to user space.
[...]
Sep 26 03:33:59 kernel: device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.
Sep 26 03:33:59 kernel: device-mapper: uevent: version 1.0.3
Sep 26 03:33:59 kernel: device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev
[...]
Sep 26 03:33:59 kernel: Loading compiled-in X.509 certificates
Sep 26 03:33:59 kernel: Loaded X.509 cert 'Fedora kernel signing key: aa041064f004a5cc3ad2ddcc53da486c4a8821d6'
Sep 26 03:33:59 kernel: Loaded X.509 cert 'Fedora IMA CA: a8a00c31663f853f9c6ff2564872e378af026b28'
Sep 26 03:33:59 kernel: Demotion targets for Node 0: null
Sep 26 03:33:59 kernel: page_owner is disabled
Sep 26 03:33:59 kernel: Key type .fscrypt registered
Sep 26 03:33:59 kernel: Key type fscrypt-provisioning registered
Sep 26 03:33:59 kernel: Btrfs loaded, zoned=yes, fsverity=yes
Sep 26 03:33:59 kernel: Key type big_key registered
Sep 26 03:33:59 kernel: Key type encrypted registered
Sep 26 03:33:59 kernel: ima: No TPM chip found, activating TPM-bypass!
Sep 26 03:33:59 kernel: Loading compiled-in module X.509 certificates
Sep 26 03:33:59 kernel: Loaded X.509 cert 'Fedora kernel signing key: aa041064f004a5cc3ad2ddcc53da486c4a8821d6'
Sep 26 03:33:59 kernel: ima: Allocated hash algorithm: sha256
Sep 26 03:33:59 kernel: ima: No architecture policies found
Sep 26 03:33:59 kernel: evm: Initialising EVM extended attributes:
Sep 26 03:33:59 kernel: evm: security.selinux
Sep 26 03:33:59 kernel: evm: security.SMACK64 (disabled)
Sep 26 03:33:59 kernel: evm: security.SMACK64EXEC (disabled)
Sep 26 03:33:59 kernel: evm: security.SMACK64TRANSMUTE (disabled)
Sep 26 03:33:59 kernel: evm: security.SMACK64MMAP (disabled)
Sep 26 03:33:59 kernel: evm: security.apparmor (disabled)
Sep 26 03:33:59 kernel: evm: security.ima
Sep 26 03:33:59 kernel: evm: security.capability
Sep 26 03:33:59 kernel: evm: HMAC attrs: 0x1
Sep 26 03:33:59 kernel: alg: No test for 842 (842-scomp)
Sep 26 03:33:59 kernel: PM:   Magic number: 9:954:576
Sep 26 03:33:59 kernel: RAS: Correctable Errors collector initialized.
Sep 26 03:33:59 kernel: ata1: found unknown device (class 0)
Sep 26 03:33:59 kernel: ata1.00: ATAPI: QEMU DVD-ROM, 2.5+, max UDMA/100
Sep 26 03:33:59 kernel: scsi 0:0:0:0: CD-ROM            QEMU     QEMU DVD-ROM     2.5+ PQ: 0 ANSI: 5
Sep 26 03:33:59 kernel: sr 0:0:0:0: [sr0] scsi3-mmc drive: 4x/4x cd/rw xa/form2 tray
Sep 26 03:33:59 kernel: cdrom: Uniform CD-ROM driver Revision: 3.20
Sep 26 03:33:59 kernel: sr 0:0:0:0: Attached scsi CD-ROM sr0
Sep 26 03:33:59 kernel: sr 0:0:0:0: Attached scsi generic sg0 type 5
[...]
Sep 26 03:33:59 kernel: Run /init as init process
Sep 26 03:33:59 kernel:   with arguments:
Sep 26 03:33:59 kernel:     /init
Sep 26 03:33:59 kernel:   with environment:
Sep 26 03:33:59 kernel:     HOME=/
Sep 26 03:33:59 kernel:     TERM=linux
Sep 26 03:33:59 kernel:     BOOT_IMAGE=(hd0,gpt2)/ostree/fedora-iot-c7bf734d902a0ceffb45bd4d10c98cfbc2776cb49bbb675d048756f7ffc3e7ad/vmlinuz-6.16.8-200.fc42.x86_64
Sep 26 03:33:59 kernel:     ostree=/ostree/boot.1/fedora-iot/c7bf734d902a0ceffb45bd4d10c98cfbc2776cb49bbb675d048756f7ffc3e7ad/0
Sep 26 03:33:59 kernel:     verity=off
Sep 26 03:33:59 systemd[1]: Successfully made /usr/ read-only.
Sep 26 03:33:59 systemd[1]: systemd 257.9-2.fc42 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +IPE +SMACK +SECCOMP -GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +BTF +XKBCOMMON +UTMP +SYSVINIT +LIBARCHIVE)
[...]
Sep 26 03:33:59 systemd[1]: Expecting device dev-disk-by\x2duuid-25c85fb7\x2deadd\x2d414c\x2d9c56\x2d475bf2a401f4.device - /dev/disk/by-uuid/25c85fb7-eadd-414c-9c56-475bf2a401f4...
Sep 26 03:33:59 systemd[1]: Expecting device dev-disk-by\x2duuid-54b902ba\x2d941a\x2d495f\x2dae19\x2d770ee22681c4.device - /dev/disk/by-uuid/54b902ba-941a-495f-ae19-770ee22681c4...
Sep 26 03:33:59 systemd[1]: Reached target initrd-usr-fs.target - Initrd /usr File System.
[...]
Sep 26 03:33:59 dracut-cmdline[265]: Using kernel command line parameters:  rd.driver.pre=btrfs   BOOT_IMAGE=(hd0,gpt2)/ostree/fedora-iot-c7bf734d902a0ceffb45bd4d10c98cfbc2776cb49bbb675d048756f7ffc3e7ad/vmlinuz-6.16.8-200.fc42.x86_64 resume=UUID=25c85fb7-eadd-414c-9c56-475bf2a401f4 root=UUID=54b902ba-941a-495f-ae19-770ee22681c4 rw ostree=/ostree/boot.1/fedora-iot/c7bf734d902a0ceffb45bd4d10c98cfbc2776cb49bbb675d048756f7ffc3e7ad/0 verity=off
[...]
Sep 26 03:33:59 systemd[1]: Found device dev-disk-by\x2duuid-54b902ba\x2d941a\x2d495f\x2dae19\x2d770ee22681c4.device - /dev/disk/by-uuid/54b902ba-941a-495f-ae19-770ee22681c4.
Sep 26 03:33:59 systemd[1]: Found device dev-disk-by\x2duuid-25c85fb7\x2deadd\x2d414c\x2d9c56\x2d475bf2a401f4.device - /dev/disk/by-uuid/25c85fb7-eadd-414c-9c56-475bf2a401f4.
Sep 26 03:33:59 systemd[1]: Reached target initrd-root-device.target - Initrd Root Device.
Sep 26 03:33:59 systemd[1]: Starting systemd-hibernate-resume.service - Resume from hibernation...
Sep 26 03:33:59 kernel: PM: Image not found (code -22)
Sep 26 03:33:59 systemd[1]: systemd-hibernate-resume.service: Deactivated successfully.
Sep 26 03:33:59 systemd[1]: Finished systemd-hibernate-resume.service - Resume from hibernation.
Sep 26 03:33:59 systemd[1]: Reached target local-fs-pre.target - Preparation for Local File Systems.
Sep 26 03:33:59 systemd[1]: Reached target local-fs.target - Local File Systems.
[...]
Sep 26 03:33:59 systemd[1]: Starting dracut-pre-mount.service - dracut pre-mount hook...
Sep 26 03:33:59 systemd[1]: Finished systemd-vconsole-setup.service - Virtual Console Setup.
Sep 26 03:33:59 systemd[1]: Finished dracut-pre-mount.service - dracut pre-mount hook.
Sep 26 03:33:59 systemd[1]: Starting systemd-fsck-root.service - File System Check on /dev/disk/by-uuid/54b902ba-941a-495f-ae19-770ee22681c4...
Sep 26 03:33:59 systemd[1]: Finished systemd-fsck-root.service - File System Check on /dev/disk/by-uuid/54b902ba-941a-495f-ae19-770ee22681c4.
Sep 26 03:34:00 systemd[1]: Mounting sys-kernel-config.mount - Kernel Configuration File System...
Sep 26 03:34:00 systemd[1]: Mounting sysroot.mount - /sysroot...
Sep 26 03:34:00 systemd[1]: Mounted sys-kernel-config.mount - Kernel Configuration File System.
Sep 26 03:34:00 systemd[1]: Reached target sysinit.target - System Initialization.
Sep 26 03:34:00 systemd[1]: Reached target basic.target - Basic System.
Sep 26 03:34:00 kernel: BTRFS: device fsid 54b902ba-941a-495f-ae19-770ee22681c4 devid 1 transid 1110634 /dev/vda4 (253:4) scanned by mount (434)
Sep 26 03:34:00 kernel: BTRFS info (device vda4): first mount of filesystem 54b902ba-941a-495f-ae19-770ee22681c4
Sep 26 03:34:00 kernel: BTRFS info (device vda4): using crc32c (crc32c-x86_64) checksum algorithm
Sep 26 03:34:00 kernel: BTRFS info (device vda4): turning on async discard
Sep 26 03:34:00 kernel: BTRFS info (device vda4): enabling free space tree
Sep 26 03:34:00 systemd[1]: Mounted sysroot.mount - /sysroot.
Sep 26 03:34:00 systemd[1]: Starting ostree-prepare-root.service - OSTree Prepare OS/...
Sep 26 03:34:00 ostree-prepare-root[452]: Resolved OSTree target to: /sysroot/ostree/deploy/fedora-iot/deploy/cd5e58f02e387ea4faaaa09542b453602fbec43f4b8a35876a0463608d8e7eac.0
Sep 26 03:34:00 ostree-prepare-root[452]: sysroot.readonly configuration value: 1 (fs writable: 1)
Sep 26 03:34:00 kernel: fs-verity: sha256 using implementation "sha256-x86_64"
Sep 26 03:34:00 kernel: netfs: FS-Cache loaded
Sep 26 03:34:00 kernel: erofs (device erofs): mounted with root inode @ nid 36.
Sep 26 03:34:00 ostree-prepare-root[452]: composefs: mounted successfully (verity=false)
Sep 26 03:34:00 systemd[1]: run-ostree-.private-cfsroot\x2dlower.mount: Deactivated successfully.
Sep 26 03:34:00 ostree-prepare-root[452]: Resolved OSTree target to: /sysroot.tmp/sysroot/ostree/deploy/fedora-iot/deploy/cd5e58f02e387ea4faaaa09542b453602fbec43f4b8a35876a0463608d8e7eac.0
Sep 26 03:34:00 systemd[1]: Finished ostree-prepare-root.service - OSTree Prepare OS/.
Sep 26 03:34:00 systemd[1]: Reached target initrd-root-fs.target - Initrd Root File System.
Sep 26 03:34:00 systemd[1]: Starting initrd-parse-etc.service - Mountpoints Configured in the Real Root...
Sep 26 03:34:00 systemd[1]: initrd-parse-etc.service: Deactivated successfully.
Sep 26 03:34:00 systemd[1]: Finished initrd-parse-etc.service - Mountpoints Configured in the Real Root.
Sep 26 03:34:00 systemd[1]: Reached target initrd-fs.target - Initrd File Systems.
Sep 26 03:34:00 systemd[1]: Reached target initrd.target - Initrd Default Target.
Sep 26 03:34:00 systemd[1]: dracut-mount.service - dracut mount hook was skipped because no trigger condition checks were met.
Sep 26 03:34:00 systemd[1]: Starting dracut-pre-pivot.service - dracut pre-pivot and cleanup hook...
Sep 26 03:34:00 systemd[1]: Finished dracut-pre-pivot.service - dracut pre-pivot and cleanup hook.
Sep 26 03:34:00 systemd[1]: Starting initrd-cleanup.service - Cleaning Up and Shutting Down Daemons...
Sep 26 03:34:00 systemd[1]: Stopped target remote-cryptsetup.target - Remote Encrypted Volumes.
Sep 26 03:34:00 systemd[1]: Stopped target timers.target - Timer Units.
Sep 26 03:34:00 systemd[1]: dracut-pre-pivot.service: Deactivated successfully.
Sep 26 03:34:00 systemd[1]: Stopped dracut-pre-pivot.service - dracut pre-pivot and cleanup hook.
Sep 26 03:34:00 systemd[1]: Stopped target initrd.target - Initrd Default Target.
Sep 26 03:34:00 systemd[1]: Stopped target basic.target - Basic System.
Sep 26 03:34:00 systemd[1]: Stopped target initrd-root-device.target - Initrd Root Device.
[...]
Sep 26 03:34:00 systemd[1]: Starting initrd-switch-root.service - Switch Root...
Sep 26 03:34:00 systemd[1]: Switching root.
Sep 26 03:34:00 systemd-journald[231]: Journal stopped
Sep 26 03:34:01 systemd-journald[231]: Received SIGTERM from PID 1 (systemd).
[...]
Sep 26 03:34:01 systemd[1]: initrd-switch-root.service: Deactivated successfully.
Sep 26 03:34:01 systemd[1]: Stopped initrd-switch-root.service - Switch Root.
[...]
Sep 26 03:34:01 systemd[1]: Expecting device dev-disk-by\x2duuid-25c85fb7\x2deadd\x2d414c\x2d9c56\x2d475bf2a401f4.device - /dev/disk/by-uuid/25c85fb7-eadd-414c-9c56-475bf2a401f4...
Sep 26 03:34:01 systemd[1]: Expecting device dev-disk-by\x2duuid-26d65031\x2dc51b\x2d4e2c\x2d999f\x2d9642f90ebcab.device - /dev/disk/by-uuid/26d65031-c51b-4e2c-999f-9642f90ebcab...
Sep 26 03:34:01 systemd[1]: Expecting device dev-disk-by\x2duuid-312A\x2d0AD0.device - /dev/disk/by-uuid/312A-0AD0...
Sep 26 03:34:01 systemd[1]: Expecting device dev-disk-by\x2duuid-54b902ba\x2d941a\x2d495f\x2dae19\x2d770ee22681c4.device - /dev/disk/by-uuid/54b902ba-941a-495f-ae19-770ee22681c4...
Sep 26 03:34:01 systemd[1]: Expecting device dev-zram0.device - /dev/zram0...
Sep 26 03:34:01 systemd[1]: Reached target bootc-status-updated-onboot.target - Target for bootc status changes on boot.
Sep 26 03:34:01 systemd[1]: Reached target cryptsetup.target - Local Encrypted Volumes.
Sep 26 03:34:01 systemd[1]: Stopped target initrd-switch-root.target - Switch Root.
Sep 26 03:34:01 systemd[1]: Stopped target initrd-fs.target - Initrd File Systems.
Sep 26 03:34:01 systemd[1]: Stopped target initrd-root-fs.target - Initrd Root File System.
Sep 26 03:34:01 systemd[1]: Reached target integritysetup.target - Local Integrity Protected Volumes.
Sep 26 03:34:01 systemd[1]: Reached target remote-cryptsetup.target - Remote Encrypted Volumes.
Sep 26 03:34:01 systemd[1]: Reached target remote-fs.target - Remote File Systems.
Sep 26 03:34:01 systemd[1]: Reached target slices.target - Slice Units.
Sep 26 03:34:01 systemd[1]: Reached target time-set.target - System Time Set.
Sep 26 03:34:01 systemd[1]: Reached target veritysetup.target - Local Verity Protected Volumes.
[...]
Sep 26 03:34:01 systemd[1]: ostree-prepare-root.service: Deactivated successfully.
Sep 26 03:34:01 systemd[1]: Stopped ostree-prepare-root.service - OSTree Prepare OS/.
Sep 26 03:34:01 systemd[1]: systemd-fsck-root.service: Deactivated successfully.
Sep 26 03:34:01 systemd[1]: Stopped systemd-fsck-root.service - File System Check on Root Device.

Another shell transcript while debugging

$ cat /etc/ostree/prepare-root.conf
[sysroot]
readonly = true
[composefs]
enabled = yes

$ find /proc/ /sys/ -name '*verity*' -exec sh -c 'echo {}; ls -la {}; cat {}'  \; 2>/dev/null
/proc/sys/fs/verity
total 0
dr-xr-xr-x. 1 root root 0 Sep 26 03:57 .
dr-xr-xr-x. 1 root root 0 Sep 26 03:33 ..
/sys/kernel/slab/fsverity_info
total 0
drwxr-xr-x.   2 root root    0 Sep 26 03:59 .
drwxr-xr-x. 519 root root    0 Sep 26 03:43 ..
-r--------.   1 root root 4096 Sep 26 03:59 aliases
-r--------.   1 root root 4096 Sep 26 03:59 align
-r--------.   1 root root 4096 Sep 26 03:59 cache_dma
-rw-------.   1 root root 4096 Sep 26 03:59 cpu_partial
-r--------.   1 root root 4096 Sep 26 03:59 cpu_slabs
-r--------.   1 root root 4096 Sep 26 03:59 ctor
-r--------.   1 root root 4096 Sep 26 03:59 destroy_by_rcu
-r--------.   1 root root 4096 Sep 26 03:59 hwcache_align
-rw-------.   1 root root 4096 Sep 26 03:59 min_partial
-r--------.   1 root root 4096 Sep 26 03:59 object_size
-r--------.   1 root root 4096 Sep 26 03:59 objects
-r--------.   1 root root 4096 Sep 26 03:59 objects_partial
-r--------.   1 root root 4096 Sep 26 03:59 objs_per_slab
-r--------.   1 root root 4096 Sep 26 03:59 order
-r--------.   1 root root 4096 Sep 26 03:59 partial
-r--------.   1 root root 4096 Sep 26 03:59 poison
-r--------.   1 root root 4096 Sep 26 03:59 reclaim_account
-r--------.   1 root root 4096 Sep 26 03:59 red_zone
-rw-------.   1 root root 4096 Sep 26 03:59 remote_node_defrag_ratio
-r--------.   1 root root 4096 Sep 26 03:59 sanity_checks
-rw-------.   1 root root 4096 Sep 26 03:59 shrink
-rw-------.   1 root root 4096 Sep 26 03:59 skip_kfence
-r--------.   1 root root 4096 Sep 26 03:59 slab_size
-r--------.   1 root root 4096 Sep 26 03:59 slabs
-r--------.   1 root root 4096 Sep 26 03:59 slabs_cpu_partial
-r--------.   1 root root 4096 Sep 26 03:59 store_user
-r--------.   1 root root 4096 Sep 26 03:59 total_objects
-r--------.   1 root root 4096 Sep 26 03:59 trace
-r--------.   1 root root 4096 Sep 26 03:59 usersize
-rw-------.   1 root root 4096 Sep 26 03:59 validate
/sys/fs/btrfs/54b902ba-941a-495f-ae19-770ee22681c4/features/verity
-r--r--r--. 1 root root 4096 Sep 26 03:57 /sys/fs/btrfs/54b902ba-941a-495f-ae19-770ee22681c4/features/verity
1
/sys/fs/btrfs/features/verity
-r--r--r--. 1 root root 4096 Sep 26 03:57 /sys/fs/btrfs/features/verity
0
/sys/fs/ext4/features/verity
-r--r--r--. 1 root root 4096 Sep 26 03:57 /sys/fs/ext4/features/verity
supported

$ sudo mount --options=remount,verity=off /
mount: /: fsconfig system call failed: overlay: No changes allowed in reconfigure.
       dmesg(1) may have more information after failed mount system call.

The weird thing is that running sudo works fine right after I reboot, but at some random time later (from 10 minutes to 18 hours), it stops working with the following error:

fish: Job 1, 'sudo [...]' terminated by signal SIGBUS (Misaligned address error)

The system logs show that the offending file is

kernel: fs-verity (vda4, inode 9242655): FILE CORRUPTED! pos=839680, level=1, want_hash=sha256:2ab94382e1e1dfdbf65e6f0dae7ace0e1ebdcea1da2eb5488087b6a2c6700925, real_hash=sha256:ad7facb2586fc6e966c004d7d1d16b024f5805ff7cb47c7a85dabd8b48892ca7

which resolves to

/sysroot//ostree/repo/objects/42/873224adf940e85ee19e09a04afab671435bf238e805283253f0ed99579524.file
/sysroot//ostree/deploy/fedora-iot/deploy/8faf3409a66b94d560eb20b6184b08a482d2b65138e4b6a359bd6f53e7536160.0/usr/lib64/libssl.so.3.2.4
/sysroot//ostree/deploy/fedora-iot/deploy/cd5e58f02e387ea4faaaa09542b453602fbec43f4b8a35876a0463608d8e7eac.0/usr/lib64/libssl.so.3.2.4

cating each of those files gives a Input/output error whenever sudo stops working. But immediately after rebooting, I get the following:

$ cat /etc/shadow | head -1
cat: /etc/shadow: Permission denied

$ sudo cat /etc/shadow | head -1
root:!::0:99999:7:::

$ sha256sum /usr/lib64/libssl.so.3.2.4
ce521af2b2e7beca94577d39a62dac580084255b0f5837d0ec0069e030f451d9  /usr/lib64/libssl.so.3.2.4

$ sha256sum /usr/bin/nl
sha256sum: /usr/bin/nl: Input/output error

[some time later]

$ sudo cat /etc/shadow | head -1
fish: Process 5568, 'sudo' from job 1, 'sudo cat /etc/shadow | head -1' terminated by signal SIGBUS (Misaligned address error)

$ sha256sum /usr/lib64/libssl.so.3.2.4
sha256sum: /usr/lib64/libssl.so.3.2.4: Input/output error

$ sha256sum /usr/bin/nl
sha256sum: /usr/bin/nl: Input/output error

So /usr/bin/nl is always broken, but /usr/lib64/libssl.so.3.2.4 only breaks some time after boot. I’m able to trigger things breaking by running

$ grep -Rn zzzzzzzzzzzzzzzzzzzzz /ostree/ 2>/dev/null

so it seems like lots of filesystem operations trigger the problem.

However, rebooting into the previous ostree deployment seems to have fixed things, even though the “corrupt” files are completely unchanged

$ sha256sum /sysroot//ostree/repo/objects/78/7cbe7098e399806d4ec03e32e882ac8b724604c9dd6ecb6cb35956ce77bc33.file /sysroot//ostree/deploy/fedora-iot/deploy/8faf3409a66b94d560eb20b6184b08a482d2b65138e4b6a359bd6f53e7536160.0/usr/bin/nl /sysroot//ostree/deploy/fedora-iot/deploy/cd5e58f02e387ea4faaaa09542b453602fbec43f4b8a35876a0463608d8e7eac.0/usr/bin/nl /usr/bin/nl
99a501630a23a250ee8b486e3fbc911c68e474fb51d46b1a96a1a2387ecd5d6d  /sysroot//ostree/repo/objects/78/7cbe7098e399806d4ec03e32e882ac8b724604c9dd6ecb6cb35956ce77bc33.file
99a501630a23a250ee8b486e3fbc911c68e474fb51d46b1a96a1a2387ecd5d6d  /sysroot//ostree/deploy/fedora-iot/deploy/8faf3409a66b94d560eb20b6184b08a482d2b65138e4b6a359bd6f53e7536160.0/usr/bin/nl
99a501630a23a250ee8b486e3fbc911c68e474fb51d46b1a96a1a2387ecd5d6d  /sysroot//ostree/deploy/fedora-iot/deploy/cd5e58f02e387ea4faaaa09542b453602fbec43f4b8a35876a0463608d8e7eac.0/usr/bin/nl
99a501630a23a250ee8b486e3fbc911c68e474fb51d46b1a96a1a2387ecd5d6d  /usr/bin/nl

diffing the package lists between the two deployments gives me the following

 kbd                              2.7.1
 kbd-legacy                       2.7.1
 kbd-misc                         2.7.1
-kernel                           6.16.7
-kernel-core                      6.16.7
-kernel-modules                   6.16.7
-kernel-modules-core              6.16.7
+kernel                           6.16.8
+kernel-core                      6.16.8
+kernel-modules                   6.16.8
+kernel-modules-core              6.16.8
 keyutils-libs                    1.6.3
 kmod                             33
 kmod-libs                        33

which suggests to me that this might be a kernel bug.

[siosm (Timothée Ravier)]

It would be good to report that in bugzilla for the kernel

[chrismurphy]

I mentioned this on https://matrix.to/#/#btrfs:fedoraproject.org and a Btrfs developer who worked on fs-verity support said the code hasn’t changed in a long time.

I suggest filing a kernel bug. Please attach a complete dmesg for both 6.16.7 and 6.16.8 to the bug report. To find out which boot maps to which kernel version

journalctl -b-1
journalctl -b-2

You can look at the first lines to see kernel version. Let’s say -b-1 uses 6.16.7 and -b-2 uses 6.17.8.

journalctl -b-1 -k -o short-monotonic --no-hostname > dmesg-6.16.7.log
journalctl -b-2 -k -o short-monotonic --no-hostname > dmesg-6.16.8.log

---

Update 1: I don’t see anything obvious in https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.16.8 that would give a clue what’s going on. Typically this calls for kernel bisect to find out what committ introduced the problem.

Update 2: Also, if you do file a bug, post the URL here or in the Fedora Btrfs matrix room so we can loop in more btrfs and fsdevel folks.

[gucci-on-fleek (Max Chernoff)]

Ok, I’ve opened a new bug here: 2399898 – fs-verity "FILE CORRUPTED!" errors on ostree files after upgrading to kernel 6.16.8

Thanks!

[gucci-on-fleek (Max Chernoff)]

This has been resolved in btrfs: fix incorrect readahead expansion length. Thanks again for all your help!