RSYNC reports input/output error and btrfs source has file read error but check ok

Ask Fedora
, ,
3

btrfs scrub found errors. Will next run smartctl. Disk showing uncorrectable errors is brand new???

# btrfs scrub start -Bdrf /srv/lpssd/archives

Scrub device /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf (id 1) done
Scrub started:    Thu Aug  3 10:55:40 2023
Status:           finished
Duration:         0:44:15
Total to scrub:   690.92GiB
Rate:             266.48MiB/s
Error summary:    csum=8
  Corrected:      0
  Uncorrectable:  8
  Unverified:     0

Scrub device /dev/mapper/luks-1e32597b-d73f-48de-a899-af443d96b150 (id 2) done
Scrub started:    Thu Aug  3 10:55:40 2023
Status:           finished
Duration:         0:16:35
Total to scrub:   86.82GiB
Rate:             89.35MiB/s
Error summary:    no errors found

Scrub device /dev/mapper/luks-9b746590-b83b-4e35-bb7c-1c07d5130a4f (id 3) done
Scrub started:    Thu Aug  3 10:55:40 2023
Status:           finished
Duration:         0:17:28
Total to scrub:   88.81GiB
Rate:             86.78MiB/s
Error summary:    no errors found
ERROR: there are uncorrectable errors
4

Smarctl tests results have always been a mystery for me to interpret. I have googled this it it always seemed like goobldegoop explanations.

Gsmarctl test results

# smartctl -a /dev/sda
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.4.4-200.fc38.x86_64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     T-FORCE 1TB
Serial Number:    TPBF2212050020101379
Firmware Version: V0718B0
User Capacity:    1,024,209,543,168 bytes [1.02 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
TRIM Command:     Available
Device is:        Not in smartctl database 7.3/5417
ATA Version is:   ACS-2 T13/2015-D revision 3
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Aug  3 16:32:07 2023 AEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x02) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                (  120) seconds.
Offline data collection
capabilities:                    (0x11) SMART execute Offline immediate.
                                        No Auto Offline data collection support.
                                        Suspend Offline collection upon new
                                        command.
                                        No Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        No Selective Self-test supported.
SMART capabilities:            (0x0002) Does not save SMART data before
                                        entering power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        (  10) minutes.
SCT capabilities:              (0x0001) SCT Status supported.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x0032   100   100   050    Old_age   Always       -       0
  5 Reallocated_Sector_Ct   0x0032   100   100   050    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   050    Old_age   Always       -       2931
 12 Power_Cycle_Count       0x0032   100   100   050    Old_age   Always       -       157
160 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       0
161 Unknown_Attribute       0x0033   100   100   050    Pre-fail  Always       -       100
163 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       16
164 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       17369
165 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       44
166 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       10
167 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       26
168 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       5050
169 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       100
175 Program_Fail_Count_Chip 0x0032   100   100   050    Old_age   Always       -       0
176 Erase_Fail_Count_Chip   0x0032   100   100   050    Old_age   Always       -       0
177 Wear_Leveling_Count     0x0032   100   100   050    Old_age   Always       -       0
178 Used_Rsvd_Blk_Cnt_Chip  0x0032   100   100   050    Old_age   Always       -       0
181 Program_Fail_Cnt_Total  0x0032   100   100   050    Old_age   Always       -       0
182 Erase_Fail_Count_Total  0x0032   100   100   050    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   050    Old_age   Always       -       59
194 Temperature_Celsius     0x0022   100   100   050    Old_age   Always       -       14
195 Hardware_ECC_Recovered  0x0032   100   100   050    Old_age   Always       -       0
196 Reallocated_Event_Count 0x0032   100   100   050    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   100   100   050    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0032   100   100   050    Old_age   Always       -       0
199 UDMA_CRC_Error_Count    0x0032   100   100   050    Old_age   Always       -       0
232 Available_Reservd_Space 0x0032   100   100   050    Old_age   Always       -       100
241 Total_LBAs_Written      0x0030   100   100   050    Old_age   Offline      -       280052
242 Total_LBAs_Read         0x0030   100   100   050    Old_age   Offline      -       527087
245 Unknown_Attribute       0x0032   100   100   050    Old_age   Always       -       618960

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      2931         -
# 2  Short offline       Completed without error       00%      2303         -
# 3  Extended offline    Aborted by host               90%        28         -
# 4  Extended offline    Aborted by host               90%        28         -
# 5  Extended offline    Completed without error       00%        27         -
# 6  Short offline       Completed without error       00%        27         -

Selective Self-tests/Logging not supported

as usual in my lack of understanding what does this mean? Note has 3 partitions sda[1…3] where sda3 is the btrfs volume merged with two other partions on another volume is the one in question.

Sigh – help re the meaning of this?

5

Do you see disk errors in dmesg?
That is typically where i have seen hardware issues reported by the kernel.

6

Apparently, the disk is not yet known in smartctl’s database, what explains a number of unknown attributes, but what you see is that the disk in rather new, everything is fine but it’s cold in your room, at least according to the disk’s sensor.

I do not expect serious messages from the disk itself in the journal, but it’s good to check it.

So this might be a case of btrfs checksum mismatch, leading to the question whether the data are corrupt or the checksum… I hope there is someone here knowing how to read data from btrfs overriding the checksum and let the application or user decide whether the data are correct.
If not, there is not more left than deleting the file and restore a backup version for the moment. Fortunately, rsync first creates a temporary file on the backup and replaces it only after succesful copy, so the last backup should be intact.

7

Hello @rkoppelh ,
Have tried to use Borg Backup to move the backup copy. I thought it had such a function.

8

Note also I forgot to mention that the sda disk on which I am getting errors has sda3 as mentioned merged with two smaller partitions on a 256 GB older SSD. Above error reported on newer sda3.

Adding a lyer of complexity is that the merged btrfs volume is luks1 encrypted and automounted by fstab, i.e.

/dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf /srv/lpssd/archives btrfs defaults,compress=zstd:9,discard=async,x-systemd.device-timeout=0 0 2

with /etc/crypttab

$ sudo cat /etc/crypttab
...
# RK 19/4/2023: btrfs, archive <- archive2 <-archive3 refer /etc/fstab
luks-dd129b86-469d-4042-8b90-00d1acb2a1bf UUID=dd129b86-469d-4042-8b90-00d1acb2a1bf /root/Documents/encrypted-disks/archive-key luks,nofail,discard
luks-9b746590-b83b-4e35-bb7c-1c07d5130a4f UUID=9b746590-b83b-4e35-bb7c-1c07d5130a4f /root/Documents/encrypted-disks/archive2-key luks,nofail,discard
luks-1e32597b-d73f-48de-a899-af443d96b150 UUID=1e32597b-d73f-48de-a899-af443d96b150 /root/Documents/encrypted-disks/archive3-key luks,nofail,discard
...

While the lsblk disk structure for btrfs backup volume is merged sda3, sdb1, sdb2

sda                                                                                                                     
├─sda1                         vfat        FAT32    ESP           3826-001F                                 1.4G     2% /boot/efi
├─sda2                         ext4        1.0      boot          667dcfd4-ff09-4f49-a7a6-08e6c7b49639    722.2M    44% /boot
└─sda3                         crypto_LUKS 1                      dd129b86-469d-4042-8b90-00d1acb2a1bf                  
  └─luks-dd129b86-469d-4042-8b90-00d1acb2a1bf
                               btrfs                archive       2d85cf85-6165-49a8-8fd1-68651fe12fdc                  /srv/lpssd/archives
sdb                                                                                                                     
├─sdb1                         crypto_LUKS 1                      1e32597b-d73f-48de-a899-af443d96b150                  
│ └─luks-1e32597b-d73f-48de-a899-af443d96b150
│                              btrfs                archive       2d85cf85-6165-49a8-8fd1-68651fe12fdc                  
└─sdb2                         crypto_LUKS 1                      9b746590-b83b-4e35-bb7c-1c07d5130a4f                  
  └─luks-9b746590-b83b-4e35-bb7c-1c07d5130a4f
                               btrfs                archive       2d85cf85-6165-49a8-8fd1-68651fe12fdc

NOW re the above comment re dmesg, for the disk T-FORCE I get:

$ dmesg | grep error
[    5.775907] sas: ata7: end_device-7:0: dev error handler
[    5.840280] sas: ata8: end_device-8:0: dev error handler
[    6.095538] sas: ata8: end_device-8:0: cmd error handler
[    6.095566] sas: ata8: end_device-8:0: dev error handler
[    6.308785] sas: ata8: end_device-8:0: cmd error handler
[    6.308843] sas: ata8: end_device-8:0: dev error handler
[    6.333809] sas: ata8: end_device-8:0: cmd error handler
[    6.333869] sas: ata8: end_device-8:0: dev error handler
[    6.392722] sas: ata8: end_device-8:0: cmd error handler
[    6.392784] sas: ata8: end_device-8:0: dev error handler
[   11.416595] sas: ata8: end_device-8:0: cmd error handler
[   11.416647] sas: ata8: end_device-8:0: dev error handler
[   11.445685] sas: ata8: end_device-8:0: cmd error handler
[   11.445723] sas: ata8: end_device-8:0: dev error handler
[   11.490737] sas: ata8: end_device-8:0: cmd error handler
[   11.490776] sas: ata8: end_device-8:0: dev error handler
[   11.547742] sas: ata8: end_device-8:0: cmd error handler
[   11.547794] sas: ata8: end_device-8:0: dev error handler
[   11.612727] sas: ata8: end_device-8:0: cmd error handler
[   11.612791] sas: ata8: end_device-8:0: dev error handler
[   24.084697] sas: ata8: end_device-8:0: cmd error handler
[   24.084708] sas: ata8: end_device-8:0: dev error handler
[   28.326576] audit: type=1338 audit(1691104184.198:56): module=crypt op=ctr ppid=1 pid=1422 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="systemd-cryptse" exe="/usr/lib/systemd/systemd-cryptsetup" subj=system_u:system_r:lvm_t:s0 dev=253:11 error_msg='success' res=1

[robertk@earth:~]$ dmesg | grep Error
[    2.584766] ERST: Error Record Serialization Table (ERST) support is initialized.
[    2.989070] RAS: Correctable Errors collector initialized.

[robertk@earth:~]$ dmesg | grep Warning
[    0.012202] ACPI BIOS Warning (bug): Invalid length for FADT/Pm1aControlBlock: 32, using default 16 (20230331/tbfadt-669)

[robertk@earth:~]$ dmesg | grep warning

[robertk@earth:~]$ dmesg | grep T-F
[    2.929265] ata1.00: ATA-9: T-FORCE 1TB, V0718B0, max UDMA/133
[    2.937528] scsi 0:0:0:0: Direct-Access     ATA      T-FORCE 1TB      8B0  PQ: 0 ANSI: 5

[robertk@earth:~]$ dmesg | grep T-FORCE
[    2.929265] ata1.00: ATA-9: T-FORCE 1TB, V0718B0, max UDMA/133
[    2.937528] scsi 0:0:0:0: Direct-Access     ATA      T-FORCE 1TB      8B0  PQ: 0 ANSI: 5

[robertk@earth:~]$ dmesg | grep '/dev'
[    0.000000] Command line: BOOT_IMAGE=(hd1,gpt2)/vmlinuz-6.4.7-200.fc38.x86_64 root=/dev/mapper/VG01_nvme_pcie-rootfs ro rd.driver.blacklist=nouveau modprobe.blacklist=nouveau rd.lvm.lv=VG01_nvme_pcie/rootfs rd.luks.uuid=luks-160cee22-ab53-47b2-a48a-382fca72928a rhgb nvidia-drm.modeset=1 initcall_blacklist=simpledrm_platform_driver_init rd.driver.blacklist=nouveau modprobe.blacklist=nouveau
[    0.309212] Kernel command line: BOOT_IMAGE=(hd1,gpt2)/vmlinuz-6.4.7-200.fc38.x86_64 root=/dev/mapper/VG01_nvme_pcie-rootfs ro rd.driver.blacklist=nouveau modprobe.blacklist=nouveau rd.lvm.lv=VG01_nvme_pcie/rootfs rd.luks.uuid=luks-160cee22-ab53-47b2-a48a-382fca72928a rhgb nvidia-drm.modeset=1 initcall_blacklist=simpledrm_platform_driver_init rd.driver.blacklist=nouveau modprobe.blacklist=nouveau
[    2.554497] input: Sleep Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0E:00/input/input0
[    2.554792] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input1
[    3.587877] input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb3/3-1/3-1.4/3-1.4:1.0/0003:046B:FF10.0001/input/input2
[    3.591407] input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb3/3-1/3-1.4/3-1.4:1.1/0003:046B:FF10.0002/input/input3
[    3.720017] systemd[1]: Listening on systemd-journald-dev-log.socket - Journal Socket (/dev/log).
[    3.808870] input: CHICONY HP Basic USB Keyboard as /devices/pci0000:00/0000:00:1d.0/usb3/3-1/3-1.2/3-1.2.2/3-1.2.2:1.0/0003:03F0:0024.0003/input/input4
[    3.840845] systemd[1]: Starting systemd-tmpfiles-setup-dev.service - Create Static Device Nodes in /dev...
[    4.043629] input: Logitech Optical USB Mouse as /devices/pci0000:00/0000:00:1d.0/usb3/3-1/3-1.2/3-1.2.3/3-1.2.3:1.0/0003:046D:C016.0004/input/input5
[   22.669586] systemd[1]: Relabelled /dev, /dev/shm, /run, /sys/fs/cgroup in 90.460ms.
[   23.478730] systemd[1]: Reached target blockdev@dev-mapper-luks\x2d160cee22\x2dab53\x2d47b2\x2da48a\x2d382fca72928a.target - Block Device Preparation for /dev/mapper/luks-160cee22-ab53-47b2-a48a-382fca72928a.
[   24.082370] input: PC Speaker as /devices/platform/pcspkr/input/input6
[   24.119728] Adding 8388604k swap on /dev/zram0.  Priority:100 extents:1 across:8388604k SSDscFS
[   24.432118] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:02.0/0000:05:00.1/sound/card1/input7
[   24.432218] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:02.0/0000:05:00.1/sound/card1/input8
[   24.432304] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:02.0/0000:05:00.1/sound/card1/input9
[   24.432386] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:02.0/0000:05:00.1/sound/card1/input10
[   24.478501] input: HDA Intel PCH Front Mic as /devices/pci0000:00/0000:00:1b.0/sound/card0/input11
[   24.478604] input: HDA Intel PCH Rear Mic as /devices/pci0000:00/0000:00:1b.0/sound/card0/input12
[   24.478684] input: HDA Intel PCH Line as /devices/pci0000:00/0000:00:1b.0/sound/card0/input13
[   24.478756] input: HDA Intel PCH Line Out Front as /devices/pci0000:00/0000:00:1b.0/sound/card0/input14
[   24.478837] input: HDA Intel PCH Line Out Surround as /devices/pci0000:00/0000:00:1b.0/sound/card0/input15
[   24.478887] input: HDA Intel PCH Line Out CLFE as /devices/pci0000:00/0000:00:1b.0/sound/card0/input16
[   24.478940] input: HDA Intel PCH Line Out Side as /devices/pci0000:00/0000:00:1b.0/sound/card0/input17
[   24.478998] input: HDA Intel PCH Front Headphone as /devices/pci0000:00/0000:00:1b.0/sound/card0/input18
[   28.341728] BTRFS: device label archive devid 1 transid 44806 /dev/dm-11 scanned by (udev-worker) (1286)
[   28.882776] BTRFS: device label archive devid 3 transid 44806 /dev/dm-12 scanned by (udev-worker) (1286)
[   29.048837] BTRFS: device label archive devid 2 transid 44806 /dev/dm-14 scanned by (udev-worker) (1297)
[   29.364894] BTRFS info (device dm-11): bdev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf errs: wr 0, rd 0, flush 0, corrupt 106, gen 0
[   29.392200] BTRFS info: devid 3 device path /dev/mapper/luks-9b746590-b83b-4e35-bb7c-1c07d5130a4f changed to /dev/dm-12 scanned by (udev-worker) (1307)
[   29.395337] BTRFS info: devid 3 device path /dev/dm-12 changed to /dev/mapper/luks-9b746590-b83b-4e35-bb7c-1c07d5130a4f scanned by (udev-worker) (1307)
[   29.568860] BTRFS info: devid 3 device path /dev/mapper/luks-9b746590-b83b-4e35-bb7c-1c07d5130a4f changed to /dev/dm-12 scanned by (udev-worker) (1326)
[   29.571065] BTRFS info: devid 3 device path /dev/dm-12 changed to /dev/mapper/luks-9b746590-b83b-4e35-bb7c-1c07d5130a4f scanned by (udev-worker) (1326)

[robertk@earth:~]$ dmesg | grep sda
[    2.939343] sd 0:0:0:0: [sda] 2000409264 512-byte logical blocks: (1.02 TB/954 GiB)
[    2.940217] sd 0:0:0:0: [sda] Write Protect is off
[    2.940811] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
[    2.940845] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    2.942074] sd 0:0:0:0: [sda] Preferred minimum I/O size 512 bytes
[    2.947042]  sda: sda1 sda2 sda3
[    2.947789] sd 0:0:0:0: [sda] Attached SCSI disk
[   24.761271] EXT4-fs (sda2): mounted filesystem 667dcfd4-ff09-4f49-a7a6-08e6c7b49639 r/w with ordered data mode. Quota mode: none.

Nothing stand out as obvious.

Now I was chasing a kernel boot failure just after the BTRFS setup that forced me to power down computer suddenly numerous times. That can be a problem for SSD’s and I am wondering if this crc problem relates to that. Refer to that incidence: Refer Fedora hangs on boot after upgrading to kernel 6.3.4

Though probably not meaningfull to this problem other then forced power downs which is known to upset SSD without relevant circuitry which I am sure applies to the T-FORCE as it was the cheap end of SSDs.

Please note in my rebuild of this computer for F38, I used BTRFS for first time. My experience was not pleasant with this Filesystem prompting my critique. In that case timeshift did not play nicely with BTRFS and I am wondering if borg falls also into this problem. Refer BTRFS - I surrender! Good features BUT Complex, wastes time, not KISS, does not play well - #41 by rkoppelh

I’ll try next to explore what I can do with borg itself to fix the checksum and file 213 given suggestions so far is that disk maybe OK???

9

Hopefully more clues for someone more knowledgeable then I. But this is when journalctl of whats available first notes a csum error

# journalctl | grep BTRFS
...


Jul 25 22:47:33 earth kernel: BTRFS info (device dm-11): relocating block group 3766783836160 flags data
Jul 25 22:47:34 earth kernel: BTRFS warning (device dm-11): csum failed root -9 ino 1052 off 334180352 csum 0x481243e1eff5d46047d009ea643d072dbb1fe867039cdafc78ae9823e5548a67 expected csum 0xc000008729b877108c133240a12dda93e56a375de21f967a7a44b3ca42d720f6 mirror 1
Jul 25 22:47:34 earth kernel: BTRFS error (device dm-11): bdev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf errs: wr 0, rd 0, flush 0, corrupt 39, gen 0
Jul 25 22:47:34 earth kernel: BTRFS warning (device dm-11): csum failed root -9 ino 1052 off 334184448 csum 0xb051d098712e396dedf10c5949e587ce270dcb3bde09068905bfabb640a205e7 expected csum 0xa269b617bf88ac228f42a4cdf52f6e75666caff7d223045b00b80d05dd165e1c mirror 1
Jul 25 22:47:34 earth kernel: BTRFS error (device dm-11): bdev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf errs: wr 0, rd 0, flush 0, corrupt 40, gen 0
Jul 25 22:47:34 earth kernel: BTRFS warning (device dm-11): csum failed root -9 ino 1052 off 334188544 csum 0xe5591b387a1abe17dddef2f71485f4a34e137d241b0d5debc6efb2297849401d expected csum 0x54614bb249f5732376faedd7004d353d7d2d6324f7aa43e70342b2c51264d82e mirror 1
Jul 25 22:47:34 earth kernel: BTRFS error (device dm-11): bdev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf errs: wr 0, rd 0, flush 0, corrupt 41, gen 0
Jul 25 22:47:34 earth kernel: BTRFS warning (device dm-11): csum failed root -9 ino 1052 off 334192640 csum 0x83f89ac128dfd364073adf183430557a0c11daa9d1f9559195ac47c041083e76 expected csum 0xa2b4b5b6045a7a5b2c49c53092f1d076a263ab7962ed6f0d90adcd65aefdaabc mirror 1
Jul 25 22:47:34 earth kernel: BTRFS error (device dm-11): bdev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf errs: wr 0, rd 0, flush 0, corrupt 42, gen 0
Jul 25 22:47:34 earth kernel: BTRFS warning (device dm-11): csum failed root -9 ino 1052 off 334196736 csum 0x8fb1232bca57481ed1fd1ed930e993e2362551f4f41fa4d039d1e3025ce2b401 expected csum 0x29ad74250e2a55564bd0ddac0c1d258c0b22b9467368baa1384ad1b8991e8a76 mirror 1
Jul 25 22:47:34 earth kernel: BTRFS error (device dm-11): bdev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf errs: wr 0, rd 0, flush 0, corrupt 43, gen 0
Jul 25 22:47:34 earth kernel: BTRFS warning (device dm-11): csum failed root -9 ino 1052 off 334200832 csum 0x31f391c2805cd26c04489bafdaebe06fc5c13c779fa4f8ddc3413c3c0929d9a3 expected csum 0xb9e349f1714927669c725b5a50e931d57af5c5f10e71173d82952617de524928 mirror 1
Jul 25 22:47:34 earth kernel: BTRFS error (device dm-11): bdev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf errs: wr 0, rd 0, flush 0, corrupt 44, gen 0
Jul 25 22:47:34 earth kernel: BTRFS warning (device dm-11): csum failed root -9 ino 1052 off 334204928 csum 0xf2e51fa30ea1a36880cfecd7f28a2fdce8160a1853c0b4614c6322a0cb4193c7 expected csum 0xf012f6ec84ff0d746ce6e9f719f09f5e67428b741574d65a6d30a3cdebfbabd6 mirror 1
Jul 25 22:47:34 earth kernel: BTRFS error (device dm-11): bdev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf errs: wr 0, rd 0, flush 0, corrupt 45, gen 0
Jul 25 22:47:34 earth kernel: BTRFS warning (device dm-11): csum failed root -9 ino 1052 off 334209024 csum 0x90f00242f3fd95e8c5001926b94170619ca6e2cd678d440cecc489a7f54c5681 expected csum 0xd61a98fbf60e23abb6396acf485abe9b1fe387d2f8a208c87b98b8932a019fff mirror 1
Jul 25 22:47:34 earth kernel: BTRFS error (device dm-11): bdev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf errs: wr 0, rd 0, flush 0, corrupt 46, gen 0
Jul 25 22:47:35 earth kernel: BTRFS warning (device dm-11): csum failed root -9 ino 1052 off 334180352 csum 0x481243e1eff5d46047d009ea643d072dbb1fe867039cdafc78ae9823e5548a67 expected csum 0xc000008729b877108c133240a12dda93e56a375de21f967a7a44b3ca42d720f6 mirror 1
Jul 25 22:47:35 earth kernel: BTRFS error (device dm-11): bdev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf errs: wr 0, rd 0, flush 0, corrupt 47, gen 0
Jul 25 22:48:24 earth kernel: BTRFS info (device dm-11): balance: ended with status: -5

this kind of continues until more recently


Aug 01 00:00:01 earth kernel: BTRFS info (device dm-12): scrub: started on devid 1
Aug 01 00:00:01 earth kernel: BTRFS info (device dm-12): scrub: started on devid 2
Aug 01 00:00:01 earth kernel: BTRFS info (device dm-12): scrub: started on devid 3
Aug 01 00:10:12 earth kernel: BTRFS error (device dm-12): unable to fixup (regular) error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf physical 147475595264
Aug 01 00:10:12 earth kernel: BTRFS warning (device dm-12): checksum error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf, physical 147475595264, root 5, inode 28336187, offset 331862016, length 4096, links 2 (path: borg-assimilation/earth/data/0/213)
Aug 01 00:10:12 earth kernel: BTRFS warning (device dm-12): checksum error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf, physical 147475595264, root 5, inode 28336187, offset 331862016, length 4096, links 2 (path: borg-assimilation/earth.before-upgrade-2023-05-14-15:32:47/data/0/213)
Aug 01 00:10:12 earth kernel: BTRFS error (device dm-12): unable to fixup (regular) error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf physical 147475595264
Aug 01 00:10:12 earth kernel: BTRFS warning (device dm-12): checksum error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf, physical 147475595264, root 5, inode 28336187, offset 331862016, length 4096, links 2 (path: borg-assimilation/earth/data/0/213)
Aug 01 00:10:12 earth kernel: BTRFS warning (device dm-12): checksum error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf, physical 147475595264, root 5, inode 28336187, offset 331862016, length 4096, links 2 (path: borg-assimilation/earth.before-upgrade-2023-05-14-15:32:47/data/0/213)
Aug 01 00:10:12 earth kernel: BTRFS error (device dm-12): unable to fixup (regular) error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf physical 147475595264
Aug 01 00:10:12 earth kernel: BTRFS warning (device dm-12): checksum error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf, physical 147475595264, root 5, inode 28336187, offset 331862016, length 4096, links 2 (path: borg-assimilation/earth/data/0/213)
Aug 01 00:10:12 earth kernel: BTRFS warning (device dm-12): checksum error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf, physical 147475595264, root 5, inode 28336187, offset 331862016, length 4096, links 2 (path: borg-assimilation/earth.before-upgrade-2023-05-14-15:32:47/data/0/213)
Aug 01 00:10:12 earth kernel: BTRFS error (device dm-12): unable to fixup (regular) error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf physical 147475595264
Aug 01 00:10:12 earth kernel: BTRFS warning (device dm-12): checksum error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf, physical 147475595264, root 5, inode 28336187, offset 331862016, length 4096, links 2 (path: borg-assimilation/earth/data/0/213)
Aug 01 00:10:12 earth kernel: BTRFS warning (device dm-12): checksum error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf, physical 147475595264, root 5, inode 28336187, offset 331862016, length 4096, links 2 (path: borg-assimilation/earth.before-upgrade-2023-05-14-15:32:47/data/0/213)
Aug 01 00:10:12 earth kernel: BTRFS error (device dm-12): unable to fixup (regular) error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf physical 147475595264
Aug 01 00:10:12 earth kernel: BTRFS warning (device dm-12): checksum error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf, physical 147475595264, root 5, inode 28336187, offset 331862016, length 4096, links 2 (path: borg-assimilation/earth/data/0/213)
Aug 01 00:10:12 earth kernel: BTRFS warning (device dm-12): checksum error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf, physical 147475595264, root 5, inode 28336187, offset 331862016, length 4096, links 2 (path: borg-assimilation/earth.before-upgrade-2023-05-14-15:32:47/data/0/213)
Aug 01 00:10:12 earth kernel: BTRFS error (device dm-12): unable to fixup (regular) error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf physical 147475595264
Aug 01 00:10:12 earth kernel: BTRFS warning (device dm-12): checksum error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf, physical 147475595264, root 5, inode 28336187, offset 331862016, length 4096, links 2 (path: borg-assimilation/earth/data/0/213)
Aug 01 00:10:12 earth kernel: BTRFS warning (device dm-12): checksum error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf, physical 147475595264, root 5, inode 28336187, offset 331862016, length 4096, links 2 (path: borg-assimilation/earth.before-upgrade-2023-05-14-15:32:47/data/0/213)
Aug 01 00:10:12 earth kernel: BTRFS error (device dm-12): unable to fixup (regular) error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf physical 147475595264
Aug 01 00:10:12 earth kernel: BTRFS warning (device dm-12): checksum error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf, physical 147475595264, root 5, inode 28336187, offset 331862016, length 4096, links 2 (path: borg-assimilation/earth/data/0/213)
Aug 01 00:10:12 earth kernel: BTRFS warning (device dm-12): checksum error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf, physical 147475595264, root 5, inode 28336187, offset 331862016, length 4096, links 2 (path: borg-assimilation/earth.before-upgrade-2023-05-14-15:32:47/data/0/213)
Aug 01 00:10:12 earth kernel: BTRFS error (device dm-12): unable to fixup (regular) error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf physical 147475595264
Aug 01 00:10:12 earth kernel: BTRFS warning (device dm-12): checksum error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf, physical 147475595264, root 5, inode 28336187, offset 331862016, length 4096, links 2 (path: borg-assimilation/earth/data/0/213)
Aug 01 00:10:12 earth kernel: BTRFS warning (device dm-12): checksum error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf, physical 147475595264, root 5, inode 28336187, offset 331862016, length 4096, links 2 (path: borg-assimilation/earth.before-upgrade-2023-05-14-15:32:47/data/0/213)
Aug 01 00:25:05 earth kernel: BTRFS info (device dm-12): scrub: finished on devid 2 with status: 0
Aug 01 00:25:32 earth kernel: BTRFS info (device dm-12): scrub: finished on devid 3 with status: 0
Aug 01 00:54:58 earth kernel: BTRFS info (device dm-12): scrub: finished on devid 1 with status: 0
Aug 01 00:54:58 earth kernel: BTRFS info (device dm-12): balance: start -dusage=0
Aug 01 00:54:58 earth kernel: BTRFS info (device dm-12): balance: ended with status: 0
Aug 01 00:54:58 earth kernel: BTRFS info (device dm-12): balance: start -dusage=5
Aug 01 00:54:58 earth kernel: BTRFS info (device dm-12): balance: ended with status: 0
Aug 01 00:54:58 earth kernel: BTRFS info (device dm-12): balance: start -dusage=10
Aug 01 00:54:58 earth kernel: BTRFS info (device dm-12): balance: ended with status: 0
Aug 01 00:54:58 earth kernel: BTRFS info (device dm-12): balance: start -musage=0 -susage=0
Aug 01 00:54:58 earth kernel: BTRFS info (device dm-12): balance: ended with status: 0
Aug 01 00:54:58 earth kernel: BTRFS info (device dm-12): balance: start -musage=5 -susage=5
Aug 01 00:54:58 earth kernel: BTRFS info (device dm-12): relocating block group 5494937747456 flags system|dup
Aug 01 00:54:58 earth kernel: BTRFS info (device dm-12): found 3 extents, stage: move data extents
Aug 01 00:54:58 earth kernel: BTRFS info (device dm-12): relocating block group 5489535483904 flags system|dup
Aug 01 00:54:59 earth kernel: BTRFS info (device dm-12): found 8 extents, stage: move data extents
Aug 01 00:54:59 earth kernel: BTRFS info (device dm-12): balance: ended with status: 0
Aug 01 11:20:40 earth kernel: BTRFS warning (device dm-12): csum failed root 5 ino 28336187 off 331874304 csum 0x481243e1eff5d46047d009ea643d072dbb1fe867039cdafc78ae9823e5548a67 expected csum 0xc000008729b877108c133240a12dda93e56a375de21f967a7a44b3ca42d720f6 mirror 1
Aug 01 11:20:40 earth kernel: BTRFS error (device dm-12): bdev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf errs: wr 0, rd 0, flush 0, corrupt 48, gen 0
Aug 01 11:20:40 earth kernel: BTRFS warning (device dm-12): csum failed root 5 ino 28336187 off 331878400 csum 0xb051d098712e396dedf10c5949e587ce270dcb3bde09068905bfabb640a205e7 expected csum 0xa269b617bf88ac228f42a4cdf52f6e75666caff7d223045b00b80d05dd165e1c mirror 1
Aug 01 11:20:40 earth kernel: BTRFS error (device dm-12): bdev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf errs: wr 0, rd 0, flush 0, corrupt 49, gen 0
Aug 01 11:20:40 earth kernel: BTRFS warning (device dm-12): csum failed root 5 ino 28336187 off 331882496 csum 0xe5591b387a1abe17dddef2f71485f4a34e137d241b0d5debc6efb2297849401d expected csum 0x54614bb249f5732376faedd7004d353d7d2d6324f7aa43e70342b2c51264d82e mirror 1
Aug 01 11:20:40 earth kernel: BTRFS error (device dm-12): bdev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf errs: wr 0, rd 0, flush 0, corrupt 50, gen 0
Aug 01 11:20:40 earth kernel: BTRFS warning (device dm-12): csum failed root 5 ino 28336187 off 331886592 csum 0x83f89ac128dfd364073adf183430557a0c11daa9d1f9559195ac47c041083e76 expected csum 0xa2b4b5b6045a7a5b2c49c53092f1d076a263ab7962ed6f0d90adcd65aefdaabc mirror 1
Aug 01 11:20:40 earth kernel: BTRFS error (device dm-12): bdev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf errs: wr 0, rd 0, flush 0, corrupt 51, gen 0
Aug 01 11:20:40 earth kernel: BTRFS warning (device dm-12): csum failed root 5 ino 28336187 off 331890688 csum 0x8fb1232bca57481ed1fd1ed930e993e2362551f4f41fa4d039d1e3025ce2b401 expected csum 0x29ad74250e2a55564bd0ddac0c1d258c0b22b9467368baa1384ad1b8991e8a76 mirror 1
Aug 01 11:20:40 earth kernel: BTRFS error (device dm-12): bdev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf errs: wr 0, rd 0, flush 0, corrupt 52, gen 0
Aug 01 11:20:40 earth kernel: BTRFS warning (device dm-12): csum failed root 5 ino 28336187 off 331894784 csum 0x31f391c2805cd26c04489bafdaebe06fc5c13c779fa4f8ddc3413c3c0929d9a3 expected csum 0xb9e349f1714927669c725b5a50e931d57af5c5f10e71173d82952617de524928 mirror 1
Aug 01 11:20:40 earth kernel: BTRFS error (device dm-12): bdev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf errs: wr 0, rd 0, flush 0, corrupt 53, gen 0
Aug 01 11:20:40 earth kernel: BTRFS warning (device dm-12): csum failed root 5 ino 28336187 off 331898880 csum 0xf2e51fa30ea1a36880cfecd7f28a2fdce8160a1853c0b4614c6322a0cb4193c7 expected csum 0xf012f6ec84ff0d746ce6e9f719f09f5e67428b741574d65a6d30a3cdebfbabd6 mirror 1
Aug 01 11:20:40 earth kernel: BTRFS error (device dm-12): bdev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf errs: wr 0, rd 0, flush 0, corrupt 54, gen 0
Aug 01 11:20:40 earth kernel: BTRFS warning (device dm-12): csum failed root 5 ino 28336187 off 331902976 csum 0x90f00242f3fd95e8c5001926b94170619ca6e2cd678d440cecc489a7f54c5681 expected csum 0xd61a98fbf60e23abb6396acf485abe9b1fe387d2f8a208c87b98b8932a019fff mirror 1
Aug 01 11:20:40 earth kernel: BTRFS error (device dm-12): bdev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf errs: wr 0, rd 0, flush 0, corrupt 55, gen 0
Aug 01 11:20:40 earth kernel: BTRFS warning (device dm-12): csum failed root 5 ino 28336187 off 331874304 csum 0x481243e1eff5d46047d009ea643d072dbb1fe867039cdafc78ae9823e5548a67 expected csum 0xc000008729b877108c133240a12dda93e56a375de21f967a7a44b3ca42d720f6 mirror 1
Aug 01 11:20:40 earth kernel: BTRFS error (device dm-12): bdev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf errs: wr 0, rd 0, flush 0, corrupt 56, gen 0
Aug 01 18:13:43 earth kernel: BTRFS warning (device dm-12): csum failed root 5 ino 28336187 off 331874304 csum 0x481243e1eff5d46047d009ea643d072dbb1fe867039cdafc78ae9823e5548a67 expected csum 0xc000008729b877108c133240a12dda93e56a375de21f967a7a44b3ca42d720f6 mirror 1
Aug 01 18:13:43 earth kernel: BTRFS error (device dm-12): bdev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf errs: wr 0, rd 0, flush 0, corrupt 57, gen 0
Aug 01 18:13:43 earth kernel: BTRFS warning (device dm-12): csum failed root 5 ino 28336187 off 331878400 csum 0xb051d098712e396dedf10c5949e587ce270dcb3bde09068905bfabb640a205e7 expected csum 0xa269b617bf88ac228f42a4cdf52f6e75666caff7d223045b00b80d05dd165e1c mirror 1
Aug 01 18:13:43 earth kernel: BTRFS error (device dm-12): bdev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf errs: wr 0, rd 0, flush 0, corrupt 58, gen 0
Aug 01 18:13:43 earth kernel: BTRFS warning (device dm-12): csum failed root 5 ino 28336187 off 331882496 csum 0xe5591b387a1abe17dddef2f71485f4a34e137d241b0d5debc6efb2297849401d expected csum 0x54614bb249f5732376faedd7004d353d7d2d6324f7aa43e70342b2c51264d82e mirror 1
Aug 01 18:13:43 earth kernel: BTRFS error (device dm-12): bdev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf errs: wr 0, rd 0, flush 0, corrupt 59, gen 0
Aug 01 18:13:43 earth kernel: BTRFS warning (device dm-12): csum failed root 5 ino 28336187 off 331886592 csum 0x83f89ac128dfd364073adf183430557a0c11daa9d1f9559195ac47c041083e76 expected csum 0xa2b4b5b6045a7a5b2c49c53092f1d076a263ab7962ed6f0d90adcd65aefdaabc mirror 1
Aug 01 18:13:43 earth kernel: BTRFS error (device dm-12): bdev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf errs: wr 0, rd 0, flush 0, corrupt 60, gen 0
Aug 01 18:13:43 earth kernel: BTRFS warning (device dm-12): csum failed root 5 ino 28336187 off 331890688 csum 0x8fb1232bca57481ed1fd1ed930e993e2362551f4f41fa4d039d1e3025ce2b401 expected csum 0x29ad74250e2a55564bd0ddac0c1d258c0b22b9467368baa1384ad1b8991e8a76 mirror 1
Aug 01 18:13:43 earth kernel: BTRFS error (device dm-12): bdev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf errs: wr 0, rd 0, flush 0, corrupt 61, gen 0
Aug 01 18:13:43 earth kernel: BTRFS warning (device dm-12): csum failed root 5 ino 28336187 off 331894784 csum 0x31f391c2805cd26c04489bafdaebe06fc5c13c779fa4f8ddc3413c3c0929d9a3 expected csum 0xb9e349f1714927669c725b5a50e931d57af5c5f10e71173d82952617de524928 mirror 1
Aug 01 18:13:43 earth kernel: BTRFS error (device dm-12): bdev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf errs: wr 0, rd 0, flush 0, corrupt 62, gen 0
Aug 01 18:13:43 earth kernel: BTRFS warning (device dm-12): csum failed root 5 ino 28336187 off 331898880 csum 0xf2e51fa30ea1a36880cfecd7f28a2fdce8160a1853c0b4614c6322a0cb4193c7 expected csum 0xf012f6ec84ff0d746ce6e9f719f09f5e67428b741574d65a6d30a3cdebfbabd6 mirror 1
Aug 01 18:13:43 earth kernel: BTRFS error (device dm-12): bdev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf errs: wr 0, rd 0, flush 0, corrupt 63, gen 0
Aug 01 18:13:43 earth kernel: BTRFS warning (device dm-12): csum failed root 5 ino 28336187 off 331902976 csum 0x90f00242f3fd95e8c5001926b94170619ca6e2cd678d440cecc489a7f54c5681 expected csum 0xd61a98fbf60e23abb6396acf485abe9b1fe387d2f8a208c87b98b8932a019fff mirror 1
Aug 01 18:13:43 earth kernel: BTRFS error (device dm-12): bdev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf errs: wr 0, rd 0, flush 0, corrupt 64, gen 0
Aug 01 18:13:43 earth kernel: BTRFS warning (device dm-12): csum failed root 5 ino 28336187 off 331874304 csum 0x481243e1eff5d46047d009ea643d072dbb1fe867039cdafc78ae9823e5548a67 expected csum 0xc000008729b877108c133240a12dda93e56a375de21f967a7a44b3ca42d720f6 mirror 1
Aug 01 18:13:43 earth kernel: BTRFS error (device dm-12): bdev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf errs: wr 0, rd 0, flush 0, corrupt 65, gen 0

until recently

Aug 03 09:34:15 earth kernel: BTRFS warning (device dm-12): csum failed root 5 ino 28336187 off 331874304 csum 0x481243e1eff5d46047d009ea643d072dbb1fe867039cdafc78ae9823e5548a67 expected csum 0xc000008729b877108c133240a12dda93e56a375de21f967a7a44b3ca42d720f6 mirror 1
Aug 03 09:34:15 earth kernel: BTRFS error (device dm-12): bdev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf errs: wr 0, rd 0, flush 0, corrupt 102, gen 0
Aug 03 09:34:57 earth kernel: BTRFS warning (device dm-12): csum failed root 5 ino 28336187 off 331874304 csum 0x481243e1eff5d46047d009ea643d072dbb1fe867039cdafc78ae9823e5548a67 expected csum 0xc000008729b877108c133240a12dda93e56a375de21f967a7a44b3ca42d720f6 mirror 1
Aug 03 09:34:57 earth kernel: BTRFS error (device dm-12): bdev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf errs: wr 0, rd 0, flush 0, corrupt 103, gen 0
Aug 03 09:36:57 earth kernel: BTRFS warning (device dm-12): csum failed root 5 ino 28336187 off 331874304 csum 0x481243e1eff5d46047d009ea643d072dbb1fe867039cdafc78ae9823e5548a67 expected csum 0xc000008729b877108c133240a12dda93e56a375de21f967a7a44b3ca42d720f6 mirror 1
Aug 03 09:36:57 earth kernel: BTRFS error (device dm-12): bdev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf errs: wr 0, rd 0, flush 0, corrupt 104, gen 0
Aug 03 09:37:31 earth kernel: BTRFS warning (device dm-12): csum failed root 5 ino 28336187 off 331874304 csum 0x481243e1eff5d46047d009ea643d072dbb1fe867039cdafc78ae9823e5548a67 expected csum 0xc000008729b877108c133240a12dda93e56a375de21f967a7a44b3ca42d720f6 mirror 1
Aug 03 09:37:31 earth kernel: BTRFS error (device dm-12): bdev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf errs: wr 0, rd 0, flush 0, corrupt 105, gen 0
Aug 03 09:39:58 earth kernel: BTRFS warning (device dm-12): csum failed root 5 ino 28336187 off 331874304 csum 0x481243e1eff5d46047d009ea643d072dbb1fe867039cdafc78ae9823e5548a67 expected csum 0xc000008729b877108c133240a12dda93e56a375de21f967a7a44b3ca42d720f6 mirror 1
Aug 03 09:39:58 earth kernel: BTRFS error (device dm-12): bdev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf errs: wr 0, rd 0, flush 0, corrupt 106, gen 0
Aug 03 10:55:40 earth kernel: BTRFS info (device dm-12): scrub: started on devid 1
Aug 03 10:55:40 earth kernel: BTRFS info (device dm-12): scrub: started on devid 2
Aug 03 10:55:40 earth kernel: BTRFS info (device dm-12): scrub: started on devid 3
Aug 03 11:03:35 earth kernel: BTRFS error (device dm-12): unable to fixup (regular) error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf physical 147475595264
Aug 03 11:03:35 earth kernel: BTRFS warning (device dm-12): checksum error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf, physical 147475595264, root 5, inode 28336187, offset 331862016, length 4096, links 2 (path: borg-assimilation/earth/data/0/213)
Aug 03 11:03:35 earth kernel: BTRFS warning (device dm-12): checksum error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf, physical 147475595264, root 5, inode 28336187, offset 331862016, length 4096, links 2 (path: borg-assimilation/earth.before-upgrade-2023-05-14-15:32:47/data/0/213)
Aug 03 11:03:35 earth kernel: BTRFS error (device dm-12): unable to fixup (regular) error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf physical 147475595264
Aug 03 11:03:35 earth kernel: BTRFS warning (device dm-12): checksum error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf, physical 147475595264, root 5, inode 28336187, offset 331862016, length 4096, links 2 (path: borg-assimilation/earth/data/0/213)
Aug 03 11:03:35 earth kernel: BTRFS warning (device dm-12): checksum error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf, physical 147475595264, root 5, inode 28336187, offset 331862016, length 4096, links 2 (path: borg-assimilation/earth.before-upgrade-2023-05-14-15:32:47/data/0/213)
Aug 03 11:03:35 earth kernel: BTRFS error (device dm-12): unable to fixup (regular) error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf physical 147475595264
Aug 03 11:03:35 earth kernel: BTRFS warning (device dm-12): checksum error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf, physical 147475595264, root 5, inode 28336187, offset 331862016, length 4096, links 2 (path: borg-assimilation/earth/data/0/213)
Aug 03 11:03:35 earth kernel: BTRFS warning (device dm-12): checksum error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf, physical 147475595264, root 5, inode 28336187, offset 331862016, length 4096, links 2 (path: borg-assimilation/earth.before-upgrade-2023-05-14-15:32:47/data/0/213)
Aug 03 11:03:35 earth kernel: BTRFS error (device dm-12): unable to fixup (regular) error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf physical 147475595264
Aug 03 11:03:35 earth kernel: BTRFS warning (device dm-12): checksum error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf, physical 147475595264, root 5, inode 28336187, offset 331862016, length 4096, links 2 (path: borg-assimilation/earth/data/0/213)
Aug 03 11:03:35 earth kernel: BTRFS warning (device dm-12): checksum error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf, physical 147475595264, root 5, inode 28336187, offset 331862016, length 4096, links 2 (path: borg-assimilation/earth.before-upgrade-2023-05-14-15:32:47/data/0/213)
Aug 03 11:03:35 earth kernel: BTRFS error (device dm-12): unable to fixup (regular) error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf physical 147475595264
Aug 03 11:03:35 earth kernel: BTRFS warning (device dm-12): checksum error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf, physical 147475595264, root 5, inode 28336187, offset 331862016, length 4096, links 2 (path: borg-assimilation/earth/data/0/213)
Aug 03 11:03:35 earth kernel: BTRFS warning (device dm-12): checksum error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf, physical 147475595264, root 5, inode 28336187, offset 331862016, length 4096, links 2 (path: borg-assimilation/earth.before-upgrade-2023-05-14-15:32:47/data/0/213)
Aug 03 11:03:35 earth kernel: BTRFS error (device dm-12): unable to fixup (regular) error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf physical 147475595264
Aug 03 11:03:35 earth kernel: BTRFS warning (device dm-12): checksum error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf, physical 147475595264, root 5, inode 28336187, offset 331862016, length 4096, links 2 (path: borg-assimilation/earth/data/0/213)
Aug 03 11:03:35 earth kernel: BTRFS warning (device dm-12): checksum error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf, physical 147475595264, root 5, inode 28336187, offset 331862016, length 4096, links 2 (path: borg-assimilation/earth.before-upgrade-2023-05-14-15:32:47/data/0/213)
Aug 03 11:03:35 earth kernel: BTRFS error (device dm-12): unable to fixup (regular) error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf physical 147475595264
Aug 03 11:03:35 earth kernel: BTRFS warning (device dm-12): checksum error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf, physical 147475595264, root 5, inode 28336187, offset 331862016, length 4096, links 2 (path: borg-assimilation/earth/data/0/213)
Aug 03 11:03:35 earth kernel: BTRFS warning (device dm-12): checksum error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf, physical 147475595264, root 5, inode 28336187, offset 331862016, length 4096, links 2 (path: borg-assimilation/earth.before-upgrade-2023-05-14-15:32:47/data/0/213)
Aug 03 11:03:35 earth kernel: BTRFS error (device dm-12): unable to fixup (regular) error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf physical 147475595264
Aug 03 11:03:35 earth kernel: BTRFS warning (device dm-12): checksum error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf, physical 147475595264, root 5, inode 28336187, offset 331862016, length 4096, links 2 (path: borg-assimilation/earth/data/0/213)
Aug 03 11:03:35 earth kernel: BTRFS warning (device dm-12): checksum error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf, physical 147475595264, root 5, inode 28336187, offset 331862016, length 4096, links 2 (path: borg-assimilation/earth.before-upgrade-2023-05-14-15:32:47/data/0/213)
Aug 03 11:12:15 earth kernel: BTRFS info (device dm-12): scrub: finished on devid 2 with status: 0
Aug 03 11:13:08 earth kernel: BTRFS info (device dm-12): scrub: finished on devid 3 with status: 0
Aug 03 11:39:55 earth kernel: BTRFS info (device dm-12): scrub: finished on devid 1 with status: 0
Aug 04 09:09:44 earth kernel: BTRFS: device label archive devid 1 transid 44806 /dev/dm-11 scanned by (udev-worker) (1286)
Aug 04 09:09:44 earth kernel: BTRFS: device label archive devid 3 transid 44806 /dev/dm-12 scanned by (udev-worker) (1286)
Aug 04 09:09:44 earth kernel: BTRFS: device label archive devid 2 transid 44806 /dev/dm-14 scanned by (udev-worker) (1297)
Aug 04 09:09:45 earth kernel: BTRFS info (device dm-11): using sha256 (sha256-generic) checksum algorithm
Aug 04 09:09:45 earth kernel: BTRFS info (device dm-11): use zstd compression, level 9
Aug 04 09:09:45 earth kernel: BTRFS info (device dm-11): turning on async discard
Aug 04 09:09:45 earth kernel: BTRFS info (device dm-11): using free space tree
Aug 04 09:09:45 earth kernel: BTRFS info (device dm-11): bdev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf errs: wr 0, rd 0, flush 0, corrupt 106, gen 0
Aug 04 09:09:45 earth kernel: BTRFS info: devid 3 device path /dev/mapper/luks-9b746590-b83b-4e35-bb7c-1c07d5130a4f changed to /dev/dm-12 scanned by (udev-worker) (1307)
Aug 04 09:09:45 earth kernel: BTRFS info: devid 3 device path /dev/dm-12 changed to /dev/mapper/luks-9b746590-b83b-4e35-bb7c-1c07d5130a4f scanned by (udev-worker) (1307)
Aug 04 09:09:45 earth kernel: BTRFS info (device dm-11): enabling ssd optimizations
Aug 04 09:09:45 earth kernel: BTRFS info: devid 3 device path /dev/mapper/luks-9b746590-b83b-4e35-bb7c-1c07d5130a4f changed to /dev/dm-12 scanned by (udev-worker) (1326)
Aug 04 09:09:45 earth kernel: BTRFS info: devid 3 device path /dev/dm-12 changed to /dev/mapper/luks-9b746590-b83b-4e35-bb7c-1c07d5130a4f scanned by (udev-worker) (1326)

I did notice in this post, while also given my bad experience to date with BTRFS mentioned earlier above, that BTRFS may also not play nicely with cheaper SSDs. /dev/sda was a cheaper one of the bunch. It’s only a archive drive for me. Except for first time not exactly heavy use is what I thought.

Refer: Reddit - Dive into anything

Near end of article re csum errors appears similar to what I am seeing here re 8 bad csums on a relatively new SSD. Given the above experience I am having in whats described in the article maybe BTRFS is not a great idea for SSDs???

Certainly I am now even more sceptical of BTRFS. Any comments on that contraversial statement?

10

Ask Facebook, they use literally tonnes of cheap ssd’s all setup with BTFRS.

11

There may be frequent writes to an ssd. I beleve that by default fedora updates the atime whenever a file is accessed.
I have seen it suggested that for SSDs or NVME devices (and even USB flash devices that are used often) that one should add an option of noatime into the fstab entry for those type devices to reduce the number of writes required.

Maybe an expert on file systems could verify if this is or is not correct, and weigh in with suggestions.

12

Note I can’t complete scan on my 2013 Samsung 830 SSD i.e. the /dev/sb[1,2] of the merged BTRFS arrangement. It gets stuck at 90% and does not complete reporting result after 12 hours, i.e

image
image883×988 114 KB

$ sudo smartctl -a /dev/sdb
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.4.7-200.fc38.x86_64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Samsung based SSDs
Device Model:     SAMSUNG SSD 830 Series
Serial Number:    S0Z4NEAC877384
LU WWN Device Id: 5 002538 043584d30
Firmware Version: CXM03B1Q
User Capacity:    256,060,514,304 bytes [256 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
TRIM Command:     Available
Device is:        In smartctl database 7.3/5440
ATA Version is:   ACS-2 T13/2015-D revision 2
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sat Aug  5 11:57:05 2023 AEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x83) Offline data collection activity
                                        is in a Reserved state.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (  17) The self-test routine was aborted by
                                        the host.
Total time to complete Offline 
data collection:                ( 1020) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        (  17) minutes.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   089   089   000    Old_age   Always       -       51237
 12 Power_Cycle_Count       0x0032   096   096   000    Old_age   Always       -       3939
177 Wear_Leveling_Count     0x0013   080   080   000    Pre-fail  Always       -       720
179 Used_Rsvd_Blk_Cnt_Tot   0x0013   100   100   010    Pre-fail  Always       -       0
181 Program_Fail_Cnt_Total  0x0032   100   100   010    Old_age   Always       -       0
182 Erase_Fail_Count_Total  0x0032   100   100   010    Old_age   Always       -       0
183 Runtime_Bad_Block       0x0013   100   100   010    Pre-fail  Always       -       0
187 Uncorrectable_Error_Cnt 0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0032   074   060   000    Old_age   Always       -       26
195 ECC_Error_Rate          0x001a   200   200   000    Old_age   Always       -       0
199 CRC_Error_Count         0x003e   253   253   000    Old_age   Always       -       0
235 POR_Recovery_Count      0x0012   099   099   000    Old_age   Always       -       589
241 Total_LBAs_Written      0x0032   099   099   000    Old_age   Always       -       38499297201

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Aborted by host               10%     51237         -
# 2  Short offline       Completed without error       00%     50855         -
# 3  Extended offline    Completed without error       00%     49358         -
# 4  Short offline       Completed without error       00%     49347         -
# 5  Extended offline    Interrupted (host reset)      10%     47092         -
# 6  Extended offline    Completed without error       00%     44935         -
# 7  Extended offline    Completed without error       00%     43416         -
# 8  Short offline       Completed without error       00%     43403         -
# 9  Extended offline    Completed without error       00%     31888         -
#10  Short offline       Completed without error       00%     16623         -
#11  Short offline       Completed without error       00%      8641         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

AND

$ sudo smartctl -x /dev/sdb
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.4.7-200.fc38.x86_64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Samsung based SSDs
Device Model:     SAMSUNG SSD 830 Series
Serial Number:    S0Z4NEAC877384
LU WWN Device Id: 5 002538 043584d30
Firmware Version: CXM03B1Q
User Capacity:    256,060,514,304 bytes [256 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
TRIM Command:     Available
Device is:        In smartctl database 7.3/5440
ATA Version is:   ACS-2 T13/2015-D revision 2
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sat Aug  5 11:58:23 2023 AEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
DSN feature is:   Unavailable
ATA Security is:  Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Unavailable

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x83) Offline data collection activity
                                        is in a Reserved state.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (  17) The self-test routine was aborted by
                                        the host.
Total time to complete Offline 
data collection:                ( 1020) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        (  17) minutes.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  5 Reallocated_Sector_Ct   PO--CK   100   100   010    -    0
  9 Power_On_Hours          -O--CK   089   089   000    -    51237
 12 Power_Cycle_Count       -O--CK   096   096   000    -    3939
177 Wear_Leveling_Count     PO--C-   080   080   000    -    720
179 Used_Rsvd_Blk_Cnt_Tot   PO--C-   100   100   010    -    0
181 Program_Fail_Cnt_Total  -O--CK   100   100   010    -    0
182 Erase_Fail_Count_Total  -O--CK   100   100   010    -    0
183 Runtime_Bad_Block       PO--C-   100   100   010    -    0
187 Uncorrectable_Error_Cnt -O--CK   100   100   000    -    0
190 Airflow_Temperature_Cel -O--CK   074   060   000    -    26
195 ECC_Error_Rate          -O-RC-   200   200   000    -    0
199 CRC_Error_Count         -OSRCK   253   253   000    -    0
235 POR_Recovery_Count      -O--C-   099   099   000    -    589
241 Total_LBAs_Written      -O--CK   099   099   000    -    38499297201
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01       GPL,SL  R/O      1  Summary SMART error log
0x02       GPL,SL  R/O      2  Comprehensive SMART error log
0x03       GPL,SL  R/O      2  Ext. Comprehensive SMART error log
0x06       GPL,SL  R/O      1  SMART self-test log
0x07       GPL,SL  R/O      2  Extended self-test log
0x09       GPL,SL  R/W      1  Selective self-test log
0x10       GPL,SL  R/O      1  NCQ Command Error log
0x11       GPL,SL  R/O      1  SATA Phy Event Counters log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log

SMART Extended Comprehensive Error Log Version: 1 (2 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (2 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Aborted by host               10%     51237         -
# 2  Short offline       Completed without error       00%     50855         -
# 3  Extended offline    Completed without error       00%     49358         -
# 4  Short offline       Completed without error       00%     49347         -
# 5  Extended offline    Interrupted (host reset)      10%     47092         -
# 6  Extended offline    Completed without error       00%     44935         -
# 7  Extended offline    Completed without error       00%     43416         -
# 8  Short offline       Completed without error       00%     43403         -
# 9  Extended offline    Completed without error       00%     31888         -
#10  Short offline       Completed without error       00%     16623         -
#11  Short offline       Completed without error       00%      8641         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Commands not supported

Device Statistics (GP/SMART Log 0x04) not supported

Pending Defects log (GP Log 0x0c) not supported

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x0001  2            0  Command failed due to ICRC error
0x0002  2        49152  R_ERR response for data FIS
0x0003  2        53248  R_ERR response for device-to-host data FIS
0x0004  2        57344  R_ERR response for host-to-device data FIS
0x0005  2         7952  R_ERR response for non-data FIS
0x0006  2         2562  R_ERR response for device-to-host non-data FIS
0x0007  2         4096  R_ERR response for host-to-device non-data FIS
0x0008  2         7952  Device-to-host non-data FIS retries
0x0009  2         7953  Transition from drive PhyRdy to drive PhyNRdy
0x000a  2        15272  Device-to-host register FISes sent due to a COMRESET
0x000b  2         6145  CRC errors within host-to-device FIS
0x000d  2         7953  Non-CRC errors within host-to-device FIS
0x000f  2         8176  R_ERR response for host-to-device data FIS, CRC
0x0010  2           15  R_ERR response for host-to-device data FIS, non-CRC
0x0012  2        49153  R_ERR response for host-to-device non-data FIS, CRC
0x0013  2        12112  R_ERR response for host-to-device non-data FIS, non-CRC

This appears to show a well used sdb since 2013. Maybe it’s the problem?

For the newer sda [ above I posted earlier only smartctl -a results not -x as well]

$ sudo smartctl -x /dev/sda
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.4.7-200.fc38.x86_64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     T-FORCE 1TB
Serial Number:    TPBF2212050020101379
Firmware Version: V0718B0
User Capacity:    1,024,209,543,168 bytes [1.02 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
TRIM Command:     Available
Device is:        Not in smartctl database 7.3/5440
ATA Version is:   ACS-2 T13/2015-D revision 3
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sat Aug  5 12:01:52 2023 AEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM level is:     254 (maximum performance)
Rd look-ahead is: Enabled
Write cache is:   Enabled
DSN feature is:   Unavailable
ATA Security is:  Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Unavailable

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                (  120) seconds.
Offline data collection
capabilities:                    (0x11) SMART execute Offline immediate.
                                        No Auto Offline data collection support.
                                        Suspend Offline collection upon new
                                        command.
                                        No Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        No Selective Self-test supported.
SMART capabilities:            (0x0002) Does not save SMART data before
                                        entering power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        (  10) minutes.
SCT capabilities:              (0x0001) SCT Status supported.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     -O--CK   100   100   050    -    0
  5 Reallocated_Sector_Ct   -O--CK   100   100   050    -    0
  9 Power_On_Hours          -O--CK   100   100   050    -    2967
 12 Power_Cycle_Count       -O--CK   100   100   050    -    158
160 Unknown_Attribute       -O--CK   100   100   050    -    0
161 Unknown_Attribute       PO--CK   100   100   050    -    100
163 Unknown_Attribute       -O--CK   100   100   050    -    16
164 Unknown_Attribute       -O--CK   100   100   050    -    17371
165 Unknown_Attribute       -O--CK   100   100   050    -    44
166 Unknown_Attribute       -O--CK   100   100   050    -    10
167 Unknown_Attribute       -O--CK   100   100   050    -    26
168 Unknown_Attribute       -O--CK   100   100   050    -    5050
169 Unknown_Attribute       -O--CK   100   100   050    -    100
175 Program_Fail_Count_Chip -O--CK   100   100   050    -    0
176 Erase_Fail_Count_Chip   -O--CK   100   100   050    -    0
177 Wear_Leveling_Count     -O--CK   100   100   050    -    0
178 Used_Rsvd_Blk_Cnt_Chip  -O--CK   100   100   050    -    0
181 Program_Fail_Cnt_Total  -O--CK   100   100   050    -    0
182 Erase_Fail_Count_Total  -O--CK   100   100   050    -    0
192 Power-Off_Retract_Count -O--CK   100   100   050    -    59
194 Temperature_Celsius     -O---K   100   100   050    -    13
195 Hardware_ECC_Recovered  -O--CK   100   100   050    -    0
196 Reallocated_Event_Count -O--CK   100   100   050    -    0
197 Current_Pending_Sector  -O--CK   100   100   050    -    0
198 Offline_Uncorrectable   -O--CK   100   100   050    -    0
199 UDMA_CRC_Error_Count    -O--CK   100   100   050    -    0
232 Available_Reservd_Space -O--CK   100   100   050    -    100
241 Total_LBAs_Written      ----CK   100   100   050    -    280113
242 Total_LBAs_Read         ----CK   100   100   050    -    571926
245 Unknown_Attribute       -O--CK   100   100   050    -    619056
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      1  Comprehensive SMART error log
0x03       GPL     R/O      1  Ext. Comprehensive SMART error log
0x04       GPL,SL  R/O      8  Device Statistics log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x24       GPL     R/O     88  Current Device Internal Status Data log
0x25       GPL     R/O     32  Saved Device Internal Status Data log
0x30       GPL,SL  R/O      9  IDENTIFY DEVICE data log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log

SMART Extended Comprehensive Error Log Version: 1 (1 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      2931         -
# 2  Short offline       Completed without error       00%      2303         -
# 3  Extended offline    Aborted by host               90%        28         -
# 4  Extended offline    Aborted by host               90%        28         -
# 5  Extended offline    Completed without error       00%        27         -
# 6  Short offline       Completed without error       00%        27         -

Selective Self-tests/Logging not supported

SCT Status Version:                  3
SCT Version (vendor specific):       0 (0x0000)
Device State:                        Active (0)
Current Temperature:                    13 Celsius
Power Cycle Min/Max Temperature:     13/13 Celsius
Lifetime    Min/Max Temperature:      8/31 Celsius
Specified Max Operating Temperature:   100 Celsius
Under/Over Temperature Limit Count:   0/0

SCT Data Table command not supported

SCT Error Recovery Control command not supported

Device Statistics (GP Log 0x04)
Page  Offset Size        Value Flags Description
0x01  =====  =               =  ===  == General Statistics (rev 1) ==
0x01  0x008  4             158  ---  Lifetime Power-On Resets
0x01  0x010  4            2967  ---  Power-on Hours
0x01  0x018  6      1177649772  ---  Logical Sectors Written
0x01  0x020  6        70604304  ---  Number of Write Commands
0x01  0x028  6      3122025331  ---  Logical Sectors Read
0x01  0x030  6       351643988  ---  Number of Read Commands
0x07  =====  =               =  ===  == Solid State Device Statistics (rev 1) ==
0x07  0x008  1               0  ---  Percentage Used Endurance Indicator
                                |||_ C monitored condition met
                                ||__ D supports DSN
                                |___ N normalized value

Pending Defects log (GP Log 0x0c) not supported

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x0001  4            0  Command failed due to ICRC error
0x0002  4            0  R_ERR response for data FIS
0x0005  4            0  R_ERR response for non-data FIS
0x000a  4            3  Device-to-host register FISes sent due to a COMRESET

Smarctl appears to be happier with newer sda.

Unfortunately my BTFS volume is not raid. So btrfs scrub can’t fix.

My only option is btrfs check --repair or btrfs rescue.

Has anyone got experience with these given the warnings in the man pages on these suggest ‘you need to know what your doing’. I don’t know what I am doing.

I also not from manuals that one should monthly do btrfs scrub scan to check for btrfs fs problems. I didn’t do that. Not that that would have made a difference.

Further in poking around with BTRFS tools I found this:

$ ls -l
total 8
-rw-------. 1 root root 1186 Aug  4 21:56 scrub.status.2d85cf85-6165-49a8-8fd1-68651fe12fdc
-rw-------. 1 root root  426 Mar 28 19:48 scrub.status.8d0543c5-d926-414d-84c0-98331b66d713

March 2023 is when I first build the btrfs volume on the new 1TB SSD and merged with older SAMSUNG 830 SSD described earliier.

Re: -rw-------. 1 root root 1186 Aug 4 21:56 scrub.status.2d85cf85-6165-49a8-8fd1-68651fe12fdc
we have

scrub status:1
2d85cf85-6165-49a8-8fd1-68651fe12fdc:1|data_extents_scrubbed:11431182|tree_extents_scrubbed:1012860|data_bytes_scrubbed:726919524352|tree_bytes_scrubbed:165946982
40|read_errors:0|csum_errors:8|verify_errors:0|no_csum:0|csum_discards:0|super_errors:0|malloc_errors:0|uncorrectable_errors:8|corrected_errors:0|last_physical:10
20093530112|t_start:1691146615|t_resumed:0|duration:3547|canceled:0|finished:1
2d85cf85-6165-49a8-8fd1-68651fe12fdc:2|data_extents_scrubbed:1458232|tree_extents_scrubbed:0|data_bytes_scrubbed:93297893376|tree_bytes_scrubbed:0|read_errors:0|c
sum_errors:0|verify_errors:0|no_csum:0|csum_discards:0|super_errors:0|malloc_errors:0|uncorrectable_errors:0|corrected_errors:0|last_physical:127776325632|t_start
:1691146615|t_resumed:0|duration:1873|canceled:0|finished:1
2d85cf85-6165-49a8-8fd1-68651fe12fdc:3|data_extents_scrubbed:1483532|tree_extents_scrubbed:0|data_bytes_scrubbed:95387672576|tree_bytes_scrubbed:0|read_errors:0|c
sum_errors:0|verify_errors:0|no_csum:0|csum_discards:0|super_errors:0|malloc_errors:0|uncorrectable_errors:0|corrected_errors:0|last_physical:127776325632|t_start
:1691146615|t_resumed:0|duration:1882|canceled:0|finished:1

and for -rw-------. 1 root root 426 Mar 28 19:48 scrub.status.8d0543c5-d926-414d-84c0-98331b66d713
we have

scrub status:1
8d0543c5-d926-414d-84c0-98331b66d713:1|data_extents_scrubbed:21203796|tree_extents_scrubbed:2738825|data_bytes_scrubbed:940177608704|tree_bytes_scrubbed:44872908800|read_errors:0|csum_errors:0|verify_errors:0|no_csum:0|csum_discards:229535549|super_errors:0|malloc_errors:0|uncorrectable_errors:0|corrected_errors:0|last_physical:988955017216|t_start:1679981493|t_resumed:0|duration:11819|canceled:0|finished:1

Now march was my first try on BTRFS and using luks1 while also experimenting with random byte write to disk to hide luks better. New to the crypto thing. Was all experiments and learning curve stuff.

Anyway this meant I had about 5 goes at creating the merged btrfs volume over the three luks1 partitions sda3, sdb1, sdb2. Each time I was unhappy, I wiped disk by writting using dd /dev/random. So maybe I killed new SSD with the write cycle? Would this show up in the smarctl output if exceeding write cycle? The above smartctl don’t seem to show anything.

Given in the 1.2TB BTRFS merged volume I have 8 csum error blocks (4k or 16 K csum blocks?) and their not raided (other then my intent of mirror using rsync of the borg archive) my guess is the impacted borg file 213 is done and unrecoverable or repairable.

Am I better of wipping the BTRFS Volume?

I also note from google search that csum errors in BTRFS are not uncommon but if a raid arrangement it repairs itself. So maybe a single BTRFS arrangement I am using is not a good idea for my use case of simple borg backups mirrored to external usb using rsync.

In that case, if I should rebuild the backup system, I am leaning towards LVM ext4 arrangement of sda3 & sdb for my borg backups backed by the original rsync intent. In the past ext4 on SSD has never given me any issues EVER.

From my F38 upgrade when I first trialled BTRFS for my backup, I have had bad experiences with BTRFS as noted in above posted link, my whine.

This csum experience only reinforces that my neutral and curious and lets give it a go position on BTRFS, has turned into being negative to this FS except maybe if you use it for raid like arrangement where it can self repair. But in that case I’d go with tried and proven ZFS.

My guess,

  1. BTRFS not a good choice for my use case?
  2. sdb is EOL and maybe the source of csum problems?
  3. Or is csum not uncommon re BTRFS as google searches suggest but not problem in merged raid arrangements due to self repair?
13

assuming lba is a 512 byte block. Then you have written 17.9TiB.
Assuming lba ia a 4k block. Then you have written 143TiB
Best i can find is that the 830 256GB drive can write 10GiB a day for 64 years.
That is a written limit of 230TiB, if i did my sums correctly.

Provided the electronics is not failing and the wear levelling is working, your old ssd should still be within the lifetime of the flash.

I have samsung ssd’s in my desktop machine and dual boot fedora and windows. That allows me to run the Samsung magician software to check driver health. Sadly they do not have tools for mac or linux.

14

Unless you have some special use case, I don’t agree. I run BTRFS on three SSD’s, 2 @ 240GB both older ssd’s , and one 1TB Samsung Evo. I find the setup stable, and no ext4 needed except for the boot. I prefer the snapshot capability of the btrfs combined with send/receive over using rsync or Borg.

Maybe, but my smaller ssd’s were purchased beginning in 2010-ish and they are bargain ones. They can handle a large number of writes, especially newer ones.

I don’t get filesystem checksum errors with btrfs as a rule, I have been using it for a couple of years. Since prior to it becoming default on Fedora. I have had it setup as a raid 1 array using the two 240GB drives in the past, when my 1TB was spinning rust.

15

I don’t get filesystem checksum errors with btrfs as a rule

Yeah but that’s what baffles me. A new SSD plus the older and from the two scrub results above ( only done twice over 5 builds) get 2 csum error reports.
Add to that the recommendations to do scrub 1 /month, I wonder.
What’s the chance of this many csum errors so soon?

Now August test could make sense since due to kernel no boot update mentioned I had to kill power abrubtly many times during June to help debug issue. I know the non-enterprise SSD general do not have protective circuitry and therefore are susceptible to bit corruption.

That’s the only explanation for a new setup csum problems or sdb is EOL? :confused:

16

To write to the ssd:

  1. A buffer in memory is created
  2. The buffer is transferred over a bus to the ssd memory buffer
  3. The ssd wear levelling firmware eventually writes the buffer to flash

The csum error can happen at any of the stages.
For (1) Have you check for memory errors?
For (2) Have you check cables and the sockets they are plugged into?

17

Well I’m certainly not going to speculate on bit corruption or bit rot, as I’m not that familiar with the mechanics of how the failures occur. In my case I use a crucial ssd, a kingston ssd, and a samsung. The Kingston is arguably the least expensive, with the crucial being slightly more costly, but neither were as costly as the samsung on a per bit of storage basis. Plus facebook I am led to understand (by an employee at the time) uses these commodity consumer ssd’s in their production servers, all using btrfs.
The power cycling will cause damage to the file system, but not always the physical device, in fact there is a likelihood that your data also has a valid backup due to the CoW nature of BTRFS.
From the BTRFS site FAQ’s …
“Please note that checksumming is also turned off when the filesystem is mounted with nodatacow”
Maybe some of your system get’s mounted as nodatacow. I’m thinking specifically the VM’s or possibly the borg backup file.
A good info source is Btrfs Documentation and Resources · Btrfs

18

I have server samsung high quality ECC memory. These are certified by Intel for this mobo. No noticeable error from system. memx86 extensively tested with no issues for days when purchased.

Re older ssds no change in cabling which appears fine and before going to btrfs ran ext 4 without ever having issues such as basic i/o error re a ‘cp’ or rsync.

Given smarctl results above and your various advice on these I am wondering if it’s a filesystem issue. I’ll study link provided.

It is curious however that various places discussing BTRFs and particularly scrub that they recommend writing a scheduled service monthly for scrub. I get the impression that csum is not uncommon.

19

The larger the filesystem, the greater the chances of a csum error.

I would look for a firmware update. Your CXM03B1Q was released in Jan. 2012. Dell has/had? a 26 Sep. 2012 CXM76D1Q, A01"Critical" Update that says “Fixed the potential issues with Dell Data Protection Access during sudden power off.” for 256GB SAMSUNG SSD 830 Series.

20

Thanks for that.

I have not updated SSD f/w sever. Mainly cause with Linux it’s more difficult cause Samsung makes tool for MS (even virtualized can’t do) and habit since one never really did that in the past re HDDs.

That could explain it and makes more sense in the context of my events. I had a suspicion re the power offs I was forced into.

Given the numerous SSDs, this power off issue should be looked into by kernel developers. In the utopian world if kernel fails boot somehow one should have an orderly shutdown option. Then again you could only do that with UEFI.

Not good killing SSDs with the big power button, certainly the ones with the lacking power down protective function the marketing brochures never mention. I.e. the cheaper ones!

JUST TO FLOG THE DEAD HORSE borg sees the same

$ sudo borg --info -p check --verify-data /srv/lpssd/archives/borg-assimilation/earth
[sudo] password for robertk: 
Starting repository check
Local Exceptionts 7.3%                                                                                                                                            
Traceback (most recent call last):
  File "/usr/lib64/python3.11/site-packages/borg/archiver.py", line 5213, in main
    exit_code = archiver.run(args)
                ^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/site-packages/borg/archiver.py", line 5144, in run
    return set_ec(func(args))
                  ^^^^^^^^^^
  File "/usr/lib64/python3.11/site-packages/borg/archiver.py", line 183, in wrapper
    return method(self, args, repository=repository, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/site-packages/borg/archiver.py", line 343, in do_check
    if not repository.check(repair=args.repair, save_space=args.save_space, max_duration=args.max_duration):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/site-packages/borg/repository.py", line 1039, in check
    objects = list(self.io.iter_objects(segment))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/site-packages/borg/repository.py", line 1512, in iter_objects
    size, tag, key, data = self._read(fd, self.header_fmt, header, segment, offset,
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/site-packages/borg/repository.py", line 1606, in _read
    data = fd.read(length)
           ^^^^^^^^^^^^^^^
**OSError: [Errno 5] Input/output error**

Platform: Linux earth 6.4.7-200.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Jul 27 20:01:18 UTC 2023 x86_64
Linux: Unknown Linux  
Borg: 1.2.4  Python: CPython 3.11.4 msgpack: 1.0.4 fuse: llfuse 1.4.2 [pyfuse3,llfuse]
PID: 17215  CWD: /home/robertk
sys.argv: ['/usr/bin/borg', '--info', '-p', 'check', '--verify-data', '/srv/lpssd/archives/borg-assimilation/earth']
SSH_ORIGINAL_COMMAND: None
21

Let power offs re SSD be a warning to others!

My Borg test suggest repo is completely kaput as a result. Note that when I first build the repo on BTRFS brog check was fine. It’s developed since, maybe due to the power off issues.

Given there’s only 8 blocks stuffed re csum. Is there away of telling the OS / BTRFS F/S and/or SSD to isolate the stuffed part and rebuild the archive from scratch after deleting the old one. Or is this csum issue inherent and will reoccur over time?

Or is the BTRFS Vol. completely compromised and I should wipe, re-test SSD and rebuild from scratch.

In that case, am tempted to scrap BTRFS idea, go back to LVM/ext4 as I did per earlier mentioned ‘timeshift’ BTRFS whine from which I haven’t looked back.

I now need a reason to persist with BTRFS?

This could mean I would turn my back on BTRFS for ever if successful ext4 archive. It seems to me that BTRFS is only ‘great’ if you use the full power of it in it’s natural inherent raid configuration. Not as I have used as ‘single drive’ as it can’t repair csum errors. It then provides no advantage over simpler LVM merged ext4 volumes?

Real problem if I can’t trust my backup system?

23

You may indeed find that ext4 does not report errors.

That does not mean that the errors are not occurring, just that ext4 is not as good as btrfs at detecting them.