RSYNC reports input/output error and btrfs source has file read error but check ok

Let power offs re SSD be a warning to others!

My Borg test suggest repo is completely kaput as a result. Note that when I first build the repo on BTRFS brog check was fine. It’s developed since, maybe due to the power off issues.

Given there’s only 8 blocks stuffed re csum. Is there away of telling the OS / BTRFS F/S and/or SSD to isolate the stuffed part and rebuild the archive from scratch after deleting the old one. Or is this csum issue inherent and will reoccur over time?

Or is the BTRFS Vol. completely compromised and I should wipe, re-test SSD and rebuild from scratch.

In that case, am tempted to scrap BTRFS idea, go back to LVM/ext4 as I did per earlier mentioned ‘timeshift’ BTRFS whine from which I haven’t looked back.

I now need a reason to persist with BTRFS?

This could mean I would turn my back on BTRFS for ever if successful ext4 archive. It seems to me that BTRFS is only ‘great’ if you use the full power of it in it’s natural inherent raid configuration. Not as I have used as ‘single drive’ as it can’t repair csum errors. It then provides no advantage over simpler LVM merged ext4 volumes?

Real problem if I can’t trust my backup system?

You may indeed find that ext4 does not report errors.

That does not mean that the errors are not occurring, just that ext4 is not as good as btrfs at detecting them.

Sorry more digging as I get familiar with btrfs tools

$ sudo btrfs scrub status -dR --human-readable /srv/lpssd/archives
UUID:             2d85cf85-6165-49a8-8fd1-68651fe12fdc

Scrub device /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf (id 1) history
Scrub started:    Mon Aug  7 15:32:46 2023
Status:           finished
Duration:         1:10:04
        data_extents_scrubbed: 11419706
        tree_extents_scrubbed: 1012878
        data_bytes_scrubbed: 726756278272
        tree_bytes_scrubbed: 16594993152
        read_errors: 0
        csum_errors: 8
        verify_errors: 0
        no_csum: 0
        csum_discards: 0
        super_errors: 0
        malloc_errors: 0
        uncorrectable_errors: 8
        unverified_errors: 0
        corrected_errors: 0
        last_physical: 1020093530112

Scrub device /dev/mapper/luks-1e32597b-d73f-48de-a899-af443d96b150 (id 2) history
Scrub started:    Mon Aug  7 15:32:46 2023
Status:           finished
Duration:         0:23:51
        data_extents_scrubbed: 1458753
        tree_extents_scrubbed: 0
        data_bytes_scrubbed: 93300781056
        tree_bytes_scrubbed: 0
        read_errors: 0
        csum_errors: 0
        verify_errors: 0
        no_csum: 0
        csum_discards: 0
        super_errors: 0
        malloc_errors: 0
        uncorrectable_errors: 0
        unverified_errors: 0
        corrected_errors: 0
        last_physical: 127776325632

Scrub device /dev/mapper/luks-9b746590-b83b-4e35-bb7c-1c07d5130a4f (id 3) history
Scrub started:    Mon Aug  7 15:32:46 2023
Status:           finished
Duration:         0:24:43
        data_extents_scrubbed: 1494444
        tree_extents_scrubbed: 0
        data_bytes_scrubbed: 95547424768
        tree_bytes_scrubbed: 0
        read_errors: 0
        csum_errors: 0
        verify_errors: 0
        no_csum: 0
        csum_discards: 0
        super_errors: 0
        malloc_errors: 0
        uncorrectable_errors: 0
        unverified_errors: 0
        corrected_errors: 0
        last_physical: 127776325632

Which means Samsungs are good but the bargain /dev/sda3 used on the T-Force is the one with the problem.

Also ran sum checks with Borg and borg clearly has a bug coping with OS I/O error that BTRFS F/S has spat out. Looks like my entire Backups are kaputt

$ sudo borg --info -p check --verify-data /srv/lpssd/archives/borg-assimilation/earth.before-upgrade-2023-05-14-15:32:47
Starting repository check
Local Exceptionts 13.0%                                                                                                                                           
Traceback (most recent call last):
  File "/usr/lib64/python3.11/site-packages/borg/archiver.py", line 5213, in main
    exit_code = archiver.run(args)
                ^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/site-packages/borg/archiver.py", line 5144, in run
    return set_ec(func(args))
                  ^^^^^^^^^^
  File "/usr/lib64/python3.11/site-packages/borg/archiver.py", line 183, in wrapper
    return method(self, args, repository=repository, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/site-packages/borg/archiver.py", line 343, in do_check
    if not repository.check(repair=args.repair, save_space=args.save_space, max_duration=args.max_duration):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/site-packages/borg/repository.py", line 1039, in check
    objects = list(self.io.iter_objects(segment))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/site-packages/borg/repository.py", line 1512, in iter_objects
    size, tag, key, data = self._read(fd, self.header_fmt, header, segment, offset,
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/site-packages/borg/repository.py", line 1606, in _read
    data = fd.read(length)
           ^^^^^^^^^^^^^^^
OSError: [Errno 5] Input/output error

Platform: Linux earth 6.4.7-200.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Jul 27 20:01:18 UTC 2023 x86_64
Linux: Unknown Linux  
Borg: 1.2.4  Python: CPython 3.11.4 msgpack: 1.0.4 fuse: llfuse 1.4.2 [pyfuse3,llfuse]
PID: 52815  CWD: /home/robertk
sys.argv: ['/usr/bin/borg', '--info', '-p', 'check', '--verify-data', '/srv/lpssd/archives/borg-assimilation/earth.before-upgrade-2023-05-14-15:32:47']
SSH_ORIGINAL_COMMAND: None

This suggest that I should try btrfs check --repair as there’s nothing to loose now. Backups are a dud anyways.

This does mean that SAMSUNG 830 firmware is not the issue.

Sadly logs are no help as I just discovered my borg-assimilate script has a bug in the logging command. At least I now know why I know nothing about this.

Not a problem cause borg backup puts in it’s own checksums. No point re checksums on top of checksums. In that sense the BTRFS doesn’t gain me anything.

Borg can at least cope with it’s own csum errors but not with BTRFS induced OS IOerror as above. As per timeshift experience a regression error with borg app.

From the article while 9 months old:
Pros and Cons of Using Btrfs Filesystem in Linux

My recommendation is that if you have an SSD and use a rolling release distribution then go for it for the seamless recovery using snapshots. Otherwise, use the good old Ext4 for stability and performance.

Meaning for my use case if BTRFS check --repair does not work next for me since I then have to rebuild backup disk anyway I will go to LVM ext4. BTFRS is done for me for another decade before I’ll consider it again.

My backups need stability - PERIOD! I currently have 5 months of duds cause of a total of 32 kB of faulty faulty data in 800 GB and so far 24 hours of labour figuring this out.

Borg can only checksum what it reads off of the ssd.
It cannot tell that that data is already corrupted.
That is why modern file systems add checksums to be able to tell you about issues.

Yes but it can only tell me about issues just like Borg will. I agree that modern file system should add checksums … but then what? they then also have problems fixing it unless you have a raid arrangement. Your no better off but have more complexity and poorer performance. Nothing gained just a lot of wasted time as this thread exhibits. Fine for the budget of Facebook has, but I can’t afford a multitude of ssds for raid yet.

My point is that you will potenitally have backups that contain corrupted files.

Personally I’d like to know that is happening sooner rather then later.

It seems that you need, some how, to get the firmware updated on your SSDs.

Refer (though 2014) to
Marc’s Public Blog - Linux Btrfs Blog Posts

Great modern filesystem add checksums to tell me my backup is corrupted. Errrgh how does that help. I still don’t have a backup. Screwed either way.

With bit rot etc OK some files are corrupted but now with this arrangement everything is corrupted and borg goes kapoom. So instead of a few faulty files I now have nothing? So exactly how did the btrfs csum help me when it tells fundamental OS commands like cp or rsycn IOerror and everything breaks due to the csum? Fundamental because everyone used them in their automation scripts in some way and now those scripts break?

I’d rather have a few corrupted files then a complete roadblock.

This is how btrfs check should work for BTRFS when you have single drive mode or you can’t repair with some other copy:

A csum error or maybe some others should be alerted to user as now but now give option to repair or remove damaged files if in some backup mode to the backup program like borg and give it file & paths such that it can recopy from original files on the actual system backed up. This can then repair the back up.

With rsync based difference backups like backintime or borg, a csum error if not addressed, the whole backup is stuffed.

BTRFS check needs more work and an API interface for backup apps to work withit to fix backups automatically and ensure reliability.

A firmware glitch probably messes with any filesystem, but btrfs can tell you sooner rather than later that there is a problem, and may allow you to travel back in time.

Which is why enterprise level backup systems are so expensive. I spent most of a summer rescuing irreproducible data archived to a fancy tape robot system from a CDC Cyber and then restored to Solaris. The data were stored as “flat” text files, but restored files had random duplicated blocks and often garbage at the end. I assume this was down to how CDC stored the “text” files. The backup for the files was in the form of printed data reports, but the orginal were discarded after they were scanned and converted to PDF’s. Some pages were not scanned properly, but I also had
summary statistics so could sometimes determine the correct choise for an illegible 5, 6, or 8.

9 years of improvement to btrfs since that post, and fedora only made it the default about 2 years back.

Things have improved a lot since that time, but I agree: I will wait some time to see what problems most users encounter and for fixes to those before I will make it a default on any of my systems.

I’ve been trialling BTRFS on my backup drives rather then root and home.

Used it as backup FS for borg and timeshift.

In both cases their apps broke regression wise. Timeshift GUI had issues reading size of drives of the BTRFS file system saying that they were full when they were not and so did not function properly. The moment I went back to ext4 voila all good.

And for borg bup drive you see the results above with blood stains on the brickwall from my head. Borg can’t handle OS IOerror due to csum error.

9 years of improvement to btrfs since that post, and fedora only made it the default about 2 years back.

My experience so far is that it’s 50/50 ready for default. Why?

Default users are often the lazy or new to Linux. They need a good easy experience. The BTRFS filesystem in general use and admin is not as easy to manage with standard current tools and need deep familiarisation with these tools which are much more complex. Eg. btrfs check --repair with all the warnings is just one extreme example. Then their btrfs scrub which is most useful if you have dup which mean either large expensive ssd or many ssds. Single drive for small scale or retail users it does not help eg this thread.

I do see that its easier for new users from scratch who go straight learning btrfs rather then being familiar with standard tried, true, reliable and true commands such as lsblk, fsck, df etc etc. These new users don’t know better. So for a transition to new FS arrangement I see the argument to go BTRFS.

BUT as my experience shows it’s not an easy default experience if things go wrong and you rely on some common standard apps such as timeshift and borg for backups. Then also btrfs tools to diagnose are another world of endless manual reading or google searches to figure out. Now true, BTRFS snapshot (haven’t used them yet) may obviate the need for borg and definitely timeshift (used due to starting out with ext4 rootfs) as new tools are developed. But their not there yet either.

So I agree wait for a while yet before using BTRFS from my 6 months experience so far. Certainly not ready for enterprise (other then with in house IT expertise), and as default for new users, I am going to be a herretic and say wait a few years yet. Too early for default. BTRFS and general apps interfacing with it still need work. Fedora should provide a warning when default BTRFS is used by new users that some apps do not work well with this FS and some BTRFS management tools are still a work in progress.

That’s my conclusion after giving BTRFS a good shake over the last 6 months. Shows much promise, needs performance improvements, improved ease of use, apps need to catch up to this FS, and re csum errors needs way, way better management tools to fix without scrubbing in my case my entire backups particularly in single drive mode (needs a lot more thought by developers) as self repair and btrfs scrub can only tell you you have a problem but can’t really fix them.

In my case better to have a backup with a few files damaged to which you have been alerted rather then the whole backups a dud.

I still have to use btrfs check --repair, I’ll post result here to see what happens. I have to go watch the Matildas at Brisbane in the World Cup. So I’ll be gone for two weeks. Ozy Ozy - oi OI Oi.

I use the default btrfs configuration on Fedora Workstation, and xfs for backups. I chose xfs because I’ve been using since IRIX64 days, and expect it will be around for the long haul. Btrfs seems headed for a long and useful life, but not sure it is there yet.

For Fedora Workstation I want to stay as close to the defaults as possible to increase the chances others will also be using the same configuration. This helps in troubleshooting – if I’m the only one seeing a problem that points to hardware, and assuming the majority of users are also on default Workstation configurations I’m better able to help with problems.

1 Like

According to the information posted, the corruption is clearly happening on /dev/mapper/luks-dd129b86... which is on /dev/sda which is

Device Model:     T-FORCE 1TB
Serial Number:    TPBF2212050020101379

It’s common for SSDs to return garbage or zeros rather than a discreet error, such as an uncorrectable read error seen on spinning hard drives. Why manufacturers produce storage device that will report bad data rather than an error is beyond my comprehension but therein lies the problem with all file systems that do not checksum everything.

The scrub result in dmesg shows all information needed to identify exactly what file(s) are affected, including which snapshot they’re located in, for example:

Aug 03 11:03:35 earth kernel: BTRFS warning (device dm-12): checksum error at logical 3767118004224 on dev /dev/mapper/luks-dd129b86-469d-4042-8b90-00d1acb2a1bf, physical 147475595264, root 5, inode 28336187, offset 331862016, length 4096, links 2 (path: borg-assimilation/earth.before-upgrade-2023-05-14-15:32:47/data/0/213)

This tells you the error occurs on /dev/mapper/luks-dd129b86..., root 5 means “subvolume ID 5” which is the default subvolume created at mkfs time (therefore you can ignore this but if you had snapshots, containing this file, they will all share extents pointing to the same corrupt blocks). The inode number might be helpful in locating the file but since btrfs subvolumes each have their own 2^64 inode pool, you need to do an inode search in a specific subvolume. Last, there is a path to the corrupt file. The specific corruption is a 4KiB block, not the entire file. There’s enough information to use dd to copy this specific 4KiB block and inspect it - but I have no idea what it is or whether you’ll be able to tell the nature of the corruption. Usually it’s easier to just replace the file and delete the corrupt copies.

It is possible to disable csum verification, which will permit the corrupt data block to pass to user space. Normally Btrfs reports EIO to the application in order to prevent the replication of corrupt data. The mount option for this is rescue=idatacsums. This is quite a lot easier to use than btrfs rescue which likewise doesn’t verify checksums. But buyer beware, using this option thwarts a central purpose of Btrfs which is to only handover to user space data that has passed checksum verification.

This are bugs and in any open source community the expectation is to file bug reports or feature requests. Nothing happens without deliberate action. Did you file bugs? If so you should reference them here so others can help contribute, either by tracking the bug, fixing the bug, or providing a work around for the bug.

And for borg bup drive you see the results above with blood stains on the brickwall from my head. Borg can’t handle OS IOerror due to csum error.

There is a corrupt datablock, Btrfs is correctly refusing to hand over that corrupt data, and issues EIO to user space. Borg Backup should properly handle it, mainly by not crashing, and also by reporting the error rather than suggesting the backup (or restore) correctly completed.

Default users are often the lazy or new to Linux. They need a good easy experience. The BTRFS filesystem in general use and admin is not as easy to manage with standard current tools and need deep familiarisation with these tools which are much more complex. Eg. btrfs check --repair with all the warnings is just one extreme example.

That command isn’t indicated in this case. Anyone suggesting it is giving bad advice. That’s pretty common on the internet these days, it’s not unique to linux file systems.

BUT as my experience shows it’s not an easy default experience if things go wrong and you rely on some common standard apps such as timeshift and borg for backups.

In the exact same situation with another file system, the corrupt block would be replicated permanently into all of your backups, silently. That is a far worse user experience because notification of the problem may come so late that the consequence is data loss. That you don’t understand the error is a little bit beside the point, Btrfs did exactly what it’s designed to do which is prevent corruption for replicating - you have a warning something is wrong and you need to figure out what it is, if you care about the data.

Whereas other file systems, assume hardware is reliable and we know that’s false.

So I agree wait for a while yet before using BTRFS from my 6 months experience so far. Certainly not ready for enterprise (other then with in house IT expertise), and as default for new users, I am going to be a herretic and say wait a few years yet. Too early for default.

I think it’s bad advice that leads to a very pernicious form of data loss, given the exact same scenario with a different file system. It is very helpful to note that Btrfs is not some magical thing that can decorrupt your data. We still need backups. We still need multiple independent copies of important files. Btrfs still makes it easier to detect such problems before they are catastrophic.

BTRFS and general apps interfacing with it still need work. Fedora should provide a warning when default BTRFS is used by new users that some apps do not work well with this FS and some BTRFS management tools are still a work in progress.

Everything is a work in progress, nothing is bug free, and it’s not a good strategy to wait indefinitely for progress to be made based on concerns about lack of progress in another area.

That’s my conclusion after giving BTRFS a good shake over the last 6 months. Shows much promise, needs performance improvements, improved ease of use, apps need to catch up to this FS, and re csum errors needs way, way better management tools to fix without scrubbing in my case my entire backups particularly in single drive mode (needs a lot more thought by developers) as self repair and btrfs scrub can only tell you you have a problem but can’t really fix them.

This isn’t a filesystem problem. The storage stack has failed to return some of your data, and Btrfs is providing notification it’s detected this fact. There isn’t a way for it to fix this without a good copy of the data, i.e. using data profile dup, raid1, or higher. In the case of raid1, this becomes a completely different experience: btrfs detects the corruption on devid 1, looks for the block on devid2, assuming it isn’t also corrupt (it’s checked first), the good data is handed over to user space and btrfs overwrites the bad block with the good data. All of these activities are reported by the kernel (dmesg) but the handling is transparent to user space. It simply works.

In my case better to have a backup with a few files damaged to which you have been alerted rather then the whole backups a dud.

This is a question for backup application developers to answer. It’s not an issue for file system development. It might seem like a good idea for the backup application to make a “best effort” backup, but it can’t do this silently or else it’s leading to a false sense of security. The backup of your data is failed because it is a partial backup. The application can choose to continue to backup, even the remaining blocks in the corrupt file. And provide a report indicating the nature of the incomplete backup - but it is definitely incomplete if even a single bit isn’t backed up.

3 Likes