Short version:
There’s a bug somewhere. Either a kernel bug, or a drive firmware bug. Somehow the log tree didn’t fully get written correctly due to the bug.
But for now I think your effort is best off with the following priorities:
- Backup. Commit to it. Do it often.
- See if the drive has a firmware update available and apply it.
- Update the kernel. We always say this because bugs are constantly being found and fixed. But also because of how kernel development works.
Long version:
The log tree is created when fsync is being used. The tree permits performance optimization for fsync heavy workloads. The fact the log tree exists means something that uses fsync was recently issuing writes to the drive.
If there’s a crash or power failure, the kernel Btrfs code will replay the log tree and update the file system. it’s similar to a journal on other file systems, except on Btrfs the file system is still consistent without replay. However, without log tree replay, some data ends up being lost because it isn’t referenced. It could be a few seconds or even up to a few minutes of writes lost. So log tree replay is a good thing.
The failure to recover the log tree means something else went wrong. The log tree itself is somehow malformed. Or possibly you’ve hit a bug (what kernel version was being used when the problem first appeared? And what kernel version was being used during the boot immediately prior to the problem first appearing?)
There are well tested write ordering rules to make sure Btrfs writes to the log tree are actually flushed to the drive before the super blocks are updated. The superblock contains the block for the log tree root. If log tree isn’t being used (no fsyncs), the superblock contains a 0 for the log tree root - there is no log tree. That should remain 0 until the log tree is fully and completely written to disk. And then the superblock is updated to point to the log tree.
But there are some cases where the drive firmware has a hiccup and doesn’t always honor the correct write order. In that case the superblock pointing to the log tree is committed to stable media before the log tree is fully committed to stable media - and then there’s a crash/power failure - and at next boot, the kernel seens the superblock with log tree address, tries to do a log replay, but the log tree is incomplete or otherwise malformed so it can’t be replayed.
There may not be a lot you can do about this. Ideally there’s a drive firmware update available and you can apply it. Maybe that fixes the problem. The end user notes for firmware updates don’t always have details on what’s been fixed. And another possibility is a kernel update fixes the problem, if it turns out this is a kernel bug.
A kernel bug tends to happen more frequently. Enough that they get caught in testing. And there’s tons of tests being run constantly by upstream developers and distributions and consumers of the kernel. Whereas drive firmware bugs are almost non-deterministic. They don’t always happen, and they also don’t matter unless there’s an ill timed crash or powerfailure at just the right (wrong) moment. So this bug could be happening often, even daily, and it just doesn’t matter because you don’t get a crash or power failure at that same time. It takes more than one problem at the same time for both problems to manifest.
Btrfs metadata might be able to provide some clues - you’d need to report this to upstream and get their opinion on it though as it will involve posting portions of the on disk file system metadata: we can infer what happened from this metadata, but only if it’s very fresh. Like, by the time you’ve run zero log on the file system, it’s already changed too much and the information developers need to figure out what went wrong is already gone. But for now, if you made it this far, just follow the advice in the “short version” wayyyyy up at the top of this post. If it happens again, then we can consider digging into it more.
Alright this is probably long enough. 