Btrfs "discard storm" on Fedora?

After using it for a some time now, I must say it’s incredibly annoying, my watercooled (360mm AIO) Ryzen 5900X spikes from normal 40 degrees to 65 every 8-10 minutes for ~15 seconds.
I have 6 drives with btrfs and probably lower drive count does not have this issue, but it’s very annoying. Cannot imagine this being great for laptops either. If nodiscard added, then the system goes back to predictable load-based thermal mode.

Are we 100% sure that discard=async is the way to go?

3 Likes

image

Just to illustrate the point

Could you report this in https://bugzilla.redhat.com/show_bug.cgi?id=2182228

Include kernel version, a description of the drives (make/model) being used by btrfs, and importantly that the problem does happen with (default) discard=async but does not happen with nodiscard. It would be useful to know something about the workload too.

For what it’s worth, I’ve used the feature on my laptop since it first appeared many kernel versions ago with no noticeable side effects. We still need to understand your edge case though.

In the meantime, you are welcome to continue to use nodiscard.

2 Likes

btrfs devs added ‘discard=async’ for a reason. If the resulting TRIM/UNMAP process is too aggro then it might be prudent to report this to btrfs devs, no? Waiting a week to push UNMAPs down to storage is somewhat unusual.

Windows trims storage in quasi real-time and so does VMware ESXi. They’ve been doing it for roughly ten years and it forced storage vendors to clean up their code paths – $500k arrays would get bogged down so badly that VMware had to disable automatic unmap for several releases. It’s all fixed now. The problem didn’t get fixed by giving up on automatic/ongoing unmaps.

As for drive wear, waiting a week to houseclean increases the chances of the drive doing read-modify-writes on blocks holding “junk” data leading to unnecessary write amplification. Better to tag those blocks for deallocation early-on. It also gives the drive’s internal wear-leveling algos more space to work with.

Don’t let the tail wag the dog here.

Edit: Nevermind… They already know about it.

Edit2: More background.

The bugzilla issue linked above has that context in it. That whole thread on kernel.org has been pretty interesting. Again, Facebook data centers were the basis for deciding it was ready for everything. Arguably, the Fedora testing for the fstrim timer likely had a more representative hardware sample for consumer ramifications than the kernel team did here. We skipped past that testing this time around because upstream shipped it, but perhaps that testing would have yet been useful.

https://lore.kernel.org/all/ZC0SyJbFJuS2ZEOY@infradead.org/

1 Like

3 posts were split to a new topic: Btrfs async discard leads to higher power consumption

So, seems to be some changes to address the problem were merged into 6.3-rc7 kernel. Since 6.3 got released yesterday, I’ll re-test as soon as there a 6.3 kernel build.

1 Like

I’ve installed 6.3.0-63.fc39.x86_64 kernel on my desktop and a laptop, so I’ll be looking into the results of this latest patch. After running for a few hours on my desktop machine, does not look like I have said spikes in CPU usage anymore, but it’s too soon to tell imo.

Laptop is a better indicator, as I have a FDE there, but it’s now in a separate thread

2 Likes

I just looked on my F39 install (upgraded from F38) and I do not see discard=async
in the fstab file. However, I also don’t see nodiscard

AFAIK, upgrade never tinkers with existing fstab, so if you want that you would need to modify it manually, I have a fresh installation of F39 on my laptop and it does have discard=async.

1 Like

The default in the kernel itself is what was changed, IIRC, so if it’s not explicit in fstab, then you have async.

While true, this actually was a change in default behavior in the middle of a release. But if you had added nodiscard to fstab, then that behavior would have persisted. IIRC, neither were given by default, so when the kernel default changed, it also changed the behavior for all Fedora btrfs users when they installed and booted into the new kernel.

Yeah, there was a change in defaults in kernel, but the problem was actually caused by a very short window for the discard that wasn’t sufficient to actually discard and hence every window it was trying to discard ever growing queue. That was also fixed later and I have not seen any issues on any of my systems since then. Don’t have a fix or links at hand, but it’s no issue at this point

1 Like