Why aren't there any "micro-optimization" tools for ext4?

Ask Fedora
,
1

Context

Nowadays less and less (probably because of the prevalence of SSDs), but there are still people asking about “how to defrag ext4 filesystems”? The current “canonical” answer is basically this gist.

Although many argue that it’s just not necessary, because of how the file system decides file allocations. Still, for slower storage systems (think media center backed by a HDD-based NAS), some people argue that continous allocation of bigger media files is the way to go. This is nuanced by the fact that most such system are running over a hardware/software RAID, and we have extensive (multi-hundred-MiB) buffering nowadays built into the HDD hardware itself, and then on top of that buffering by the OS and the media player, so it’s just not a problem if a 2 hour film is stored in 100+ extents on a file system, even if it is multiple tens of GiBs of data.

However…

Question

… some people still are overly sensitive about this, and for NTFS there even exist what I call “micro-optimization” tools (e.g., DiskTrix’s UltimateDefrag), which allow quite fine-grained rule-based allocation on the physical disk itself. Their argument is that putting many small files towards the outer edges of the disks makes for less head-movement (alas, I have no idea how they achieve that, because I don’t think that an HDD provides the data about the physical platter layout, how many they have, etc.), thus making file access faster.

The question is: How come there are no such tools for Linux/ext4 (and more generally for other file systems)? Is it that way, because it’s basically snake-oil and in the open source community nobody thought of investing uncountable hours of work into something so futile?

P.S.: During writing this question I realized some things myself, but I still thought, I should post the question, because

a) I haven’t found anywhere a “definite” answer to this;

b) Instead of writing a blog-post that nobody will read, here some experts can chime in and validate my conclusions;

c) Validating my assumption about defragmentation of ext4 in general, in case somebody answers to this.

2

Have you measured your fragementation levels?

As I found this blog https://www.baeldung.com/linux/ext4-filesystem-defragment

Note that these days servers tend to use xfs and desktop systems use btrfs.
ext4 I only have on my old server system.

1 Like
3

My question is actually filesystem independent and independent of defragmentation. And yes, I had fragmentation on my partitions, even though they were mild if seen percentage-wise, but significant if seen on the level of the some files I’m interested in (even though whether this fragmentation has any significant real-world impact is another question). I “want” to have a defragmented file system, because of whatever reason (OCD, cyber-homeopathy, etc.).

The same arguments hold for xfs and btrfs too: currently I am not aware of any tool that lets me decide the “exact” placement of data mapped onto the physical disk. The nuanced question is: why aren’t there such tools?

  • They don’t have a raison d’être? (Is it really a snake-oil argument nowadays?)
  • Nobody built the tools? (Yet?)
  • The tools are too hard/impossible to build? (With btrfs it feels extremely tricky to do)
4

This I think. The reason for placement was all to do with rotating disks and numbers of heads. Most of that detail is now hidden behind a logical to physical mapping that drives implement in their firmware. So you don’t know where data to going on the physical disk to attempt an optimisation.

5

Some may share the belief that spinning rust storage is a good replacement for tape archival systems and should be used primarily as such.

In several large data centers I have observed HDDs configured in RAID arrays where each drive was used until it failed before being replaced. Massive amounts of storage on each disk tended to minimize the effects of fragmentation.

Modern M.2 NVMe SSDs are faster and less power hungry than HDDs but trimming them periodically is required to minimize write amplification. UASP compatible USB sticks can and should also be trimmed regularly.