Fedora 44 grinds to a halt when copying large files off an NFS server

Ask Fedora
, ,
1

When I try to copy a large dir ~75GB from an NFS server to my downloads folder, the system quickly grinds to a halt. Apps start slowing down and freezing, animations freeze, and eventually the whole display freezes and touching the keyboard or mouse does nothing so I have to hard reset the computer.

I have tried copying the files through 2 ways and got the same result. The first way was through dragging and dropping in Nautilus, and the second was by using rsync:

rsync -avh --progress \
        "/path/to/dir/on/nfs/mount" \
        "$HOME/Downloads/"

The NFS share is simply mounted to a dir in /mnt with sudo mount -t nfs

Both the root and home dir are on the same NVME SSD.

While the transfer happens, I have tried to keep an eye on RAM anc IO, but neither looks fully saturated. Only about 6GB of the 32GB of RAM installed are used by the whole system. IO write tops out at only around 20-40MiB/s.

Any help would be greatly appreciated.

2

What does top show is happen to CPU usage?
Is the system logging anything while the transfer is happening?

You could run

sudo journalctl --follow

Then do the transfer to see if there is anything interesting reported.

3

Hi, I think I found the problem.

I think it could simply be the fact that I was transferring many small files on a LUKS encrypted BTRFS volume on a Crucial P3. Not a good combination for a DRAM-less SSD I suspect.

4

The writes from the copy starve the requests from the rest of the system.

Fedora uses the “none” io scheduler by default for ssds. See Set the I/O scheduler based on file system type? - #8 by chrismurphy for selection of io scheduler.

5

There it is. . . That’s surely going to go to a snail’s crawl after the first 6GB are through.

6

In comparison the Samsung EVO 980 that I have in my system has 2GiB of dram.

The other thing you need to care about with SSDs is the “endurance” of the drive expressed in TiB written. Cheap drives will have a low “endurance” rating and will fail early once you hit the write limit.

Beware of any SSD that the vendor does not publish the endurance figure in its spec of the drive.

7

Btrfs aggressively delays allocation, accumulating small writes into sequential writes, and also will inline very small data extents (around 4KiB or less, I think) into the metadata leaf, which is 16KiB by default.

I would try another IO scheduler, but only expect modest improvement. BFQ potentially could do better in this case by artificially prioritizing IO for other processes that are stalling out. My main complaint about BFQ is that it’s complex, but that shouldn’t prevent you from trying it out. It is a counterintuitive choice for an SSD. The more expected choice would be mq-deadline.

Ideally we’d have some kind of dynamic estimator of IO max throughput and minimum latency, and then assign a minimum throughput and latency resource to processes using systemd scope and cgroupsv2 support, to make sure the system responds like a human user expects. That is still on the radar and would help both the server and desktop use cases, even though the selected processes that get minimum resources will vary among use cases.

8

See BFQ, Multiqueue-Deadline, or Kyber? Performance Characterization of Linux Storage Schedulers in the NVMe Era.

A key finding is:

“(iii) Kyber and BFQ can deliver 99.3% lower P99 latency than None or MQ-Deadline schedulers in the presence of multiple interfering workloads.”

For this specific case, background writes interfere with low latency reads, the Kyber scheduler may also be a good option:

“(i) BFQ offers better control over interference (lower P99 latencies) for latency-sensitive workloads (while trading performance, §4) than other schedulers;
(ii) Kyber and BFQ excel in managing read-write interferences among concurrent workloads. However, Kyber has poor performance with read-read interference.”