Full disk encryption performance can be really improved in NVMe and SSD devices

I was having performance issues downloading at 1 Gbps to my fully encrypted 3500 MB/s NVMe, which left the system unresponsive for a second or two, specially with Steam that does CPU and I/O intensive tasks while downloading games, related to decompressing and shaders processing.

I found this article from Cloudflare claiming that dm-crypt queuing is an unnecessary overhead with fast storage: Speeding up Linux disk encryption

So following this guide dm-crypt/Specialties - ArchWiki I enabled the flags

discard,no-read-workqueue,no-write-workqueue

in my /etc/crypttab and regenerated the initramfs with

$ sudo dracut --regenerate-all --force

This really improved the throughput for me, confirming Cloudflare conclusions and solving the freezing issues I had.

I’m thinking that this setting could be the default in Fedora, can other people test this? What do you think?

3 Likes

Thanks for posting this; I just enabled the workqueue-related flags, but I did it in the LUKS2 header (persistent mode) instead of crypttab.

2 Likes

Do you have a CPU with AES-NI ?

If not, the bottleneck is maybe using AES for disk encryption (it is used by default as most models have it). If the kernel cannot use a hardware-implementation (which is AES-NI), it has to fall back to a software implementation, which is a strong bottleneck (the bugzilla report below contains a comparison as example).

You can check with lscpu or lscpu | grep aes (there has to be a flag “aes”) or by checking your CPU model on the vendor website.

Some more elaboration about that issue:
https://bugzilla.redhat.com/show_bug.cgi?id=2077532

Adiantum is the alternative.

If you have questions about it, let me know.

2 Likes

Thanks for your suggestion, but I have it, it’s a Ryzen 5 3600, I did the benchmark just to confirm AES performance and it’s fine.

➜  ~ cryptsetup benchmark
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1      1804777 iterations per second for 256-bit key
PBKDF2-sha256    3449263 iterations per second for 256-bit key
PBKDF2-sha512    1593580 iterations per second for 256-bit key
PBKDF2-ripemd160  800439 iterations per second for 256-bit key
PBKDF2-whirlpool  678250 iterations per second for 256-bit key
argon2i      10 iterations, 1048576 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time)
argon2id     10 iterations, 1048576 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time)
#     Algorithm |       Key |      Encryption |      Decryption
        aes-cbc        128b      1258.5 MiB/s      4114.0 MiB/s
    serpent-cbc        128b       118.9 MiB/s       740.8 MiB/s
    twofish-cbc        128b       243.6 MiB/s       429.7 MiB/s
        aes-cbc        256b       957.2 MiB/s      3395.9 MiB/s
    serpent-cbc        256b       119.1 MiB/s       741.9 MiB/s
    twofish-cbc        256b       243.4 MiB/s       429.1 MiB/s
        aes-xts        256b      3334.1 MiB/s      3380.0 MiB/s
    serpent-xts        256b       654.6 MiB/s       641.9 MiB/s
    twofish-xts        256b       397.7 MiB/s       396.8 MiB/s
        aes-xts        512b      2829.4 MiB/s      2838.7 MiB/s
    serpent-xts        512b       654.3 MiB/s       643.0 MiB/s
    twofish-xts        512b       397.4 MiB/s       396.6 MiB/s

What scheduler did your system selected?

cat /sys/block/nvme0n1/queue/scheduler

none, the default for NVMe, the first thing I did was trying to change it to bfq and it helped with responsiveness but decreased throughput, changing dm-crypt flags was the best solution overall

1 Like

I agree. While this is just anecdata, overall performance seems to be better with the flags set, and this is especially noticeable when dnf update is installing a large package like kernel-headers.

Does anyone have updates on this?

I just added the lines to the /etc/crypttab with an Enter before.

sudo dracut --regenerate-all --force                                       ✘ 1
dracut: Can't write to /boot/efi/762dea3743ad4b369882c46e6992d7e9/6.2.13-300.fc38.x86_64: Directory /boot/efi/762dea3743ad4b369882c46e6992d7e9/6.2.13-300.fc38.x86_64 does not exist or is not accessible.

I am on Fedora Kinoite (kinoite-main from ublue) so this will probably be different. Would just updating to a new image fix the issue, as dracut will be reloaded?

I’m currently using Silverblue and initramfs seems to be handled different, so I did this:

$ sudo dmsetup table

When the flags are enabled you’ll see allow_discards no_read_workqueue no_write_workqueue at the end, probably that’s not the case after editing crypttab. You have to copy the device name from the output, it’s in the first column luks-blablabla, then you can run:

$ sudo cryptsetup --allow-discards --perf-no_read_workqueue --perf-no_write_workqueue --persistent refresh luks-blablabla

and if you check again in the table, the flags should be enabled.

1 Like

After patching /etc/crypttab, run:

rpm-ostree initramfs-etc --track /etc/crypttab
1 Like

Thanks, awesome! This should be documented

1 Like

Added docs-team

I played around some with the no_read_workqueue and no_write_workqueue options but they don’t result in any meaningful performance difference for me. I benchmarked the disks before and after setting these options and the results are very similar. I did have allow_discard enabled during the benchmarks, that one is probably useful but is a bit difficult to benchmark and depends greatly on the trim implementation of the disk.

Reading benchmark using fio with a buffer size of 1KB

fio --readwrite=read --numjobs 8 --size 50MB --bs=1KB --direct=1 --loops 100 --name=crypt
# before: bw=3385MiB/s (3550MB/s), 423MiB/s-427MiB/s (444MB/s-448MB/s), io=39.1GiB (41.9GB), run=11703-11816msec
# after: bw=3373MiB/s (3537MB/s), 422MiB/s-429MiB/s (442MB/s-450MB/s), io=39.1GiB (41.9GB), run=11647-11860msec

Reading benchmark using a buffer of 1MB

fio --readwrite=read --numjobs 8 --size 50MB --bs=1MB --direct=1 --loops 100 --name=crypt
# before: bw=3415MiB/s (3581MB/s), 427MiB/s-430MiB/s (448MB/s-451MB/s), io=39.1GiB (41.9GB), run=11618-11713msec
# after: bw=3412MiB/s (3577MB/s), 426MiB/s-436MiB/s (447MB/s-457MB/s), io=39.1GiB (41.9GB), run=11461-11725msec
1 Like

This is something that should for sure be documented. If somebody wants to…

Fedora Quick Docs

Badly written docs are better than no docs :wink:


FYI, I applied these parameters:

>> cat /etc/crypttab
luks-e755629a-0756-4ada-b584-e3dcc8a7f1b4 UUID=e755629a-0756-4ada-b584-e3dcc8a7f1b4 none discard,no-read-workqueue,no-write-workqueue

and afaik the performance on my 2TB Crucial CT2000P5PSSD8 NVME did decrease, at least with a KDiskMark test.

I disagree, Badly written Docs are the same as not having any. Badly written, can have the potential for mishaps and a bad experience. An example of this is the akmods + 5mins + other things for secureboot ordeal most users stumble over.

Also, the arduous update process of the Quick Docs or the GitLab vs. Pagure is VERY disillusioning. If we the community need Quick Docs, let’s write them ourselves. We can use the current doc as a basis.

It is worse. Badly written Docs content belongs to the reasons why people loose trust in Docs at all, and thus badly written Docs pages can cause damage also to other Docs contents. (I intentionally leave the definition of “badly written” open as this simplification can be used to describe many issues in Docs history and related discussions, but I would add that “badly written” is complemented by “badly maintained”, which should also always be considered)

I suggest to get in touch with the Docs team before organizing Docs stuff. I suggest to contact them with a dedicated topic or at the best, on Matrix: https://chat.fedoraproject.org/#/room/#docs:fedora.im if you are interested.

Also, please be more careful about tagging. You can open a docs-team topic and make them aware of your suggestions and link to this one - then the new one will be a docs-team topic. But just adding the tag to a topic that is not explicitly intended for docs-team might cause work, as one of them would then need to read throughout this topic to find out what this is about and why it is about docs-team - it is thus likely that it will be ignored and thus just blurs the tag in the end. This tag might also make it harder to find this topic if people are interested in its content (technical content about NVMe performance and what/why to make Fedora defaults - not how and when to write a Docs page about it and how/who to maintain it). Team tags are for teams, and team topics are dedicated to organize the very team and its tasks: this topic definitely had a different purpose.

Imho, this post and everything later/below belongs to a new topic, in which you could make a proposal/request/suggestion in/to the Docs team and get in touch with them about if/when/why to do something in this respect (or not, whatever). In that new topic, you could link to here.

Bad docs will end in a PR and there they can be fixed.

PR create work. And if PR are not appropriate or not intended, they cause unnecessary /avoidable work. One issue the Docs team has had throughout its history is a lack of human resources compared to the amount of tasks that are allocated to them and compared to what they wanted to achieve.

Creating new Docs pages (and when and how) is something that should contain some considerations, and it should be aligned/coordinated with the team before publicly encouraging people (especially as this topic currently creates a little the perception that this encouragement comes from the Docs team). Again, please get in touch with the Docs team before doing that. I’m quite sure they would be thankful for any contributor. Even handling and “pre-processing” PR could be valuable. When I was there, we had periods when some PR were unprocessed for months because no one had the time to process them, although I do not know if this is still a critical issue (and why / why not). But actions and encouragements for actions have to be coordinated and aligned. :classic_smiley:

1 Like

That’s not quite right I think. REF

The fourth field, if present, is a comma-delimited list of options.

discard,no-read-workqueue,no-write-workqueue

EDIT:
These options can be stored in the LUKS2 header / metadata instead. TBH I’m not even sure if the crypttab file is necessary if you have only root in it (on Arch it’s empty), I read somewhere that root is always mounted but I can’t find it now.

$ sudo cryptsetup --allow-discards --perf-no_read_workqueue --perf-no_write_workqueue --persistent refresh root

ref: see --persistent option

2 Likes