Much slower USB transfer speed after updating from stock .iso install

Ask Fedora
, ,
1

Hi,
I recently had “rawhide” installed because my laptop is fairly new and has a hybrid GPU(i9, rtx4080), so I wanted to be uptodate with latest kernel and stuff.
But after realising that a VM which I was hosting on an external SSD(Samsung T7 1tb , usb 3.2 gen2) was running slow, I decided to investigate a bit.
At first I blamed the w10 guest doing some updates and going crazy but it wasnt the case.
The VM files actually reside inside an encrypted veracrypt container located on the external SSD, so what I did after, was to test the SSD speed thinking it might have been compromised by the wear level.
So, on the same host machine with F40, I’ve formatted the ssd (was and is ext4 btw) and tried to generate this huge 700GB VC container file. The speed was rubbish (40-50MiB/s) and would take 4 hours to complete the task. At that point I wasnt sure if the speed was indeed rubbish but it raised a brow when I saw it estimated 4hrs, because it didnt feel right compared to how long it took first time, which was a good while ago, so i wasnt totally sure. So at like 50%, I decided to abort the file creation and start digging some more into the SSD.
So I did 2 tests with it:
1 - ive hooked it up to another laptop I had around(with lower specs and running linux mint) and tried to generate the VC container there. The speed was 180-200 MiB/s !!
2- on the original laptop, I downloaded f40 workstation .iso, booted live, installed only the veracrypt pkg and did the file generation. Speed ? 250-300 MiB/s !!

So now I knew its not the ssd at fault (I was still skeptical a bit though).

The next thing I did, I backed up my home folder and did a clean install of F40 workstation , because I thought rawhide and the latest kernels might be the issue, so I went for stable stuff.

After the install completed, AND BEFORE doing any “dnf upgrade”, I ve tested the SSD again with VC : 200+ MiB/s . Btw, Kernel is 6.8.5-301 on the .iso

So I did a “dnf upgrade” for the whole thing, kernel and packages, which got me to 6.10.3-200.
After reboot, I tried the file generation again : 40-50 MiB/s !!!
I thought I was going crazy !

I went to grub, fired up 3.8.5 again, same result : 40-50 MiBs !

After going back and forth with multiple times reinstalling F40 and testing just after clean install and after updating some packages , I realised Im only getting the higher transfer speeds with stock .iso … And btw, when testing, I also had other external ssd drives I used for testing, not just the original one!

So my question is now, what the hex is going on, is it the kernel, is it some package, or is it both ?!?

Now the machine is with 6.8.5 after clean install, waiting for your guys input to troubleshoot.

Thanks !!

2

How are you testing the speed of the SSD?

One possible reason the T7 is slower now then the first time is that the SSD firmware will need to erase flash blocks before it can write more data.

When you first wrote to the SSD the firmware did not need to do that.

3

FYI this is the speed claim for the T7 for Samsung.

Performance may vary depending on host configuration. To reach maximum read/write speeds of up to 1,050/1,000 MB/s, respectively, the host device and connection cables must support USB 3.2 Gen 2 and the UASP mode must be enabled.

4

I am not sure I understand correctly what do you mean by “first time” and “now”.
The ssd is external and its empty.

5

On a hardware level, when sectors on the SSD start to age, regular maintenance operations will be done to keep the drive healthy. This will have an effect on write performance and happens independently of whether the SSD is formatted or full. It’s mosty determined by cell wear.

Since your performance seems to change depending on the OS installation, this might not be what is affecting your current testing, but it’s good to keep in mind when you are comparing performance in the future.

6

Agreed, but definitely not the case here.
I get same test results for 2 more SSDs , theyre faster before update, slower after !

7

It’s certainly possible that the kernel, USB and/or block device drivers are the cause of what you are seeing. But it might be difficult to find out where exactly it originates, without bisecting kernel commits and that would be a lot of work…

8

I guess so, but given the different scenarios ive already tested, we can exclude the ssd being at fault which is what the most people experience when their external drives start working way slower.
I am going to test with a openSUSE tumbleweed up-to-date and see what I get.

9

I’ve tested in tumbleweed with latest kernel and updated packages and everything is fine. So definitely a Fedora package bug.

10

I would file a bug report with all the info you have provided so far, including the data with the speed test on the clean install then after the upgrade.
I think it would apply to the kernel so that is what I would use.

It really seems strange but has been known to happen.

1 Like
11

Hi Jeff
I would like to understand why I still keep on getting the slow speed even when choosing to boot the 6.8.x kernel… I mean after installing 6.10.x update , then at grub choosing 6.8.x

12

UPDATE: After clean install, I have updated ONLY dnf and the kernel to 6.10.3 and the drive is still fast.
This is the update log:

Return-Code    : Success
Releasever     : 40
Command Line   : update dnf
Comment        : 
Packages Altered:
    Upgrade  dnf-4.21.0-1.fc40.noarch                     @updates
    Upgraded dnf-4.19.0-1.fc40.noarch                     @@System
    Upgrade  dnf-data-4.21.0-1.fc40.noarch                @updates
    Upgraded dnf-data-4.19.0-1.fc40.noarch                @@System
    Upgrade  dnf-plugins-core-4.8.0-1.fc40.noarch         @updates
    Upgraded dnf-plugins-core-4.5.0-1.fc40.noarch         @@System
    Upgrade  libdnf-0.73.2-1.fc40.x86_64                  @updates
    Upgraded libdnf-0.73.0-1.fc40.x86_64                  @@System
    Upgrade  librepo-1.18.0-1.fc40.x86_64                 @updates
    Upgraded librepo-1.17.0-3.fc40.x86_64                 @@System
    Upgrade  python3-dnf-4.21.0-1.fc40.noarch             @updates
    Upgraded python3-dnf-4.19.0-1.fc40.noarch             @@System
    Upgrade  python3-dnf-plugins-core-4.8.0-1.fc40.noarch @updates
    Upgraded python3-dnf-plugins-core-4.5.0-1.fc40.noarch @@System
    Upgrade  python3-hawkey-0.73.2-1.fc40.x86_64          @updates
    Upgraded python3-hawkey-0.73.0-1.fc40.x86_64          @@System
    Upgrade  python3-libdnf-0.73.2-1.fc40.x86_64          @updates
    Upgraded python3-libdnf-0.73.0-1.fc40.x86_64          @@System
    Upgrade  yum-4.21.0-1.fc40.noarch                     @updates
    Upgraded yum-4.19.0-1.fc40.noarch                     @@System

And then:

Begin time     : Mon 12 Aug 2024 09:47:45 AM EEST
Begin rpmdb    : b9823c6b0237ff375adf54da4bf2bcf9cedcc1c8bf15debd8d91492c9d62fc25
End time       : Mon 12 Aug 2024 09:48:21 AM EEST (36 seconds)
End rpmdb      : c5359bba8c8756257e7eb515750aecf94f31fa754e298eecea79e8514e80bd54
User           : grgr <grgr>
Return-Code    : Success
Releasever     : 40
Command Line   : update kernel
Comment        : 
Packages Altered:
    Install kernel-6.10.3-200.fc40.x86_64              @updates
    Install kernel-core-6.10.3-200.fc40.x86_64         @updates
    Install kernel-modules-6.10.3-200.fc40.x86_64      @updates
    Install kernel-modules-core-6.10.3-200.fc40.x86_64 @updates

Plus

Begin time     : Mon 12 Aug 2024 09:48:30 AM EEST
Begin rpmdb    : c5359bba8c8756257e7eb515750aecf94f31fa754e298eecea79e8514e80bd54
End time       : Mon 12 Aug 2024 09:48:35 AM EEST (5 seconds)
End rpmdb      : 93d576b6472d26b3ea284dd3e85a3c247dd1bb311df2a1515644ce857fd08239
User           : grgr <grgr>
Return-Code    : Success
Releasever     : 40
Command Line   : update kernel*
Comment        : 
Packages Altered:
    Install kernel-modules-extra-6.10.3-200.fc40.x86_64 @updates

I will try to install some more updates in order to understand WHEN things start going crazy

1 Like
13

Okay, after installing more packages and “sudo reboot” after each, I finally got to MAYBE the one(s) at fault:

$ sudo dnf  history
ID     | Command line              | Date and time    | Action(s)      | Altered
--------------------------------------------------------------------------------
    28 | history rollback 26       | 2024-08-13 14:30 | Downgrade      |    7   
    27 | update nemo*              | 2024-08-13 14:09 | Upgrade        |    7   
    26 | update gvfs-*             | 2024-08-13 14:06 | Upgrade        |    9   
    25 | update intel-*            | 2024-08-13 14:03 | I, O, U        |    6   
    24 | update nvidia-gpu-firmwar | 2024-08-13 13:55 | Upgrade        |    1   
    23 | update zenity             | 2024-08-13 11:47 | Upgrade        |    1   
    22 | update zlib-ng-*          | 2024-08-13 11:46 | Upgrade        |    1   
    21 | update zram-*             | 2024-08-13 11:45 | Upgrade        |    2   
    20 | upgrade PackageKit*       | 2024-08-12 11:48 | Upgrade        |    3   
    19 | update NetworkManager*    | 2024-08-12 11:40 | Upgrade        |   14   
    18 | update ImageMagick        | 2024-08-12 11:40 | Upgrade        |    2   
    17 | remove transmission       | 2024-08-12 11:39 | Removed        |    4   
    16 | update default-fonts-*    | 2024-08-12 11:38 | Upgrade        |   67   
    15 | update python3            | 2024-08-12 10:43 | Upgrade        |    3   
    14 | update linux-firmware*    | 2024-08-12 10:39 | Upgrade        |    2   
    13 | update firefox            | 2024-08-12 10:36 | Upgrade        |    8   
    12 | update gtk4               | 2024-08-12 10:33 | Upgrade        |    1   
    11 | update gtk3               | 2024-08-12 10:33 | Upgrade        |    2   
    10 | update cinnamon-*         | 2024-08-12 10:31 | Upgrade        |   10   
     9 | update glibc              | 2024-08-12 10:27 | Upgrade        |    4   
     8 | upgrade lightdm           | 2024-08-12 10:23 | Upgrade        |    3   
     7 | upgrade cinnamon          | 2024-08-12 10:23 | I, U           |    5   
     6 | update systemd            | 2024-08-12 10:10 | Upgrade        |   11   
     5 |                           | 2024-08-12 09:57 | Install        |    1   
     4 | update kernel*            | 2024-08-12 09:48 | Install        |    1   
     3 | update kernel             | 2024-08-12 09:47 | Install        |    4   
     2 | update dnf                | 2024-08-12 09:47 | Upgrade        |   10  <
     1 |                           | 2024-04-15 01:56 | Install        | 1928 >E

Apparently something to do with “nemo” :

sudo dnf history info 27
Transaction ID : 27
Begin time     : Tue 13 Aug 2024 02:09:11 PM EEST
Begin rpmdb    : bd47dc4f1bcbd8e3e047a6880e3f078ade51490a9649e80525430c6c5be80f96
End time       : Tue 13 Aug 2024 02:09:15 PM EEST (4 seconds)
End rpmdb      : a5bd489abc697924d88d55721747d16affb57341e13003e3d3fab6deeef0ac2c
User           : grgr <grgr>
Return-Code    : Success
Releasever     : 40
Command Line   : update nemo*
Comment        : 
Packages Altered:
    Upgrade  nemo-6.2.5-1.fc40.x86_64                 @updates
    Upgraded nemo-6.0.2-4.fc40.x86_64                 @@System
    Upgrade  nemo-extensions-6.2.5-1.fc40.x86_64      @updates
    Upgraded nemo-extensions-6.0.2-4.fc40.x86_64      @@System
    Upgrade  nemo-fileroller-6.2.0-1.fc40.x86_64      @updates
    Upgraded nemo-fileroller-6.0.1-5.fc40.x86_64      @@System
    Upgrade  nemo-image-converter-6.2.0-1.fc40.x86_64 @updates
    Upgraded nemo-image-converter-6.0.1-5.fc40.x86_64 @@System
    Upgrade  nemo-preview-6.2.0-1.fc40.x86_64         @updates
    Upgraded nemo-preview-6.0.1-5.fc40.x86_64         @@System
    Upgrade  nemo-python-6.2.0-1.fc40.x86_64          @updates
    Upgraded nemo-python-6.0.1-5.fc40.x86_64          @@System
    Upgrade  nemo-search-helpers-6.2.5-1.fc40.x86_64  @updates
    Upgraded nemo-search-helpers-6.0.2-4.fc40.x86_64  @@System

I have rolledback and the USB speed went back to normal, so what I am gonna do next is to try to install each of those nemo packages one by one rather than all at once.

14

I file manager. I wonder if the speed only drops if it is running?
If so use lsof and see if its accessing usb disk to scan it or something.

15

So i’m testing by creating a new veracrypt file container on the external SSD, and im using a bash script to do that. Before nemo update, the speed was ok, after nemo update, the speed dropped 5x. I’ve rolled back the nemo update, now speed is good again.

The external volume is mounted at /run/media/grgr/soho/ .

This is the lsof AFTER ROLLING BACK nemo , while veracrypt was working on creating that huge 50GB file:

lsof /run/media/grgr/soho/
COMMAND    PID USER   FD   TYPE DEVICE   SIZE/OFF NODE NAME
veracrypt 3545 grgr    4u   REG   8,17 2289696768   15 /run/media/grgr/soho/XOXO.vc

I will post next whats up AFTER updating nemo

16

This is after the nemo update:

$ lsof -r 1 /run/media/grgr/soho/

=======
=======
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
veracrypt 3726 grgr    4u   REG   8,17        0   15 /run/media/grgr/soho/XOXO.vc
=======
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
veracrypt 3726 grgr    4u   REG   8,17 22937600   15 /run/media/grgr/soho/XOXO.vc
=======
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
veracrypt 3726 grgr    4u   REG   8,17 59375616   15 /run/media/grgr/soho/XOXO.vc
=======
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
veracrypt 3726 grgr    4u   REG   8,17 96862208   15 /run/media/grgr/soho/XOXO.vc
=======
COMMAND    PID USER   FD   TYPE DEVICE  SIZE/OFF NODE NAME
veracrypt 3726 grgr    4u   REG   8,17 133562368   15 /run/media/grgr/soho/XOXO.vc
=======
COMMAND    PID USER   FD   TYPE DEVICE  SIZE/OFF NODE NAME
veracrypt 3726 grgr    4u   REG   8,17 169738240   15 /run/media/grgr/soho/XOXO.vc
=======
COMMAND    PID USER   FD   TYPE DEVICE  SIZE/OFF NODE NAME
veracrypt 3726 grgr    4u   REG   8,17 205389824   15 /run/media/grgr/soho/XOXO.vc
=======
COMMAND    PID USER   FD   TYPE DEVICE  SIZE/OFF NODE NAME
veracrypt 3726 grgr    4u   REG   8,17 236584960   15 /run/media/grgr/soho/XOXO.vc
=======
COMMAND    PID USER   FD   TYPE DEVICE  SIZE/OFF NODE NAME
veracrypt 3726 grgr    4u   REG   8,17 254410752   15 /run/media/grgr/soho/XOXO.vc
=======
=======
=======
^C

Speed is slow now.

17

I wonder if anyone runs F40 Cinnammon with nemo FM, up-to-date to reproduce this.

18

I’m going to ask a Random question here :

What are you using to test these speeds out ? Is it a larger file, a collection of smaller files? What tool is used for the transfer? ssh, cp, mv, rsync other?

19

As a tool i’m using veracrypt from console. Im asking it to create a new encrypted file container of 50 gigs on the external SSD. I am getting same results no matter if i use the gui or console.

20

Just out of curiosity, Are you in the mounted SSD when you launch the the application ?