Dnf cache generation horribly slow

Hi,

I’m building Docker images using the Fedora base image occasionally and am facing massiv issues with it being slow. Also to be honest creating the cache is slow in general with all my devices. It also seems to request all resources sequentially instead of in parallel.

There is more than enough bandwidth available, but the latency is varying (1000/50 koaxial cable connection). Location wise I live close to one of the largest internet exchanges in the world.

Building without already build cache

Executed in  161,70 secs      fish           external
   usr time   86,13 millis  652,00 micros   85,48 millis
   sys time   72,28 millis  163,00 micros   72,11 millis

Building with an already build cache (I introduced a layer for the cache which is already cached):

Executed in   39,12 secs      fish           external
   usr time  132,37 millis    0,00 micros  132,37 millis
   sys time   83,41 millis  989,00 micros   82,42 millis

Is there anything which can be done? Is parallelization already in work?

Kind regards

Although, I think Debian could perform better as well, here and impression:

time docker run --rm -it debian bash -c "time apt-get update && apt-get install -y bash"
Get:1 http://deb.debian.org/debian buster InRelease [121 kB]
Get:2 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:4 http://security.debian.org/debian-security buster/updates/main amd64 Packages [291 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 Packages [7907 kB]
Get:6 http://deb.debian.org/debian buster-updates/main amd64 Packages [10.9 kB]
Fetched 8447 kB in 3s (2484 kB/s)                       
Reading package lists... Done

real	0m3.820s
user	0m1.895s
sys	0m0.560s
Reading package lists... Done
Building dependency tree       
Reading state information... Done
bash is already the newest version (5.0-4).
bash set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 8 not upgraded.

________________________________________________________
Executed in    4,71 secs      fish           external
   usr time   14,76 millis  605,00 micros   14,15 millis
   sys time   21,16 millis  411,00 micros   20,75 millis

time docker run --rm -it fedora bash -c "time dnf install -y bash"
Fedora 33 openh264 (From Cisco) - x86_64                       156  B/s | 2.5 kB     00:16    
Fedora Modular 33 - x86_64                                     206 kB/s | 3.3 MB     00:16    
Fedora Modular 33 - x86_64 - Updates                           201 kB/s | 3.1 MB     00:15    
Fedora 33 - x86_64 - Updates                                   2.4 MB/s |  27 MB     00:11    
Fedora 33 - x86_64                                             3.3 MB/s |  72 MB     00:21    
Package bash-5.0.17-2.fc33.x86_64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!

real	1m42.244s
user	0m22.724s
sys	0m1.620s

________________________________________________________
Executed in  102,65 secs      fish           external
   usr time   38,34 millis    0,00 millis   38,34 millis
   sys time   58,31 millis    1,06 millis   57,24 millis

And running some speedtest:

speedtest

   Speedtest by Ookla

        ISP: Vodafone Germany Cable
    Latency:     7.35 ms   (3.54 ms jitter)
   Download:   931.28 Mbps (data used: 1.1 GB)                               
     Upload:    38.51 Mbps (data used: 35.3 MB)                               
Packet Loss:     0.0%

(yes, Vodafone is a bad operator, but it is also slow when being run within data centers)

Please show us your /etc/dnf/dnf.conf
There are some tunings you can do.

docker run --rm -it fedora cat /etc/dnf/dnf.conf

[main]
gpgcheck=1
installonly_limit=3
clean_requirements_on_remove=True
best=False
skip_if_unavailable=True
tsflags=nodocs

But to what I’ve seen tuning it doesn’t help. It only effects the download itself, but not the metadata fetching.

Mine is like this:

[main]
fastestmirror=True
gpgcheck=1
max_parallel_downloads=10
installonly_limit=2
clean_requirements_on_remove=True
keepcache=True
ip_resolve=4

You also have to take in consideration that you might playing with to different cache’s. On the workstation it self at least. I don’t know how it looks like with docker … do you always use sudo to use dnf?

I had the problem that when i just wanted to get an info of a package that i used dnf without sudo. This creates a cache for my own user … if you do with sudo it creates a other cache for the root privileged user. I always was creating cache while waiting to execute the command it self.

See here:
https://discussion.fedoraproject.org/t/can-make-dnf-cache-system-wide-instead-of-per-user/73981

1 Like

Yep, so to ensure that you only have one cache and update it only once for each transaction you should always run dnf either as root or with sudo. That avoids the extra cache actions and time requirements for the individual user.

I just did a dnf list as my regular user and it took about 2.5 minutes to build the cache. That was with the fastest mirror option in dnf.conf. As root it took ~1 second.

1 Like

Thank you, both. But it doesn’t bring this forward. The Fedora Docker image has no cache (and shouldn’t have one, as it would blow up the image). The issue is that the creation is taking too long and one issue with that is that dnf does not seem to download all data in parallel but one by one. And especially the cisco repo has very high latency.