CentOS 10's dnf is painfully slow

Our Neighbors CentOS
1

This is crazy:

root@cs10:~# time dnf --help
usage: dnf [options] COMMAND

List of Main Commands:

alias                     List or create command aliases
autoremove                remove all unneeded packages that were originally installed as dependencies
check                     check for problems in the packagedb
check-update              check for available package upgrades
clean                     remove cached data
deplist                   [deprecated, use repoquery --deplist] List package's dependencies and what packages provide them
distro-sync               synchronize installed packages to the latest available versions
downgrade                 Downgrade a package
group                     display, or use, the groups information
help                      display a helpful usage message
history                   display, or use, the transaction history
info                      display details about a package or group of packages
install                   install a package or packages on your system
list                      list a package or groups of packages
makecache                 generate the metadata cache
mark                      mark or unmark installed packages as installed by user.
module                    Interact with Modules.
provides                  find what package provides the given value
reinstall                 reinstall a package
remove                    remove a package or packages from your system
repolist                  display the configured software repositories
repoquery                 search for packages matching keyword
repository-packages       run commands on top of all packages in given repository
search                    search package details for the given string
shell                     run an interactive DNF shell
swap                      run an interactive DNF mod for remove and install one spec
updateinfo                display advisories about packages
upgrade                   upgrade a package or packages on your system
upgrade-minimal           upgrade, but only 'newest' package match which fixes a problem that affects your system

List of Plugin Commands:

builddep                  Install build dependencies for package or spec file
changelog                 Show changelog data of packages
config-manager            manage dnf configuration options and repositories
copr                      Interact with Copr repositories.
debuginfo-install         install debuginfo packages
download                  Download package to current directory
groups-manager            create and edit groups metadata file
kpatch                    Toggles automatic installation of kpatch-patch packages
needs-restarting          determine updated binaries that need restarting
offline-distrosync        Prepare offline distrosync of the system
offline-upgrade           Prepare offline upgrade of the system
playground                Interact with Playground repository.
repoclosure               Display a list of unresolved dependencies for repositories
repodiff                  List differences between two sets of repositories
repograph                 Output a full package dependency graph in dot format
repomanage                Manage a directory of rpm packages
reposync                  download all packages from remote repo
system-upgrade            Prepare system for upgrade to a new release

General DNF options:
  -c [config file], --config [config file]
                        config file location
  -q, --quiet           quiet operation
  -v, --verbose         verbose operation
  --version             show DNF version and exit
  --installroot [path]  set install root
  --nodocs              do not install documentations
  --noplugins           disable all plugins
  --enableplugin [plugin]
                        enable plugins by name
  --disableplugin [plugin]
                        disable plugins by name
  --releasever RELEASEVER
                        override the value of $releasever in config and repo files
  --setopt SETOPTS      set arbitrary config and repo options
  --skip-broken         resolve depsolve problems by skipping packages
  -h, --help, --help-cmd
                        show command help
  --allowerasing        allow erasing of installed packages to resolve dependencies
  -b, --best            try the best available package versions in transactions.
  --nobest              do not limit the transaction to the best candidate
  -C, --cacheonly       run entirely from system cache, don't update cache
  -R [minutes], --randomwait [minutes]
                        maximum command wait time
  -d [debug level], --debuglevel [debug level]
                        debugging output level
  --debugsolver         dumps detailed solving results into files
  --showduplicates      show duplicates, in repos, in list/search commands
  -e ERRORLEVEL, --errorlevel ERRORLEVEL
                        error output level
  --obsoletes           enables dnf's obsoletes processing logic for upgrade or display capabilities that the package obsoletes for info, list and repoquery
  --rpmverbosity [debug level name]
                        debugging output level for rpm
  -y, --assumeyes       automatically answer yes for all questions
  --assumeno            automatically answer no for all questions
  --enablerepo [repo]   Temporarily enable repositories for the purpose of the current dnf command. Accepts an id, a comma-separated list of ids, or a glob of ids. This option can be specified multiple times.
  --disablerepo [repo]  Temporarily disable active repositories for the purpose of the current dnf command. Accepts an id, a comma-separated list of ids, or a glob of ids. This option can be specified multiple
                        times, but is mutually exclusive with `--repo`.
  --repo [repo], --repoid [repo]
                        enable just specific repositories by an id or a glob, can be specified multiple times
  --enable              enable repos with config-manager command (automatically saves)
  --disable             disable repos with config-manager command (automatically saves)
  -x [package], --exclude [package], --excludepkgs [package]
                        exclude packages by name or glob
  --disableexcludes [repo], --disableexcludepkgs [repo]
                        disable excludepkgs
  --repofrompath [repo,path]
                        label and path to an additional repository to use (same path as in a baseurl), can be specified multiple times.
  --noautoremove        disable removal of dependencies that are no longer used
  --nogpgcheck          disable gpg signature checking (if RPM policy allows)
  --color COLOR         control whether color is used
  --refresh             set metadata as expired before running the command
  -4                    resolve to IPv4 addresses only
  -6                    resolve to IPv6 addresses only
  --destdir DESTDIR, --downloaddir DESTDIR
                        set directory to copy packages to
  --downloadonly        only download packages
  --comment COMMENT     add a comment to transaction
  --bugfix              Include bugfix relevant packages, in updates
  --enhancement         Include enhancement relevant packages, in updates
  --newpackage          Include newpackage relevant packages, in updates
  --security            Include security relevant packages, in updates
  --advisory ADVISORY, --advisories ADVISORY
                        Include packages needed to fix the given advisory, in updates
  --bz BUGZILLA, --bzs BUGZILLA
                        Include packages needed to fix the given BZ, in updates
  --cve CVES, --cves CVES
                        Include packages needed to fix the given CVE, in updates
  --sec-severity {Critical,Important,Moderate,Low}, --secseverity {Critical,Important,Moderate,Low}
                        Include security relevant packages matching the severity, in updates
  --forcearch ARCH      Force the use of an architecture

real	0m15.143s
user	0m0.104s
sys	0m0.026s

Here’s an strace of it: CS10: strace -tf dnf --help - Pastebin Service

2

In the strace log, you can check line 4252 to see the first big delay.

3

These are the crucial parts:

19:02:31 socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 3
19:02:31 setsockopt(3, SOL_IP, IP_RECVERR, [1], 4) = 0
19:02:31 connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.122.1")}, 16) = 0
19:02:31 poll([{fd=3, events=POLLOUT}], 1, 0) = 1 ([{fd=3, revents=POLLOUT}])
19:02:31 sendmmsg(3, [{msg_hdr={msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\26\177\1\0\0\1\0\0\0\0\0\0\4cs10\0\0\1\0\1", iov_len=22}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, msg_len=22}, {msg_hdr={msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\301~\1\0\0\1\0\0\0\0\0\0\4cs10\0\0\34\0\1", iov_len=22}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, msg_len=22}], 2, MSG_NOSIGNAL) = 2
19:02:31 poll([{fd=3, events=POLLIN}], 1, 5000) = 1 ([{fd=3, revents=POLLIN}])
19:02:31 ioctl(3, FIONREAD, [38])       = 0
19:02:31 recvfrom(3, "\26\177\205\200\0\1\0\1\0\0\0\0\4cs10\0\0\1\0\1\300\f\0\1\0\1\0\0\0\0"..., 2048, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.122.1")}, [28 => 16]) = 38
19:02:31 poll([{fd=3, events=POLLIN}], 1, 4999) = 0 (Timeout)
19:02:36 poll([{fd=3, events=POLLOUT}], 1, 0) = 1 ([{fd=3, revents=POLLOUT}])
19:02:36 sendto(3, "\26\177\1\0\0\1\0\0\0\0\0\0\4cs10\0\0\1\0\1", 22, MSG_NOSIGNAL, NULL, 0) = 22
19:02:36 poll([{fd=3, events=POLLIN}], 1, 5000) = 1 ([{fd=3, revents=POLLIN}])
19:02:36 ioctl(3, FIONREAD, [38])       = 0
19:02:36 recvfrom(3, "\26\177\205\200\0\1\0\1\0\0\0\0\4cs10\0\0\1\0\1\300\f\0\1\0\1\0\0\0\0"..., 2048, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.122.1")}, [28 => 16]) = 38
19:02:36 poll([{fd=3, events=POLLOUT}], 1, 4999) = 1 ([{fd=3, revents=POLLOUT}])
19:02:36 sendto(3, "\301~\1\0\0\1\0\0\0\0\0\0\4cs10\0\0\34\0\1", 22, MSG_NOSIGNAL, NULL, 0) = 22
19:02:36 poll([{fd=3, events=POLLIN}], 1, 4999) = 0 (Timeout)
19:02:41 close(3)                       = 0
19:02:41 socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 3
19:02:41 setsockopt(3, SOL_IP, IP_RECVERR, [1], 4) = 0
19:02:41 connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.122.1")}, 16) = 0
19:02:41 poll([{fd=3, events=POLLOUT}], 1, 0) = 1 ([{fd=3, revents=POLLOUT}])
19:02:41 sendto(3, "\26\177\1\0\0\1\0\0\0\0\0\0\4cs10\0\0\1\0\1", 22, MSG_NOSIGNAL, NULL, 0) = 22
19:02:41 poll([{fd=3, events=POLLIN}], 1, 5000) = 1 ([{fd=3, revents=POLLIN}])
19:02:41 ioctl(3, FIONREAD, [38])       = 0
19:02:41 recvfrom(3, "\26\177\205\200\0\1\0\1\0\0\0\0\4cs10\0\0\1\0\1\300\f\0\1\0\1\0\0\0\0"..., 2048, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.122.1")}, [28 => 16]) = 38
19:02:41 close(3)                       = 0
19:02:41 socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 3
19:02:41 setsockopt(3, SOL_IP, IP_RECVERR, [1], 4) = 0
19:02:41 connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.122.1")}, 16) = 0
19:02:41 poll([{fd=3, events=POLLOUT}], 1, 4999) = 1 ([{fd=3, revents=POLLOUT}])
19:02:41 sendto(3, "\301~\1\0\0\1\0\0\0\0\0\0\4cs10\0\0\34\0\1", 22, MSG_NOSIGNAL, NULL, 0) = 22
19:02:41 poll([{fd=3, events=POLLIN}], 1, 4999) = ? ERESTART_RESTARTBLOCK (Interrupted by signal)
19:02:45 --- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
19:02:45 restart_syscall(<... resuming interrupted poll ...>) = 0
19:02:46 close(3)                       = 0
19:02:46 getuid()                       = 0
19:02:46 geteuid()                      = 0
19:02:46 getgid()                       = 0
19:02:46 getegid()                      = 0
19:02:46 openat(AT_FDCWD, "/usr/lib64/sasl2", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
19:02:46 fstat(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
19:02:46 getdents64(3, 0x562eeb5aaa20 /* 17 entries */, 32768) = 632
19:02:46 openat(AT_FDCWD, "/usr/lib64/sasl2/libanonymous.la", O_RDONLY) = -1 ENOENT (No such file or directory)
19:02:46 openat(AT_FDCWD, "/usr/lib64/sasl2/libanonymous.so", O_RDONLY|O_CLOEXEC) = 4
19:02:46 read(4, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
19:02:46 fstat(4, {st_mode=S_IFREG|0755, st_size=20128, ...}) = 0
19:02:46 mmap(NULL, 20688, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 4, 0) = 0x7f9d46b72000
4

There’s likely a problem with your DNS setup on the host.
Try using exclusively Google DNS or Cloudflare DNS on the guest.

1 Like
5

Fifteen seconds for the help output is definitely not normal. For me it’s about a tenth of a second.

real	0m0.113s
user	0m0.097s
sys	0m0.015s

Can you reproduce that behavior on any other systems?

6

I think @vgaetera has the cause diagnosed.
Suggest you test your dns setup with both resolvectl query and dig commands.

Start with dig using localhost as the server then move on to test each of your DNS resolves.

1 Like
7

@carlwgeorge > Here are some facts that might help to find the root cause of the problem :

I am running CentOS Stream 10 in a KVM virtual machine on a Fedora 41 Workstation host since the first ISO has been made available. The first months everything worked as expected, but a few weeks ago the problem suddenly appeared. I could reproduce it on another customer’s machine.

Not sure if it was caused by a NetworkManager package update. After the official release announcement I performed a fresh installation and realized that the issue still appeared.

What makes me wonder is that it doesn’ t appear on a RHEL 10 Beta VM on the same host.
As a workaround to successfully get rid of the problem, I issued the following commands :

nmcli con modify "cl-VLAN" ipv4.dns "1.1.1.1 8.8.8.8"

systemctl restart NetworkManager.service

Note :

cl-VLAN is the connection name of my adapter inside the VM. Customers who want to apply this solution have to replace cl-VLAN with their connection name. In order to find the name, just run :

nmcli device status

Hope this information is useful for you … and for other community members of course, too.

1 Like