Entire session slow for offline update boots

Ask Fedora
, , , ,
1

When I reboot my computer for offline updates, whether via DNF or Gnome Software, everything runs extremely slowly for the duration of the special boot for doing the actual upgrades. The boot process takes a few minutes instead of a few seconds, and the actual updates take maybe half an hour instead of thirty seconds (rough guess). When I press escape to watch the boot process messages, everything looks uniformly slow; no one process appears at fault. Logs from these sessions look normal as best I can tell. It’s as though the chip were simply throttled to a fraction of its normal clock speed (but I’m not sure the chip actually has a frequency mode that slow).

This does not happen with 100% reliability; maybe 80% of offline updates are affected. And once it actually happened for a normal boot. Any ideas what the problem might be, or how I might investigate further? Or some subtle hints to look for in my logs?

Output of fpaste --sysinfo --printonly 
=== fpaste 0.4.2.0 System Information ===
* OS Release (cat /etc/*-release | uniq):
     Fedora release 35 (Thirty Five)
     NAME="Fedora Linux"
     VERSION="35 (Workstation Edition)"
     ID=fedora
     VERSION_ID=35
     VERSION_CODENAME=""
     PLATFORM_ID="platform:f35"
     PRETTY_NAME="Fedora Linux 35 (Workstation Edition)"
     ANSI_COLOR="0;38;2;60;110;180"
     LOGO=fedora-logo-icon
     CPE_NAME="cpe:/o:fedoraproject:fedora:35"
     HOME_URL="https://fedoraproject.org/"
     DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f35/system-administrators-guide/"
     SUPPORT_URL="https://discussion.fedoraproject.org/"
     BUG_REPORT_URL="https://bugzilla.redhat.com/"
     REDHAT_BUGZILLA_PRODUCT="Fedora"
     REDHAT_BUGZILLA_PRODUCT_VERSION=35
     REDHAT_SUPPORT_PRODUCT="Fedora"
     REDHAT_SUPPORT_PRODUCT_VERSION=35
     PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
     VARIANT="Workstation Edition"
     VARIANT_ID=workstation
     Fedora release 35 (Thirty Five)
     
* Kernel (uname -r ; cat /proc/cmdline):
     5.16.16-200.fc35.x86_64
     BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.16.16-200.fc35.x86_64 root=UUID=4c685f19-0f82-4718-b5be-5b3cf9f444f2 ro rootflags=subvol=root rd.luks.uuid=luks-bbf48173-4bae-4757-a5ec-f0cc42c8d369 rhgb quiet rd.driver.blacklist=nouveau modprobe.blacklist=nouveau nvidia-drm.modeset=1 i915.enable_psr=0 mem_sleep_default=deep
     
* Desktop(s) Running (ps -eo comm= | grep -E '(gnome-session|startkde|startactive|xfce.?-session|fluxbox|blackbox|hackedbox|ratpoison|enlightenment|icewm-session|od-session|wmaker|wmx|openbox-lxde|openbox-gnome-session|openbox-kde-session|mwm|e16|fvwm|xmonad|sugar-session|mate-session|lxqt-session|cinnamon|lxdm-session)' ):
     gnome-session-b
     gnome-session-c
     gnome-session-b
     
* Desktop(s) Installed (ls -m /usr/share/xsessions/ | sed 's/\.desktop//g' ):
     gnome-classic, gnome, gnome-xorg
     
* SELinux Status (sestatus):
     SELinux status:                 enabled
     SELinuxfs mount:                /sys/fs/selinux
     SELinux root directory:         /etc/selinux
     Loaded policy name:             targeted
     Current mode:                   enforcing
     Mode from config file:          enforcing
     Policy MLS status:              enabled
     Policy deny_unknown status:     allowed
     Memory protection checking:     actual (secure)
     Max kernel policy version:      33
     
* SELinux Errors (selinuxenabled && journalctl --since yesterday |grep avc: | grep -Eo comm="[^ ]+" | sort |uniq -c |sort -rn):
          18 comm="fprintd"
           1 comm="systemd-user-ru"
     
* CPU Model (grep 'model name' /proc/cpuinfo | awk -F: '{print $2}' | uniq -c |
     sed -re 's/^ +//' ):
     16  11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz
     
* 64-bit Support (grep -q ' lm ' /proc/cpuinfo && echo Yes || echo No):
     Yes
     
* Hardware Virtualization Support (grep -Eq '(vmx|svm)' /proc/cpuinfo && echo Yes || echo No):
     Yes
     
* Load average (uptime):
      11:10:45 up 47 min,  1 user,  load average: 0.18, 0.23, 0.21
     
* Memory usage (free -m):
                    total        used        free      shared  buff/cache   available
     Mem:           15731        3847        7458         863        4425       10710
     Swap:           8191           0        8191
     
* Top 5 CPU hogs (ps axuScnh | awk '$2!=12819' | sort -rnk3 | head -5):
         1000    8098  6.8  4.0 13210300 651704 ?     Rl   10:30   2:44 firefox-bin
         1000   10890  1.7  1.5 2640064 247376 ?      Sl   10:45   0:27 Isolated Web Co
         1000    9873  1.3  1.6 2747408 265824 ?      Sl   10:38   0:27 Isolated Web Co
         1000    6234  1.2  0.8 27653568 137636 tty2  Sl+  10:28   0:30 Xorg
         1000   10914  1.2  2.2 3307772 362104 ?      Sl   10:45   0:18 Isolated Web Co
     
* Top 5 Memory hogs (ps axuScnh | sort -rnk4 | head -5):
         1000    8098  6.8  4.0 13210300 651616 ?     Sl   10:30   2:44 firefox-bin
         1000   10914  1.2  2.2 3307772 362104 ?      Sl   10:45   0:18 Isolated Web Co
            0    5721  0.3  2.2 968344 355404 ?       Ssl  10:28   0:08 packagekitd
         1000    8345  0.2  1.8 2726448 303920 ?      Sl   10:30   0:07 WebExtensions
         1000    6994  0.1  1.8 1412440 301636 ?      Sl   10:28   0:03 gnome-software
     
* Disk space usage (df -hT):
     Filesystem     Type      Size  Used Avail Use% Mounted on
     devtmpfs       devtmpfs  4.0M     0  4.0M   0% /dev
     tmpfs          tmpfs     7.7G     0  7.7G   0% /dev/shm
     tmpfs          tmpfs     3.1G  2.3M  3.1G   1% /run
     /dev/dm-0      btrfs     953G  697G  256G  74% /
     tmpfs          tmpfs     7.7G   48K  7.7G   1% /tmp
     /dev/dm-0      btrfs     953G  697G  256G  74% /home
     /dev/loop0     squashfs   56M   56M     0 100% /var/lib/snapd/snap/core18/2284
     /dev/loop2     squashfs   44M   44M     0 100% /var/lib/snapd/snap/snapd/14978
     /dev/loop1     squashfs  167M  167M     0 100% /var/lib/snapd/snap/freeplane-mindmapping/31
     /dev/nvme0n1p2 ext4      974M  256M  652M  29% /boot
     /dev/nvme0n1p1 vfat      599M   14M  585M   3% /boot/efi
     tmpfs          tmpfs     1.6G  172K  1.6G   1% /run/user/1000
     
* Block devices (without results: "blkid" AND "/sbin/blkid"):
     N/A

* PCI devices (lspci -nn):
     0000:00:00.0 Host bridge [0600]: Intel Corporation 11th Gen Core Processor Host Bridge/DRAM Registers [8086:9a36] (rev 05)
     0000:00:01.0 PCI bridge [0604]: Intel Corporation 11th Gen Core Processor PCIe Controller #1 [8086:9a01] (rev 05)
     0000:00:02.0 VGA compatible controller [0300]: Intel Corporation TigerLake-H GT1 [UHD Graphics] [8086:9a60] (rev 01)
     0000:00:04.0 Signal processing controller [1180]: Intel Corporation TigerLake-LP Dynamic Tuning Processor Participant [8086:9a03] (rev 05)
     0000:00:06.0 System peripheral [0880]: Intel Corporation Device [8086:09ab]
     0000:00:07.0 PCI bridge [0604]: Intel Corporation Tiger Lake-H Thunderbolt 4 PCI Express Root Port #0 [8086:9a2b] (rev 05)
     0000:00:0a.0 Signal processing controller [1180]: Intel Corporation Tigerlake Telemetry Aggregator Driver [8086:9a0d] (rev 01)
     0000:00:0d.0 USB controller [0c03]: Intel Corporation Tiger Lake-H Thunderbolt 4 USB Controller [8086:9a17] (rev 05)
     0000:00:0d.2 USB controller [0c03]: Intel Corporation Tiger Lake-H Thunderbolt 4 NHI #0 [8086:9a1f] (rev 05)
     0000:00:0e.0 RAID bus controller [0104]: Intel Corporation Volume Management Device NVMe RAID Controller [8086:9a0b]
     0000:00:12.0 Serial controller [0700]: Intel Corporation Device [8086:43fc] (rev 11)
     0000:00:14.0 USB controller [0c03]: Intel Corporation Tiger Lake-H USB 3.2 Gen 2x1 xHCI Host Controller [8086:43ed] (rev 11)
     0000:00:14.2 RAM memory [0500]: Intel Corporation Tiger Lake-H Shared SRAM [8086:43ef] (rev 11)
     0000:00:14.3 Network controller [0280]: Intel Corporation Tiger Lake PCH CNVi WiFi [8086:43f0] (rev 11)
     0000:00:15.0 Serial bus controller [0c80]: Intel Corporation Tiger Lake-H Serial IO I2C Controller #0 [8086:43e8] (rev 11)
     0000:00:15.1 Serial bus controller [0c80]: Intel Corporation Device [8086:43e9] (rev 11)
     0000:00:16.0 Communication controller [0780]: Intel Corporation Tiger Lake-H Management Engine Interface [8086:43e0] (rev 11)
     0000:00:1f.0 ISA bridge [0601]: Intel Corporation Tiger Lake-H LPC/eSPI Controller [8086:438b] (rev 11)
     0000:00:1f.3 Multimedia audio controller [0401]: Intel Corporation Tiger Lake-H HD Audio Controller [8086:43c8] (rev 11)
     0000:00:1f.4 SMBus [0c05]: Intel Corporation Tiger Lake-H SMBus Controller [8086:43a3] (rev 11)
     0000:00:1f.5 Serial bus controller [0c80]: Intel Corporation Tiger Lake-H SPI Controller [8086:43a4] (rev 11)
     0000:01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA106M [GeForce RTX 3060 Mobile / Max-Q] [10de:2520] (rev a1)
     0000:01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:228e] (rev a1)
     10000:e0:06.0 PCI bridge [0604]: Intel Corporation 11th Gen Core Processor PCIe Controller #0 [8086:9a0f] (rev 05)
     10000:e1:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO [144d:a80a]
     
* USB devices (lsusb):
     Bus 004 Device 002: ID 0424:5734 Microchip Technology, Inc. (formerly SMSC) USB5734
     Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
     Bus 003 Device 004: ID 27c6:639c Shenzhen Goodix Technology Co.,Ltd. Goodix USB2.0 MISC
     Bus 003 Device 003: ID 0c45:6725 Microdia Integrated_Webcam_HD
     Bus 003 Device 008: ID 0424:274c Microchip Technology, Inc. (formerly SMSC) Hub Controller
     Bus 003 Device 007: ID 413c:2113 Dell Computer Corp. KB216 Wired Keyboard
     Bus 003 Device 006: ID 0461:4e22 Primax Electronics, Ltd Dell Mouse, 2 Buttons, Modell: MS111-P
     Bus 003 Device 002: ID 0424:2734 Microchip Technology, Inc. (formerly SMSC) USB2734
     Bus 003 Device 005: ID 8087:0026 Intel Corp. AX201 Bluetooth
     Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
     Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
     Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
     
* DRM Information (journalctl -k -b | grep -o 'kernel:.*drm.*$' | cut -d ' ' -f 2- ):
     Command line: BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.16.16-200.fc35.x86_64 root=UUID=4c685f19-0f82-4718-b5be-5b3cf9f444f2 ro rootflags=subvol=root rd.luks.uuid=luks-bbf48173-4bae-4757-a5ec-f0cc42c8d369 rhgb quiet rd.driver.blacklist=nouveau modprobe.blacklist=nouveau nvidia-drm.modeset=1 i915.enable_psr=0 mem_sleep_default=deep
     Kernel command line: BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.16.16-200.fc35.x86_64 root=UUID=4c685f19-0f82-4718-b5be-5b3cf9f444f2 ro rootflags=subvol=root rd.luks.uuid=luks-bbf48173-4bae-4757-a5ec-f0cc42c8d369 rhgb quiet rd.driver.blacklist=nouveau modprobe.blacklist=nouveau nvidia-drm.modeset=1 i915.enable_psr=0 mem_sleep_default=deep
     ACPI: bus type drm_connector registered
     i915 0000:00:02.0: [drm] Finished loading DMC firmware i915/tgl_dmc_ver2_12.bin (v2.12)
     i915 0000:00:02.0: [drm] Protected Xe Path (PXP) protected content support initialized
     [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.0 on minor 0
     fbcon: i915drmfb (fb0) is primary device
     i915 0000:00:02.0: [drm] fb0: i915drmfb frame buffer device
     [drm] [nvidia-drm] [GPU ID 0x00000100] Loading driver
     [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:01:00.0 on minor 1
     i915 0000:00:02.0: [drm] Reducing the compressed framebuffer size. This may lead to less power savings than a non-reduced-size. Try to increase stolen memory size if available in BIOS.
     
* Xorg modules (grep LoadModule /var/log/Xorg.0.log ~/.local/share/xorg/Xorg.0.log | cut -d \" -f 2 | xargs):
     
     
* GL Support (glxinfo | grep -E "OpenGL version|OpenGL renderer"):
     OpenGL renderer string: Mesa Intel(R) UHD Graphics (TGL GT1)
     OpenGL version string: 4.6 (Compatibility Profile) Mesa 21.3.7
     
* Xorg errors (without results: "grep '^\[.*(EE)' /var/log/Xorg.0.log ~/.local/share/xorg/Xorg.0.log | cut -d ':' -f 2- "):
     N/A

* Kernel buffer tail (dmesg | tail):
     [  312.410230] IPv6: ADDRCONF(NETDEV_CHANGE): wlp0s20f3: link becomes ready
     [  313.475728] i915 0000:00:02.0: [drm] Reducing the compressed framebuffer size. This may lead to less power savings than a non-reduced-size. Try to increase stolen memory size if available in BIOS.
     [  314.288080] rfkill: input handler disabled
     [  314.561388] Bluetooth: RFCOMM TTY layer initialized
     [  314.561393] Bluetooth: RFCOMM socket layer initialized
     [  314.561422] Bluetooth: RFCOMM ver 1.11
     [  315.185288] usb 3-9: reset full-speed USB device number 4 using xhci_hcd
     [  323.704092] rfkill: input handler enabled
     [  325.636536] rfkill: input handler disabled
     [  335.102683] ACPI: button: The lid device is not compliant to SW_LID.
     
* Last few reboots (last -x -n10 reboot runlevel):
     runlevel (to lvl 5)   5.16.16-200.fc35 Wed Mar 23 10:28   still running
     reboot   system boot  5.16.16-200.fc35 Wed Mar 23 10:28   still running
     runlevel (to lvl 5)   5.16.16-200.fc35 Wed Mar 23 01:24 - 01:24  (00:00)
     reboot   system boot  5.16.16-200.fc35 Wed Mar 23 01:23 - 01:24  (00:01)
     reboot   system boot  5.16.14-200.fc35 Wed Mar 23 00:46 - 01:21  (00:34)
     runlevel (to lvl 5)   5.16.14-200.fc35 Wed Mar 16 11:34 - 00:43 (6+13:08)
     reboot   system boot  5.16.14-200.fc35 Wed Mar 16 11:33 - 00:43 (6+13:09)
     reboot   system boot  5.16.13-200.fc35 Wed Mar 16 00:32 - 00:58  (00:25)
     runlevel (to lvl 5)   5.16.13-200.fc35 Tue Mar 15 05:19 - 00:30  (19:11)
     reboot   system boot  5.16.13-200.fc35 Tue Mar 15 05:19 - 00:30  (19:11)
     
     wtmp begins Sat Feb 19 18:21:28 2022
     
* DNF Repositories (dnf repolist):
     repo id                         repo name
     fedora                          Fedora 35 - x86_64
     fedora-cisco-openh264           Fedora 35 openh264 (From Cisco) - x86_64
     fedora-modular                  Fedora Modular 35 - x86_64
     google-chrome                   google-chrome
     phracek-PyCharm                 Copr repo for PyCharm owned by phracek
     rpmfusion-free                  RPM Fusion for Fedora 35 - Free
     rpmfusion-free-updates          RPM Fusion for Fedora 35 - Free - Updates
     rpmfusion-nonfree               RPM Fusion for Fedora 35 - Nonfree
     rpmfusion-nonfree-nvidia-driver RPM Fusion for Fedora 35 - Nonfree - NVIDIA Driver
     rpmfusion-nonfree-steam         RPM Fusion for Fedora 35 - Nonfree - Steam
     rpmfusion-nonfree-updates       RPM Fusion for Fedora 35 - Nonfree - Updates
     updates                         Fedora 35 - x86_64 - Updates
     updates-modular                 Fedora Modular 35 - x86_64 - Updates
     
* DNF Extras (dnf -C list extras):
     Last metadata expiration check: 0:18:33 ago on Wed 23 Mar 2022 10:52:13 AM EDT.
     Extra Packages
     kmod-nvidia-5.16.13-200.fc35.x86_64.x86_64   3:510.47.03-2.fc35    @@commandline
     kmod-nvidia-5.16.14-200.fc35.x86_64.x86_64   3:510.47.03-2.fc35    @@commandline
     kmod-nvidia-5.16.16-200.fc35.x86_64.x86_64   3:510.47.03-2.fc35    @@commandline
     zoom.x86_64                                  5.9.3.1911-1          @@commandline
     
* Last 20 packages installed (rpm -qa --nodigest --nosignature --last | head -20):
     kmod-nvidia-5.16.16-200.fc35.x86_64-510.47.03-2.fc35.x86_64 Wed 23 Mar 2022 01:24:32 AM EDT
     adobe-mappings-pdf-20190401-1.fc35.noarch     Wed 23 Mar 2022 01:10:22 AM EDT
     alsa-sof-firmware-2.0-2.fc35.noarch           Wed 23 Mar 2022 01:10:21 AM EDT
     distribution-gpg-keys-1.67-1.fc35.noarch      Wed 23 Mar 2022 01:10:19 AM EDT
     gnome-user-docs-41.5-1.fc35.noarch            Wed 23 Mar 2022 01:10:16 AM EDT
     xorg-x11-drv-amdgpu-22.0.0-1.fc35.x86_64      Wed 23 Mar 2022 01:08:36 AM EDT
     python3-pyparted-3.12.0-1.fc35.x86_64         Wed 23 Mar 2022 01:08:36 AM EDT
     go-srpm-macros-3.0.15-1.fc35.noarch           Wed 23 Mar 2022 01:08:36 AM EDT
     python3-peewee-3.14.10-1.fc35.x86_64          Wed 23 Mar 2022 01:08:35 AM EDT
     python3-kiwisolver-1.4.0-1.fc35.x86_64        Wed 23 Mar 2022 01:08:33 AM EDT
     openexr-libs-3.1.4-1.fc35.x86_64              Wed 23 Mar 2022 01:08:33 AM EDT
     mtools-4.0.38-1.fc35.x86_64                   Wed 23 Mar 2022 01:08:32 AM EDT
     duplicity-0.8.22-1.fc35.x86_64                Wed 23 Mar 2022 01:08:32 AM EDT
     containernetworking-plugins-1.1.0-1.fc35.x86_64 Wed 23 Mar 2022 01:08:30 AM EDT
     glib2-devel-2.70.5-1.fc35.x86_64              Wed 23 Mar 2022 01:08:26 AM EDT
     gjs-1.70.2-1.fc35.x86_64                      Wed 23 Mar 2022 01:08:24 AM EDT
     openvpn-2.5.6-1.fc35.x86_64                   Wed 23 Mar 2022 01:08:23 AM EDT
     openssl-1.1.1n-1.fc35.x86_64                  Wed 23 Mar 2022 01:08:22 AM EDT
     libgphoto2-2.5.29-1.fc35.x86_64               Wed 23 Mar 2022 01:08:21 AM EDT
     cups-ipptool-2.3.3op2-16.fc35.x86_64          Wed 23 Mar 2022 01:08:21 AM EDT
     
* EFI boot manager output (efibootmgr -v):
     BootCurrent: 0003
     Timeout: 0 seconds
     BootOrder: 0003,0000
     Boot0000* UEFI RST PM9A1 NVMe Samsung 1024GB S65VNX0RB48832 	HD(1,GPT,fa408d13-cf01-41ae-bfa4-1e76998e9949,0x800,0x12c000)/File(\EFI\Boot\BootX64.efi)N.....YM....R,Y.
     Boot0003* Fedora	HD(1,GPT,fa408d13-cf01-41ae-bfa4-1e76998e9949,0x800,0x12c000)/File(\EFI\fedora\shimx64.efi)
2

Did you forget to press accidentally press the Turbo button? :wink:

Kidding, I have no idea, I’m not familiar with booting systems that use NVME devices, or BTRFS volumes.

You might try looking through the journal for the slow boots, see if anything stands out. From any current session, you can view the journal for the previous boot with sudo journalctl -b -1, for the one before that with sudo journalctl -b -2, and so on, and so on…

(i see what you mean about the slow boots, though. Your last two offline updates apparently took 34 and 25 minutes; I’d expect you to be able to install Fedora in that amount of time!)

     reboot   system boot  5.16.14-200.fc35 Wed Mar 23 00:46 - 01:21  (00:34)
     reboot   system boot  5.16.13-200.fc35 Wed Mar 16 00:32 - 00:58  (00:25)

(OT: Don’t call it a comeback. And if you’re running a 12th-generation Intel CPU — you are not — then be careful with your Scroll Lock key, the Turbo button’s ugly 21st-Century stepchild.)

3

(BTW, why the #amdgpu tag? Based on your system hardware profile, glxinfo output, and the X server logs, AFAICT your laptop(?) has an Intel CPU with integrated graphics, and an intimidatingly monstrous Nvidia GeForce RTX 3060 discrete chip. No AMD hardware in evidence. I also see xorg-x11-drv-amdgpu in your package logs, but it doesn’t appear to be necessary or actually used for anything.)

4

No idea about the AMD tag! Actually I didn’t mean to add the intel or nvidia tags either; maybe the BBS parses any attached sysinfo and adds tags automatically? Anyway, I have been through the logs, but nothing stands out to me. They look normal, like everything is happening that should, just super slow. I’m not sure what I should be looking for, though. I’d be happy to post a log of one of these slow boots, but it’s 5000 lines.

EDIT: By the way, it really is as though I’d pressed a turbo button!

5

I removed the amdgpu tag for you.

One thing to remember with the offline updates it that there is a requirement to download and install all the updates. I have been told different things about what happens when, but it is my impression that it all happens during the limited boot. If that is the case then you are waiting for the system to do the download, install the software, and reboot to normal mode during those extended time boots.

I don’t like the offline updates so I always do a dnf update while my system is running and reboot after the update if anything that needs a reboot (graphics, kernel, drivers, etc.) is updated. Doing it that way saves me time during boots and allows me to see what is being updated before I commit to the actual update.

2 Likes
6

Thanks. This slowness is not just the update process. The boot process beforehand is slow, too. Even entering my LUKS password is noticeably laggy. Though it is surprising what you say about downloading during the limited boot; “dnf offline-upgrade download” certainly does seem to download things, but I guess it might be downloading them a second time during the special boot.

7

I figured out how to make journalctl show microseconds, and it turns out that the slowness is apparent from the very beginning of logging. The first line in journalctl is “Linux version…” and the second line is “Command line: …” The interval between these is 12 microseconds for a sample normal boot, and 179 microseconds for a sample offline updates boot.

8

dnf offline-upgrade download is different than the offline upgrade automatically done during normal boots. The normal boot upgrades are managed by packagekit + gnome-software while the dnf upgrades are managed by dnf. Two different cache & database paths for metadata and packages.

It seems possible that if you are doing both a dnf offline-upgrade and a packagekit upgrade during the same boot there could be conflicts as the different processes are trying to do essentially the same thing, duplicating effort and packages, etc., as well as managing the same rpm database. You probably should not be using both methods to do your updates.

Having other slowdowns during a boot that does not involve an update is a different issue.

9

Sniff sniff, this smells like a hardware issue. The behavior is similar to that of having a zillion hardware interrupts due to either a hardware issue or driver issue. If you have added hardware to your system recently, I’d recommend starting with temporarily removing it and checking for boot time differences. Multiple attempts required since you said this is frequent but not guaranteed.
If your BIOS supports running diagnostics, that’s something else to try.

But I’ll admit, I’ve nothing concrete to go on here, just past experience. Been there, done that.

1 Like
10

To Dave: Thanks. Maybe hardware interrupts are one cause, but software seems to be part of it, because it (almost) only happens when booting into the offline update environment. Any idea what could distinguish an offline-updates boot from the very beginning? Unfortunately, it happens even when all peripherals are unplugged. My Nvidia card might have something to do with it, but the card is always present. EDIT: But it may happen less often when the external monitor and keyboard/mouse are unplugged. I’ll keep trying with and without, though of course I can only really test this when updates happen to be available.

To Jeff: Thanks. I only do packagekit or dnf offline-upgrade, never both. I checked the logs; they don’t seem to be trying to run together. And the problem begins before the actual updating starts.

11

I’m with you. I’m not buying that the offline update process is itself causing this. (And if there really is any offline-update process that delays the package download until after the reboot, then please file a bug report because that is STUPID.)

There have been issues in the past with certain combinations of motherboard and SSD — for sure check if your m/b manufacturer has any BIOS updates, and check for possible firmware updates for your NVMe storage, @amcbride . Those would be my first two candidates for culprit here.

(Edit: Also, isn’t one of the justifications for the update-integrity theater of offline updates, on BTRFS volumes specifically, that some sort of snapshotting or versioning is done during the update process, to make it rollback-able? I wonder if something could be going wrong with that part of the process? …Or is that not enabled by default?)

12

Thanks! I’ll watch for BIOS updates, though there are none at the moment. As for SSD firmware updates, those are just built into the kernel, right?

13

Not the kernel, no… they might be offered by GNOME Software, if they’re submitted to the LVFS. Samsung has a vendor account there, so they could be submitting firmware, but it’s not clear to me that they are.

The NVMe 980 PRO has definitely had firmware updates released, though… the latest revision is currently firmware 5B2QGXA7.

You can check what firmware any supported devices are running with the fwupdmgr command:

fwupdmgr get-devices

On my desktop the output looks like this:

$ fwupdmgr get-devices
WARNING: UEFI capsule updates not available or enabled in firmware setup
  See https://github.com/fwupd/fwupd/wiki/PluginFlag:capsules-unsupported for more information.
HP Compaq 6200 Pro MT PC
│
└─TEAM T253X2001T:
      Device ID:          602b0a6cc821d155208724f0e22f8d111542b74c
      Summary:            ATA drive
      Current version:    S1024A0
      Vendor:             Team Group (ATA:0x0)
      GUIDs:              a1866d41-b70d-528c-94a5-b51933938f63
                          ed048768-7b7c-5298-a428-a8bac025d42c
                          d7e8f9e6-a06e-5510-a30a-b93c6b053952
      Device Flags:       • Internal device
                          • Updatable
                          • System requires external power source
                          • Needs a reboot after installation
                          • Device is usable for the duration of the update
15

Some month ago I experienced the same, the reason for it was a “dead” harddisk, after replacement by a new one all was working like a rocket. :slightly_smiling_face:

16

Wait, how much the same? Like it was fine normally, but slow just for offline updates?

17

Frank: Thanks. I didn’t know about fwupdmgr. It says I have no firmware updates available when I try “fwupdmgr update”. It lists my firmware version as 36307029, which isn’t even the same kind of identifier as what’s on that website. I thought those listings on their website were just for Windows?

18

PS to heliosstyx: Were you able to confirm a problem with your drive before you replaced it? Like maybe with manufacturer’s diagnostics in Windows or something? Or did you just have to take a shot in the dark by buying a new one?

19

If that was a spinning-rust traditional magnetic HDD, I can certainly see how that could happen.

(The package transaction process is orders of magnitude more likely to cause thrashing and heavy disk load than anything else you do with your computer, normally, so a failing drive would definitely make itself known during that process.)

But if that was the case, I don’t see how it could apply here, given the differences in storage technology. An SSD can’t thrash, for one thing.

There certainly could be storage issues at play here, especially with such young hardware, but they’d be very different than a dying magnetic platter drive, even if they happened to present the same way.

It does raise the point, though, that it’s probably worth checking the SMART data for the NVMe device, to see if it’s logging any interface errors or resets… even temperature extremes could be a problem, and should be logged. Something like this will dump its entire sordid history…

sudo smartctl -x /dev/whatever

(I’m not sure what “whatever” would be… whatever the root block device for the disk is called. With a SATA drive it’d be /dev/sda though I assume NVMe devices aren’t addressed the same way.)

On the firmware, the downloads on Seagate’s site would be geared towards Windows, yeah – that’s why the LVFS exists – but the actual firmware usually gets flashed into NVRAM on the microcontroller of the hardware itself. It should be OS agnostic for the most part. (It’s not like the firmware currently on the drive is in any way Linux-specific.)

I have no explanation for the difference in version formatting, though, and I’d certainly recommend against flashing firmware unless it’s 100% confirmed to be a match for the device, so let’s put that aside.

(Most likely wrong firmware would refuse to install itself, but no sense taking chances.)

20

Thanks! I had forgotten that firmware resided on the device itself. In a couple weeks I plan to install a second NVMe card and put Windows on it. Assuming Windows can flash the drive that Fedora is installed on, I should be able to update the firmware that way. Smartctl shows a surprising number of unsafe shutdowns, but otherwise all seems well; zero for Media and Data Integrity Errors and zero for Error Information Log Entries.

21

I am not a shooter into the dark. The manufacturer diagnostic program(WD) has warned of the upcoming defect of the device. Windows has not warned me. Afterwards Fedora Workstation was
going too into the super-slow motion mode (booting, Login, and the desktop itself). Another warning for a normal consumer ssd device: avoid file-systems with high I/O rates (esp. writing and dynamic swapping), because the wear rate of the cells will increase strongly and the device will die fast.