Boot takes a long time

Hi,
For some time now the boot times of my system are much longer than they used to be.
Output of rpm-ostree status:

State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: no runs since boot
Deployments:
● ostree://fedora-workstation:fedora/30/x86_64/silverblue
                Version: 30.20190813.0 (2019-08-13T00:52:45Z)
            BaseCommit: 152958ea105a1a076db7e7514e64b99d227e53085aa4aede2f1a99a3a602352a
          GPGSignature: Valid signature by F1D8EC98F241AAF20DF69420EF3C111FCFC659B9
       LayeredPackages: chromium samba

 ostree://fedora-workstation:fedora/30/x86_64/silverblue
               Version: 30.20190811.0 (2019-08-11T00:52:07Z)
            BaseCommit: 8763bfd44c22a70a903023d3fbea3a9b9b4b3e5654aedca3330010cf8561ba76
          GPGSignature: Valid signature by F1D8EC98F241AAF20DF69420EF3C111FCFC659B9
       LayeredPackages: chromium samba

 ostree://fedora-workstation:fedora/29/x86_64/silverblue
               Version: 29.20190123.0 (2019-01-23T01:18:07Z)
            BaseCommit: c44dbb8bed4e1e3b6a94c51749e96f9c6d4f5841a5179ef89ebd7472263ca93f
          GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4
   RemovedBasePackages: firefox 64.0.2-1.fc29
       LayeredPackages: baobab chromium snapd
                Pinned: yes

My hardware setup is a Lenovo E480 using an internal ssd + additional 2.5 inch ssd and normally should boot quite fast. Instead since some time now, the boot times are inconsistent but nearly always longer than one minute.
Here are some example times:
Output of systemd-analyze blame:

     47.935s plymouth-quit-wait.service
     26.096s firewalld.service
     23.513s ldconfig.service
     13.728s dracut-initqueue.service
     12.945s systemd-cryptsetup@luks\x2d34a2958f\x2d0801\x2d45e7\x2dbe21\x2d2b805884bf62.service
     12.833s systemd-cryptsetup@luks\x2d8fa4c62c\x2d4820\x2d470b\x2d8c3d\x2dd15e6da50f02.service
     10.575s initrd-switch-root.service
     10.138s NetworkManager-wait-online.service
      5.849s lvm2-monitor.service
      5.640s fwupd.service
      5.047s bolt.service
      4.742s udisks2.service
      2.527s systemd-hwdb-update.service
      2.417s NetworkManager.service
      2.339s polkit.service
      2.297s systemd-cryptsetup@luks\x2ddc89f89d\x2d2edd\x2d4da2\x2d932d\x2d75f273b13032.service
      2.135s user@1000.service
      1.739s sssd.service
      1.734s systemd-udevd.service
      1.728s ModemManager.service
      1.715s accounts-daemon.service
      1.670s cups.service
      1.267s systemd-journal-flush.service
      1.085s lvm2-pvscan@259:1.service
      1.082s systemd-vconsole-setup.service
      1.054s lvm2-pvscan@8:3.service
       986ms bluetooth.service
       780ms chronyd.service
       773ms avahi-daemon.service
       763ms wpa_supplicant.service
       735ms systemd-logind.service
       662ms rtkit-daemon.service
       602ms systemd-journald.service
       565ms systemd-fsck@dev-disk-by\x2duuid-4e7821d0\x2d9401\x2d4aed\x2d8f7d\x2d3c7833aa1966.service
       564ms dev-mapper-luks\x2d8fa4c62c\x2d4820\x2d470b\x2d8c3d\x2dd15e6da50f02.swap
       540ms systemd-tmpfiles-setup.service
       526ms systemd-sysctl.service
       523ms systemd-sysusers.service
       521ms gdm.service
       498ms systemd-udev-trigger.service
       470ms gssproxy.service
       454ms nfs-convert.service
       423ms systemd-fsck@dev-disk-by\x2duuid-0F88\x2d18CA.service
       388ms systemd-remount-fs.service
       364ms colord.service
       361ms kmod-static-nodes.service
       361ms dev-mqueue.mount
       359ms sys-kernel-debug.mount
       358ms dev-hugepages.mount
       345ms switcheroo-control.service
       308ms rpc-statd-notify.service
       293ms systemd-tmpfiles-setup-dev.service
       253ms boot-efi.mount
       252ms systemd-modules-load.service
       237ms user-runtime-dir@1000.service
       211ms plymouth-read-write.service
       200ms sys-fs-fuse-connections.mount
       198ms rpm-ostreed.service
       190ms ostree-remount.service
       184ms dbus-broker.service
       181ms systemd-backlight@backlight:intel_backlight.service
       180ms systemd-fsck@dev-mapper-luks\x2ddc89f89d\x2d2edd\x2d4da2\x2d932d\x2d75f273b13032.service
       154ms initrd-parse-etc.service
       133ms upower.service
       131ms systemd-update-done.service
       108ms systemd-user-sessions.service
        95ms systemd-random-seed.service
        93ms systemd-fsck-root.service
        84ms dracut-cmdline.service
        53ms plymouth-switch-root.service
        49ms geoclue.service
        41ms flatpak-system-helper.service
        39ms sysroot.mount
        38ms var-lib-nfs-rpc_pipefs.mount
        37ms dracut-pre-trigger.service
        35ms auditd.service
        30ms var-home.mount
        28ms dracut-shutdown.service
        18ms systemd-update-utmp-runlevel.service
        17ms sssd-kcm.service
        16ms dracut-pre-udev.service
        16ms boot.mount
        16ms plymouth-start.service
        16ms initrd-cleanup.service
        14ms systemd-journal-catalog-update.service
        13ms var.mount
        12ms tmp.mount
        11ms tpm2-abrmd.service
        11ms dracut-pre-pivot.service
        10ms dracut-pre-mount.service
         8ms systemd-rfkill.service
         8ms systemd-update-utmp.service
         6ms initrd-udevadm-cleanup-db.service
         2ms sys-kernel-config.mount

Output of systemd-analyze critical-chain:

The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.

graphical.target @1min 59.579s
└─multi-user.target @1min 59.579s
 └─systemd-user-sessions.service @1min 11.528s +108ms
  └─network.target @1min 11.526s
   └─wpa_supplicant.service @1min 14.626s +763ms
    └─dbus-broker.service @43.575s +184ms
      └─dbus.socket @42.198s
        └─sysinit.target @42.197s
          └─systemd-update-done.service @42.066s +131ms
            └─ldconfig.service @18.549s +23.513s
              └─local-fs.target @18.539s
                └─run-user-1000.mount @1min 11.875s
                  └─local-fs-pre.target @15.899s
                    └─lvm2-monitor.service @10.044s +5.849s
                      └─lvm2-lvmetad.service @10.972s
                        └─lvm2-lvmetad.socket @10.042s
                          └─-.mount
                            └─systemd-journald.socket
                              └─-.mount
                                └─...

The output of the corresponding journalctl -b can be found here:
https://paste.fedoraproject.org/paste/k~59ALdOkrSNLxirpQHOBw

Hmm ldconfig.service seems to be taking quite a bit, how long does it take if you run sudo ldconfig -X yourself?

It takes around 100 ms.

For comparison some outputs for the same system booting much faster, which happens sometimes.
Maybe the reason for the long boot times can be determined using this as a comparison:

Output of journalctl -b:

https://paste.fedoraproject.org/paste/JsuGk14MFNLuksZHBP6mxQ#

Output of systemd-analyze blame:

     19.322s dracut-initqueue.service
     18.669s systemd-cryptsetup@luks\x2d34a2958f\x2d0801\x2d45e7\x2dbe21\x2d2b805884bf62.service
     12.017s systemd-cryptsetup@luks\x2d8fa4c62c\x2d4820\x2d470b\x2d8c3d\x2dd15e6da50f02.service
      6.296s NetworkManager-wait-online.service
      5.514s fwupd.service
      5.037s bolt.service
      3.605s plymouth-quit-wait.service
      3.175s systemd-cryptsetup@luks\x2ddc89f89d\x2d2edd\x2d4da2\x2d932d\x2d75f273b13032.service
      1.205s firewalld.service
       876ms initrd-switch-root.service
       570ms lvm2-monitor.service
       465ms udisks2.service
       463ms systemd-logind.service
       412ms systemd-hostnamed.service
       340ms lvm2-pvscan@8:3.service
       335ms user@1000.service
       330ms systemd-localed.service
       277ms lvm2-pvscan@259:1.service
       255ms systemd-udevd.service
       252ms cups.service
       245ms rpm-ostreed.service
       242ms systemd-journald.service
       227ms sssd.service
       201ms polkit.service
       192ms ModemManager.service
       170ms systemd-journal-flush.service
       144ms accounts-daemon.service
       139ms systemd-fsck@dev-disk-by\x2duuid-4e7821d0\x2d9401\x2d4aed\x2d8f7d\x2d3c7833aa1966.service
       133ms avahi-daemon.service
       130ms systemd-vconsole-setup.service
       128ms systemd-tmpfiles-setup.service
       114ms bluetooth.service
       110ms upower.service
       105ms rtkit-daemon.service
       103ms systemd-udev-trigger.service
        92ms initrd-parse-etc.service
        84ms dracut-cmdline.service
        62ms geoclue.service
        61ms dbus-broker.service
        60ms systemd-fsck@dev-disk-by\x2duuid-0F88\x2d18CA.service
        58ms dev-mqueue.mount
        58ms sys-kernel-debug.mount
        57ms systemd-remount-fs.service
        54ms kmod-static-nodes.service
        49ms boot-efi.mount
        48ms dev-mapper-luks\x2d8fa4c62c\x2d4820\x2d470b\x2d8c3d\x2dd15e6da50f02.swap
        48ms NetworkManager.service
        43ms nfs-convert.service
        41ms colord.service
        41ms systemd-modules-load.service
        39ms chronyd.service
        37ms dracut-pre-trigger.service
        36ms var-lib-nfs-rpc_pipefs.mount
        35ms dev-hugepages.mount
        35ms auditd.service
        34ms user-runtime-dir@1000.service
        32ms flatpak-system-helper.service
        32ms gssproxy.service
        30ms systemd-rfkill.service
        28ms systemd-tmpfiles-setup-dev.service
        27ms systemd-fsck@dev-mapper-luks\x2ddc89f89d\x2d2edd\x2d4da2\x2d932d\x2d75f273b13032.service
        27ms realmd.service
        26ms gdm.service
        25ms boot.mount
        25ms plymouth-switch-root.service
        22ms var-home.mount
        20ms systemd-fsck-root.service
        19ms systemd-backlight@backlight:intel_backlight.service
        19ms wpa_supplicant.service
        18ms plymouth-read-write.service
        17ms switcheroo-control.service
        17ms dracut-pre-udev.service
        16ms initrd-cleanup.service
        15ms rpc-statd-notify.service
        15ms plymouth-start.service
        15ms systemd-update-utmp-runlevel.service
        15ms sssd-kcm.service
        12ms systemd-sysctl.service
        12ms systemd-random-seed.service
        11ms sys-fs-fuse-connections.mount
        11ms dracut-pre-pivot.service
        10ms systemd-user-sessions.service
         8ms tpm2-abrmd.service
         8ms ostree-remount.service
         8ms sysroot.mount
         8ms dracut-pre-mount.service
         7ms systemd-update-utmp.service
         7ms dracut-shutdown.service
         7ms tmp.mount
         7ms initrd-udevadm-cleanup-db.service
         7ms var.mount
         1ms sys-kernel-config.mount

Output of systemd-analyze critical-chain:

The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.

graphical.target @9.819s
└─multi-user.target @9.818s
 └─systemd-user-sessions.service @6.195s +10ms
  └─network.target @6.192s
   └─wpa_supplicant.service @6.737s +19ms
    └─dbus-broker.service @4.965s +61ms
      └─dbus.socket @4.918s
        └─sysinit.target @4.916s
          └─systemd-update-utmp.service @4.908s +7ms
            └─auditd.service @4.871s +35ms
              └─systemd-tmpfiles-setup.service @4.740s +128ms
                └─local-fs.target @4.735s
                  └─run-user-1000.mount @6.252s
                    └─local-fs-pre.target @1.382s
                      └─lvm2-monitor.service @797ms +570ms
                        └─lvm2-lvmetad.service @818ms
                          └─lvm2-lvmetad.socket @795ms
                            └─system.slice
                              └─-.slice

Two things immediately stick out:

  • ldconfig didn’t run this time.
  • firewalld was far, far faster at starting.