Empty Boxes with libvirtd 9 failing

[riabenko (Roman Riabenko)]

GNOME Boxes does not display existing VMs. If selecting the menu entry to add a new VM, GNOME Boxes crashes. It appears that libvirtd daemon fails to start due to an issue with the storage driver plugin for rbd:

$ LC_ALL=C gnome-boxes
Error setting up default broker: Failed to fetch list of pools: Failed to connect socket to '/run/user/1000/libvirt/virtstoraged-sock': No such file or directory

(gnome-boxes:3826): Boxes-WARNING **: 14:49:49.090: (../src/app.vala:440):boxes_app_setup_default_source_co: runtime check failed: (default_connection != null)

$ systemctl start libvirtd
Job for libvirtd.service failed because the control process exited with error code.
See "systemctl status libvirtd.service" and "journalctl -xeu libvirtd.service" for details.

$ systemctl status libvirtd.service
× libvirtd.service - Virtualization daemon
     Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; disabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Thu 2023-05-04 14:50:47 EEST; 33s ago
TriggeredBy: × libvirtd-admin.socket
             × libvirtd-ro.socket
             × libvirtd.socket
             ○ libvirtd-tcp.socket
             ○ libvirtd-tls.socket
       Docs: man:libvirtd(8)
             https://libvirt.org
    Process: 4068 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS (code=exited, status=3)
   Main PID: 4068 (code=exited, status=3)
        CPU: 45ms

тра 04 14:50:47 fedora systemd[1]: libvirtd.service: Scheduled restart job, restart counter is at 5.
тра 04 14:50:47 fedora systemd[1]: Stopped libvirtd.service - Virtualization daemon.
тра 04 14:50:47 fedora systemd[1]: libvirtd.service: Start request repeated too quickly.
тра 04 14:50:47 fedora systemd[1]: libvirtd.service: Failed with result 'exit-code'.
тра 04 14:50:47 fedora systemd[1]: Failed to start libvirtd.service - Virtualization daemon.

$ journalctl -xeu libvirtd.service
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The unit libvirtd.service has entered the 'failed' state with result 'exit-code'.
тра 04 14:50:47 fedora systemd[1]: Failed to start libvirtd.service - Virtualization daemon.
░░ Subject: A start job for unit libvirtd.service has failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit libvirtd.service has finished with a failure.
░░ 
░░ The job identifier is 4180 and the job result is failed.
тра 04 14:50:47 fedora systemd[1]: libvirtd.service: Scheduled restart job, restart counter is at 5.
░░ Subject: Automatic restarting of a unit has been scheduled
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ Automatic restarting of the unit libvirtd.service has been scheduled, as the result for
░░ the configured Restart= setting for the unit.
тра 04 14:50:47 fedora systemd[1]: Stopped libvirtd.service - Virtualization daemon.
░░ Subject: A stop job for unit libvirtd.service has finished
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A stop job for unit libvirtd.service has finished.
░░ 
░░ The job identifier is 4318 and the job result is done.
тра 04 14:50:47 fedora systemd[1]: libvirtd.service: Start request repeated too quickly.
тра 04 14:50:47 fedora systemd[1]: libvirtd.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The unit libvirtd.service has entered the 'failed' state with result 'exit-code'.
тра 04 14:50:47 fedora systemd[1]: Failed to start libvirtd.service - Virtualization daemon.
░░ Subject: A start job for unit libvirtd.service has failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit libvirtd.service has finished with a failure.
░░ 
░░ The job identifier is 4318 and the job result is failed.

$ LC_ALL=C libvirtd
2023-05-04 11:38:33.644+0000: 2561: info : libvirt version: 9.0.0, package: 2.fc38 (Fedora Project, 2023-01-19-19:58:16, )
2023-05-04 11:38:33.644+0000: 2561: info : hostname: fedora
2023-05-04 11:38:33.644+0000: 2561: error : virModuleLoadFile:53 : internal error: Failed to load module '/usr/lib64/libvirt/storage-backend/libvirt_storage_backend_rbd.so': /usr/lib64/ceph/libceph-common.so.2: invalid ELF header

$ LC_ALL=C dnf provides /usr/lib64/libvirt/storage-backend/libvirt_storage_backend_rbd.so
Last metadata expiration check: 4:20:53 ago on Thu May  4 10:35:36 2023.
libvirt-daemon-driver-storage-rbd-9.0.0-2.fc38.x86_64 : Storage driver plugin for rbd
Repo        : @System
Matched from:
Filename    : /usr/lib64/libvirt/storage-backend/libvirt_storage_backend_rbd.so

libvirt-daemon-driver-storage-rbd-9.0.0-2.fc38.x86_64 : Storage driver plugin for rbd
Repo        : fedora
Matched from:
Filename    : /usr/lib64/libvirt/storage-backend/libvirt_storage_backend_rbd.so

I reinstalled the libvirt-daemon-driver-storage-rbd package with dnf, but it did not have any effect. What should I do to make libvirtd start?

[vgaetera (Vladislav Grigoryev)]

Check the output:

gnome-boxes --checks
virsh pool-list --all
systemctl --failed
journalctl --no-pager -b -g avc
sudo rpm -V -a

[riabenko (Roman Riabenko)]

Thank you! In the output of sudo rpm -V -a I noticed a few files ending with .so. So, I identified with dnf provides the packages that those files originate from, and reinstalled those packages to get those libraries fixes. The most suspicious one was from elfutils-libs. Now, everything seems working fine.