Since upgrading my headless server to F43, the vncserver instance I use to run applications on that box has been having severe troubles with the systemd user session.
I can’t rule out that I may have created the problem myself due to the way the whole thing is set up. (It’s a configuration that’s developed organically, in bits and pieces, over some 10 years of Fedora releases. So, there may be suboptimal aspects of how everything is lashed together with duct tape and stubbornness.) But, I seem to mostly be using supported tools and standard configurations, and it was working before the upgrade, so I’m hoping there’s some way to get things back on track. Here’s what’s going on, and what I’ve been able to unearth so far regarding why:
The VNC desktop on the machine is an LXQT session, configured as a vncserver@:1.service templated unit in the root systemd that’s started at boot. I access the machine and its files by sftp-mounting it to my desktop system, by ssh-ing to its command line from gnome-terminal windows, and by opening vncviewer windows tunneled over ssh to use some GUI applications I keep running on it at all times. It’s those last interactions where the trouble has manifested.
The primary symptom is that any attempts at executing systemctl --user commands in the LXQT session’s terminal windows result in an instant failure:
$ systemctl --user
Failed to list units: Process org.freedesktop.systemd1 exited with status 1
In addition, most of the user units for my session aren’t being started like they should be when vncserver starts. systemd appears to be extremely broken in the LXQT session / vncserver X11 instance.
I can run sudo systemctl commands just fine (even from the VNC session), and I can use sudo -u ferd systemctl --user commands inside those VNC-hosted terminal windows to talk to some version of my user session manager without the commands crashing. It’s that aspect that led me to the most useful discoveries I’ve made so far about what seems to be happening.
By running all of the commands I mentioned above with SYSTEMD_LOG_LEVEL=debug in the process environment, I see that the difference between the sudo -u ferd comands and the ones without it are that they’re talking to different D-Bus session bus instances — apparently, one of which works, and the other which instantly crashes systemctl.
When run with the sudo wrapper, systemctl connects to a bus manager socket at /run/user/1000/bus and is able to activate and communicate with systemd to manage a user session. Without the sudo -u ferd, systemctl --user is attempting to access a session bus socket located in /tmp/. This d-bus instance, which apparently breaks systemctl, does appear to be the session bus for the LXQT session on the vncserver desktop. It’s owned by /usr/bin/dbus-daemon, and its address is set as the DBUS_SESSION_BUS_ADDRESS in my vncserver-hosted environment:
$ env |grep DBUS
DBUS_SESSION_BUS_ADDRESS=unix:path=/tmp/dbus-Qjt4Svew6q,guid=3f27532d8c889c8b4b9
$ sudo -u ferd env|grep DBUS
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
$ SYSTEMD_LOG_LEVEL=debug systemctl --user |& grep -E '(bus|error)'
$XDG_RUNTIME_DIR not set, unable to connect to private bus. Falling back to session bus.
sd-bus: starting bus by connecting to /tmp/dbus-Qjt4Svew6q...
Sent message type=method_call sender=n/a destination=org.freedesktop.DBus path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=Hello cookie=1 reply_cookie=0 signature=n/a error-name=n/a error-message=n/a
Got message type=method_return sender=org.freedesktop.DBus destination=:1.70 path=n/a interface=n/a member=n/a cookie=1 reply_cookie=1 signature=s error-name=n/a error-message=n/a
Sent message type=method_call sender=n/a destination=org.freedesktop.systemd1 path=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=ListUnitsByPatterns cookie=2 reply_cookie=0 signature=asas error-name=n/a error-message=n/a
Got message type=error sender=org.freedesktop.DBus destination=:1.70 path=n/a interface=n/a member=n/a cookie=3 reply_cookie=2 signature=s error-name=org.freedesktop.DBus.Error.Spawn.ChildExited error-message=Process org.freedesktop.systemd1 exited with status 1
$ sudo -u ferd SYSTEMD_LOG_LEVEL=debug systemctl --user |& grep -E '(bus|error)'
sd-bus: starting bus by connecting to /run/user/1000/systemd/private...
Sent message type=method_call sender=n/a destination=org.freedesktop.systemd1 path=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=ListUnitsByPatterns cookie=1 reply_cookie=0 signature=asas error-name=n/a error-message=n/a
Got message type=method_return sender=org.freedesktop.systemd1 destination=n/a path=n/a interface=n/a member=n/a cookie=1 reply_cookie=1 signature=a(ssssssouso) error-name=n/a error-message=n/a
dbus-broker.service loaded active running D-Bus User Message Bus
dbus.socket loaded active running D-Bus User Message Bus Socket
The reason dbus-daemon is running in the vncserver session is, it’s started by /etc/X11/xinit/xinitrc.d/00-start-message-bus.sh when Xvnc starts up:
$ grep -v '^#' /etc/X11/xinit/xinitrc.d/00-start-message-bus.sh
if [ -z "$DBUS_SESSION_BUS_ADDRESS" ]; then
eval `dbus-launch --sh-syntax --exit-with-session`
fi
But there’s also a dbus-broker daemon running under my UID, apparently started by /usr/lib/systemd/systemd --user , and that appears to be the owner of /run/user/1000/bus:
$ sudo lsof -p 6212 |grep -i bus
systemd 6212 ferd 12u unix 0x000000006040bea8 0t0 35252 /run/user/1000/bus type=STREAM (LISTEN)
systemd 6212 ferd 26u unix 0x00000000cde65c0e 0t0 34367 @ecbeecd98f2e02d6/bus/systemd/bus-system type=STREAM (CONNECTED)
systemd 6212 ferd 44u unix 0x00000000d75cab3c 0t0 38095 @faf1d3104b5def6f/bus/systemd/bus-api-user type=STREAM (CONNECTED)
The dbus-daemon instance is provided by the dbus-daemon package, and the config file that launches it comes from dbus-x11. But I can’t remove either of those packages without them taking major parts of the desktop session with them:
$ sudo dnf remove dbus-x11
Package Arch Version Repository Size
Removing:
dbus-x11 x86_64 1:1.16.0-4.fc43 fedora 35.9 KiB
Removing dependent packages:
firewall-config noarch 2.3.1-5.fc43 fedora 1.0 MiB
lxqt-panel x86_64 2.2.2-3.fc43 updates 5.6 MiB
lxqt-panel-devel x86_64 2.2.2-3.fc43 updates 16.7 KiB
lxqt-runner x86_64 2.2.0-2.fc43 fedora 420.4 KiB
lxqt-x11-session noarch 2.2.0-5.fc43 fedora 6.2 KiB
system-config-printer x86_64 1.5.18-16.fc43 fedora 1.8 MiB
tigervnc-server x86_64 1.15.0-7.fc43 fedora 613.3 KiB
tigervnc-server-minimal x86_64 1.15.0-7.fc43 fedora 2.9 MiB
xpra x86_64 1:6.3.4-1.fc43 updates 26.3 MiB
Removing unused dependencies:
libgle x86_64 3.1.0-36.fc43 fedora 138.8 KiB
libstatgrab x86_64 1:0.92.1-14.fc43 fedora 140.2 KiB
libsysstat x86_64 1.1.0-3.fc43 fedora 106.3 KiB
log4cplus x86_64 2.1.2-3.fc43 fedora 1.3 MiB
lxqt-globalkeys x86_64 2.2.0-2.fc43 fedora 797.4 KiB
lxqt-menu-data noarch 2.2.0-2.fc43 fedora 195.0 KiB
python3-lz4 x86_64 4.4.4-3.fc43 fedora 6.1 MiB
tigervnc-selinux noarch 1.15.0-7.fc43 fedora 14.8 KiB
xorg-x11-drv-dummy x86_64 0.4.1-7.fc43 fedora 29.1 KiB
xscreensaver x86_64 1:6.12-3.fc43 fedora 0.0 B
xscreensaver-base x86_64 1:6.12-3.fc43 fedora 1.7 MiB
xscreensaver-extras x86_64 1:6.12-3.fc43 fedora 10.5 MiB
xscreensaver-extras-base x86_64 1:6.12-3.fc43 fedora 684.0 KiB
xscreensaver-gl-base x86_64 1:6.12-3.fc43 fedora 20.3 KiB
xscreensaver-gl-extras x86_64 1:6.12-3.fc43 fedora 40.4 MiB
Transaction Summary:
Removing: 25 packages
After this operation, 101 MiB will be freed (install 0 B, remove 101 MiB).
Is this ok [y/N]: n
Operation aborted by the user.
$ sudo dnf remove dbus-daemon
Package Arch Version Reposit Size
Removing:
dbus-daemon x86_64 1:1.16.0-4.fc43 fedora 440.2 KiB
Removing dependent packages:
dbus-tests x86_64 1:1.16.0-4.fc43 fedora 1.8 MiB
dbus-x11 x86_64 1:1.16.0-4.fc43 fedora 35.9 KiB
firewall-config noarch 2.3.1-5.fc43 fedora 1.0 MiB
gdm x86_64 1:49.1-1.fc43 fedora 5.0 MiB
gnome-initial-setup x86_64 49.0-1.fc43 fedora 1.9 MiB
gnome-session-wayland-session x86_64 49.1-1.fc43 fedora 16.2 KiB
gnome-shell x86_64 49.1-2.fc43 updates 13.9 MiB
gnome-shell-extension-apps-menu noarch 49.0-2.fc43 fedora 22.1 KiB
gnome-shell-extension-common noarch 49.0-2.fc43 fedora 615.4 KiB
gnome-shell-extension-launch-new-instance noarch 49.0-2.fc43 fedora 1.4 KiB
gnome-shell-extension-openweather noarch 121-9.20240421git911737 fedora 575.4 KiB
gnome-shell-extension-user-theme noarch 49.0-2.fc43 fedora 7.3 KiB
lxqt-panel x86_64 2.2.2-3.fc43 updates 5.6 MiB
lxqt-panel-devel x86_64 2.2.2-3.fc43 updates 16.7 KiB
lxqt-runner x86_64 2.2.0-2.fc43 fedora 420.4 KiB
lxqt-x11-session noarch 2.2.0-5.fc43 fedora 6.2 KiB
recoll-gssp x86_64 1.43.6-1.fc43 updates 24.5 KiB
system-config-printer x86_64 1.5.18-16.fc43 fedora 1.8 MiB
tigervnc-server x86_64 1.15.0-7.fc43 fedora 613.3 KiB
tigervnc-server-minimal x86_64 1.15.0-7.fc43 fedora 2.9 MiB
xpra x86_64 1:6.3.4-1.fc43 updates 26.3 MiB
Removing unused dependencies:
gnome-shell-common noarch 49.1-2.fc43 updates 16.4 KiB
gnome-tour x86_64 48.1-3.fc43 fedora 2.4 MiB
libgle x86_64 3.1.0-36.fc43 fedora 138.8 KiB
libstatgrab x86_64 1:0.92.1-14.fc43 fedora 140.2 KiB
libsysstat x86_64 1.1.0-3.fc43 fedora 106.3 KiB
log4cplus x86_64 2.1.2-3.fc43 fedora 1.3 MiB
lxqt-globalkeys x86_64 2.2.0-2.fc43 fedora 797.4 KiB
lxqt-menu-data noarch 2.2.0-2.fc43 fedora 195.0 KiB
python3-lz4 x86_64 4.4.4-3.fc43 fedora 6.1 MiB
python3-pam noarch 2.0.2-16.fc43 fedora 53.7 KiB
python3-pydbus noarch 0.6.0-33.fc43 fedora 149.4 KiB
qadwaitadecorations-qt5 x86_64 0.1.7-2.fc43 updates 151.8 KiB
qt6-qtwayland-adwaita-decoration x86_64 6.10.0-2.fc43 updates 126.5 KiB
tigervnc-selinux noarch 1.15.0-7.fc43 fedora 14.8 KiB
xdg-desktop-portal-gnome x86_64 49.0-1.fc43 fedora 963.3 KiB
xorg-x11-drv-dummy x86_64 0.4.1-7.fc43 fedora 29.1 KiB
xscreensaver x86_64 1:6.12-3.fc43 fedora 0.0 B
xscreensaver-base x86_64 1:6.12-3.fc43 fedora 1.7 MiB
xscreensaver-extras x86_64 1:6.12-3.fc43 fedora 10.5 MiB
xscreensaver-extras-base x86_64 1:6.12-3.fc43 fedora 684.0 KiB
xscreensaver-gl-base x86_64 1:6.12-3.fc43 fedora 20.3 KiB
xscreensaver-gl-extras x86_64 1:6.12-3.fc43 fedora 40.4 MiB
Transaction Summary:
Removing: 44 packages
After this operation, 129 MiB will be freed (install 0 B, remove 129 MiB).
Is this ok [y/N]: n
Operation aborted by the user.
So I’m not sure what to do. Is my vncserver session not supposed to be running dbus-daemon, but instead using the dbus-broker bus that systemd --user starts? Should I just delete the 00-start-message-bus.sh file from the xinitrc.d directory, to prevent it from being started by Xvnc? And why did this only now change, in Fedora 43?