How to fix infinitely restarting `systemd-journald` job at boot?

Ask Fedora
, , , , , , , ,
1

I very recently invoked the undermentioned:

#!/usr/bin/env -S bash
sudo dnf upgrade --offline -y && \
sudo dnf offline reboot -y

This should be safe (I did check the output, despite -y) and has worked this far.

Unfortunately, at the end of the update dmesg log on the first reboot of the two it required to apply, something appeared to fail.

Subsequently, on every boot attempt, systemd-journald fails with ever-increasing timers, automatically restarting itself.

Can this be remediated? I’ve cut my diagnosis short, since:

  1. I don’t want to do damage to the filesystem during each force-reboot when the fails due to fedora - Why is Magic SysRq not enabled by default on some systems? Is there a risk? - Unix & Linux Stack Exchange.

  2. There doesn’t appear to be a way to skip the job at boot, per systemd: How to stop a start-job without limit - Unix & Linux Stack Exchange.

    1. systemd skip troublesome target at boot / Newbie Corner / Arch Linux Forums isn’t applicable to this, because, systemd - How can journald be disabled? - Unix & Linux Stack Exchange explains that journald appears to be impossible to disable without breaking the installation due to dependency failures.

    2. I don’t know what linux wont boot, how do u skip the starting of a service??? means.

  3. According to the answers to systemd - How to clear journalctl - Unix & Linux Stack Exchange, there is no way to entirely clear its logs (and the methods mentioned don’t remediate my problem, despite appearing to delete almost every log).

The sole success I’ve had thus far is at least making the TTY be useable, per How to block messages from being printed in active ttys? - Ask Ubuntu, so at least I’ve learnt something…

2

Start by checking for disk full or corrupt filesystem.

If your console is usable (e.g., using the grub editor to add <space>3 to the end of the kernel command-line, you should be able to check space (note that df is not useful on btrfs (and really should warn users who run df on a btrfs filsystem) and also do a safe shutdown or restart.

If console is not usable, you can boot a Live Installer, mount the system root drive (e.g., with Gnome Disks) and check for filesystem problems.

If the root filesystem is OK, please provide the output from running inxi -Fzxx in a terminal (if you are using a “Live” environment, you may have to install inxi) so we can understand your hardware.

1 Like
3

@gnwiii, many thanks for the verbose diagnostic instructions. Fortunately for me, I merely reinstalled the OS, as undermentioned:

However, should this occur again (it may, for similar has occurred in the past, for me) I shan’t forget to provide this information.

Irrespective, for posterity:

If, by console, you refer to the rescue kernel TTY1 CLI (rather than GRUB2’s CLI, accessible via Esc from its pre-boot TUI), then it was. I could even access journalctl, and systemctl reported that it was healthy. This is presumably as perplexing to you as it was me.

1 Like
4

We mean the text console you get to after booting the system.
If a GUI is running then you type Ctrl-Alt-F3 (for example) and you should see a login: prompt.

I use systemctl --failed as a quick check that services are working.

I not sure that you would be able to see the logs in the status of the systemd-journald.service as I’m not sure where its logs go when it fails.
Logs from service go to the journal normally, but when journald is broken that cannot happen.

5

@barryascott, after? If you mean during, that’s dmesg, right? I had GRUB_CMDLINE_LINUX= #'d (rather than the default of GRUB_CMDLINE_LINUX="rhgb quiet") so I could see everything after my monitor initialized (which is annoyingly slow).

I couldn’t boot using the regular kernel, so couldn’t get to a TTY (much less TTY3, since the rescue kernel didn’t permit any TTYs other than TTY1).

That mostly listed snapd processes, weirdly. I presume they’re not designed to cope with a rescue kernel. I uninstalled snapd, systemctl reboot’d, and saw solely two remained. Their names were cryptic, so I forget. Apologies.

It was functioning in the rescue kernel’s TTY(1), just not the standard kernel.