GRUB2 takes way too long to load kernel and initrd

plymouth may also be waiting for the nvidia graphic driver to load, and not using
simpledrm because you are using LUKS.
Try to add to the Linux entry in grub: plymouth.use-simpledrm=1 (or suppress rhgb)

Ref: Changes/PlymouthUseSimpledrm - Fedora Project Wiki

The loader time reported by systemd-analyze is calculated from information provided by firmware. It isn’t related to anything the kernel has done so tools like dmesg won’t help - that 6 minute delay has already happened.

Firmware is the time the BIOS/UEFI reports that it needed to get as far as the bootloader (GRUB).
Loader time is calculated based on the time between then and when the kernel’s clock started.
Kernel time (~1.4s) then covers everything in the log until systemd is run as the init process (line 696), and initrd covers the time up until root is mounted and switched to (so somewhere between line 794 and 832 in the log)

(You can dig around in the code from boot-timestamps.c and acpi-fpdt.c if like me this made you wonder where time information from before the kernel is running comes from)

Assuming it’s not a bug in how the numbers are calculated (which has apparently happened before, but seems unlikely if @genteal checked it with a stopwatch) then reducing or diagnosing that loader delay time is going to mean plenty of reboots and testing and debugging things in GRUB itself.

@genteal - Just to double check, but I assume GRUB is configured with a timeout, so that it hasn’t had to wait for you to press anything before it starts loading?
A copy of /etc/default/grub might be useful just to see that nothing odd is configured there.

Just wanted to have this made clear. Thanks.

Sadly, I tried booting while everything was unplugged apart from AC and monitors, it did not work.

I also tried disabling Legacy USB as suggested, but it did not help either (that one even locked me out of UEFI Setup and GRUB, I did clear CMOS to revert that after measuring boot time).

These suggestions are very good.

To test that you can force the grub menu to display during boot by holding down the shift key or the esc key. Once the menu is displayed press the e key (for edit) and edit the line that begins with linux to add the one suggested option (plymouth.use-simpledrm=1) and/or remove the other option (rhgb). Continue booting with ctrl+x

If that test improves things then you can add the argument to the kernel cmdline permanently by using grubby. sudo grubby --update-kernel=ALL --args="plymouth.use-simpledrm=1" and remove the other option with sudo grubby --update-kernel=ALL --remove-args="rhgb"

That what I am thinking too.

That’s actually smart, thank you for the extra information.

Its configured with the default 5 seconds delay, I measured using the stopwatch from the time I pressed enter on one of the the kernels to the LUKS prompt appearing, and I can confirm the time systemd calculated is indeed accurate.

I actually don’t mind doing some digging and debugging for this problem, as I said, this isn’t a recent issue but one that has been with me for at least 3 years. Is there any option to enable verbose GRUB output during image loading, I really want to know what GRUB is scrapping on the SSD for 6 straight minutes because during those 6 mins the HDD LED on the tower stays on without a single flicker.

here is /etc/default/grub

❯ cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rd.luks.uuid=luks-98277076-da41-4d1c-8531-29749937c4a3 rhgb quiet rd.driver.blacklist=nouveau,nova_core modprobe.blacklist=nouveau,nova_core"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true

And I thank everyone again for your help.

remove rhgb (removes the splash screen) and quiet (gives you full logging on screen as the system boots.

Alternatively, just press escape after selecting a kernel to boot from and plymouth will vanish.

Sadly, that only displayed Linux logs, GRUB did not log anything that way.

Don’t think grub produces any logs at all, as until the filesystem is mounted it has nowhere to store anything.

See Re: GRUB2 Enable Debug Messages.

Maybe that works to enable grub logging. I haven’t tested it, but you may give it a try. Perhaps you my need to use a video recorder to save the messages.

I used debug=all and manually inspected a 14mins boot (probably slower because of all the paging), sadly I went over every single line and it was just detecting filesystem via brute-force, reading 0x40 sectors of some location, countless reading 0x40 sectors, allocating more ram, more reading and the cycle repeats, the same thing happened for initrd (with many many more reading ram extending cycles), then just setting attributes (?) for some ram locations and passing the baton to the Linux kernel.

No notable error or anything related to timeout surfaced. Just for comparison, I did the same on my laptop running the same flavor and version of Fedora which didn’t have such problems, and the output was the same.

One thing of note was that GRUB on the problematic PC, seemingly running on 1024x768 resolution, struggled a lot to even scroll the log through the pager compared the laptop, which reminds me of when I first experienced this problem, my monitor at the time booted in Full HD, which caused GRUB to function at 1fps or even lower, and I am not exaggerating. Only configuring UEFI to always boot at 768p did remedy the problem a little bit. It is still very slow compared to its function (GRUB should not struggle cycling a pager on a 2.8GHz CPU even if running only 1 core).

The comparison laptop is also a 6-core Ryzen 5500u so it shouldn’t outperform the i5-8400 on GRUB pager scrolling.