I have Fedora Workstation 38 installed on my ThinkPad E14 Gen 3 (Ryzen 7, 16GB RAM). Everything was working fine, until kernel version 6.3.8, which doesn’t boot. I tried installing the testing kernel 6.3.11 and the issue is still present.
I already used versionlock to keep the functioning kernel version, but that’s obviously not ideal, and since this is a work laptop, I really need to have this Linux install functional in the future.
I’m attaching a screenshot of where the boot freezes. I’ll be happy to provide any additional information.
Just to be sure I understand your situation: do earlier kernels (before 6.3.8) still boot properly without issues?
If so: please boot once with 6.3.8. I assume you will then experience the issue. Then, reset and boot the most recent kernel with which your system works. Then, go to the terminal and get the output of sudo journalctl -k --boot=-1 (-k means kernel messages, and --boot=-1 implies the logs of the then-last boot, which will then be the corrupted 6.3.8 boot; so, 0 is the current boot you are working in, and minus 1 is the one before the current one). Additionally, at this point, you might also get the output of sudo journalctl -k, which then gets the kernel messages of the then-current boot (so, 0), which will be a working one. Just to compare. You could repeat that process and replace 6.3.8 with 6.3.11 (but you do not need to get another log of the then-current working boot). Therefore, you will then have a log from a broken 6.3.8, from a broken 6.3.11 and from a working kernel of before 6.3.8. These files are major indicators for us.
As you suggested, I updated the BIOS, and as I was getting the logs for you, to my incredible surprise, 6.3.8 booted. I tried rebooting a few times, and it now consistently boots every time. I thought the issue was resolved completely, but 6.3.11 still shows the same issues.
What’s even stranger is that booting 6.3.11 doesn’t produce a log. Maybe it’s because the system can’t write anything to the drive?
Here’s a log of one of the successful 6.3.8 boots:
I’m also once again attaching a picture of my screen while booting 6.3.11 with the updated BIOS, as that seems to be the only thing I can provide.
Be aware that the 6.3.11 kernel is still in testing, and since we do not
yet know for sure if this kernel is stable, I suggest to remain on 6.3.8
anyway. It is of course worth to try if the next kernel already solves a
problem when you experience one. So it was a good idea from you to test
it. But for now, I suggest to remain with stable kernels if they work
out and not focus on testing kernels in here since these discussions
belong to bodhi. However, since the error still looks the same to what
you had on 6.3.8, let’s keep digging a bit deeper:
First of all, are you 100% sure that this issue did already occur on
6.3.8 and that you did not accidentally boot 6.3.11 when you had the
issue first? (Please excuse the question, but we need some security here
since this fact makes a large difference in analysis compared to that
the issue occurs only on 6.3.11)
If yes, have you done anything different or changed anything before
6.3.8 started to work again?
How have you installed 6.3.11?
Concerning your question, nothing is clear yet, but I expect that the
error occurs before the log is initialized, which means the error occurs
very early in the boot.
Yes, I’m 100% certain, as the only reason I installed 6.3.11 was that so I could test if it boots, because 6.3.8 didn’t.
I updated the BIOS. Not sure if I did anything else, I definitely updated packages with dnf a few times because I do that regularly, so it’s possible that that affected it?
I installed 6.3.11 by typing sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-2846d5650e.
Since 6.3.8 works and 6.3.11 is in testing, I’m thinking that it might be a bit too early to troubleshoot this, because the next kernel version might not have this issue. If you however need anything else from me, like logs or any additional information to diagnose this issue, I’ll be happy to provide it.
Concerning your error log, the journalctl does not get logs because the root file system could not be mounted (and this means that /var/log/ is not available so that journalctl cannot start logging). The issue seems to start with the kernel being not able to mount your file system from nvme0n1p6. The kernel seems to have a problem with what it gets in a very seldom way, but this is at least already a hint. With this in mind, please:
Try several times to boot with 6.3.8 and 6.3.11 and switch arbitrarily between the two kernels: Please check if the behavior is consistent, which means that 6.3.8 now works every time and 6.3.11 has the error every time.
Then, boot your system with a working kernel and get the output of cat /etc/fstab ? Feel free to randomize UUIDs if you consider them private or so.
The same for sudo cat /boot/loader/entries/*6.3.8* and sudo cat /boot/loader/entries/*6.3.11* .
Can you also check your hardware: get the output of sudo smartctl --scan and then do sudo smartctl -a <nvm device> once for each device output by the --scan where <nvm device> is the respective device as output by --scan (e.g., sudo smartctl -a /dev/nvm0). Just to reduce the possibility that this is a damaged hardware issue.
dnf could be linked if it had issues when creating the kernel-related files. This could be indeed a initramfs issue if dnf didn’t do its “job” for some reason. Please boot the working 6.3.8 and do sudo dnf remove kernel*6.3.11* and then do again sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-2846d5650e.
Please let us know the requested data and if any of these means make a difference in some way.
Generally, troubleshooting at this time with 6.3.11 is indeed too early for here. But I am worrying since the error log of 6.3.8 and 6.3.11 seems equal, and if you are sure that the first screenshot you provided is from 6.3.8, I think it is worth to already have a rough look if we can identify the issue/origin (and to what it is related). I reviewed the testings of 6.3.11 we have so far, and your issue is not yet documented there.
Supplement: 6.3.11 has been submitted for stable and will be in the stable update repos tomorrow or the day after.
My previous post remains valid, except the command sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-2846d5650e:
Instead, remove 6.3.11 as described with sudo dnf remove kernel*6.3.11* but then do not re-install it from updates-testing but instead with the normal updates once it appears there, which should be tomorrow or the day after → dnf update --refresh. This should normally not make a difference, but since an issue in creating the kernel files cannot be excluded at this point, I tend to consider each possibility.
Hey Christopher, sorry it took me so long to get back to you.
I uninstalled 6.3.11 and reinstalled it with sudo dnf update --refresh, and unfortunately, it still doesn’t boot. I tried booting the kernels repeatedly, switching between the two, and I can 100% confirm that 6.3.8 boots every time and 6.3.11 never does.
Hi, I also have thinkpad e14 (but g2) and have the same boot issue( I didn’t check your logs but 6.3.11 doesn’t boot in my computer). Today I boot 6.3.8, after check this post I updated fedora ( sudo dnf upgrade -refresh) than I tried to boot 6.3.11. Now everything works, I didn’t check which packages did update. However I noticed pulsefire related packages, I think 6.3.11 imcompatible with few packages. Anyways if you upgrade your system, It should be fixed. Have a great day.
Nothing of that indicates your issue unfortunately.
First of all, test 6.3.12 with sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-94ec7a8b52
If this does not work, then:
We have already builds of the future Linux kernel (6.4.X). This is not yet eligible for production. This kernel is thus not intended to become an update atm. But you can install it to try if 6.4.X solves your issue more reliably.
Only use this kernel to boot and to see if it works out without the issue. Do not use that kernel to work. So go back to the previous one even if it works. At the best, after booting once to see if it works, reboot with 6.3.8 (as far as I understand it, this is the newest one that works for you) and directly remove 6.4.2 again until it is added as official stable update.
@oschl Just to ensure I got it right, before the BIOS update, 6.3.8 did not work always, and immediately after the BIOS update, it did? So, there was no 6.3.8 boot that worked before the BIOS update, but all 6.3.8 boots did work after?
I see that both of you have dnf updates as one of the activities that occur when a change in working kernels happen. However, I do not think that dnf itself or non-kernel packages are the cause since the error occurs before the root file system is mounted (at least based upon the logs of Ondrej). Thus, the only thing dnf can have an impact on is creating the kernel-related files. So far it cannot be excluded that invoking dnf updates with kernel relations have an impact in triggering/(temporary-)resolving the issue for any of the kernels. The constant is at the moment some of your hardware / BIOS. However, an issue that was introduced in one of the 6.3.X kernels and that is sometimes triggered but sometimes not (on your hardware) is another possibility. My hope remains that 6.4.X solves the issue.
Btw, can you provide the output of cat /proc/sys/kernel/tainted when booting with a working kernel?
Here is my output of tainted file:
$ cat /proc/sys/kernel/tainted
0
I didn’t upgrade my computer to unstable kernels I always upgrade my fedora with sudo dnf upgrade --refresh, I don’t add testing parameters.
Today when I try to boot up 6.3.11, It failed. Then I switch 6.3.8 do some upgrades reboot it and againt I can boot up 6.3.11. So I think about I can boot 11 after rebooting 8, What is the glitch here power off and booting again vs rebooting Or starting 6.3.8 and switching(reboot or power down and start it again) 11. I tried these condition here is my results:
Powering off the machine versus rebooting it is not the issue here, cause when I try 6.3.11 boot reboot and power off to start 6.3.11 It fails. However If I start fedora with 6.3.8 than reboot it boot 11 it works now I’m writing this reply in 11. But there is a condition which is whole another possibility, rebooting machine to 6.3.11 after successfully booted 6.3.11, When I try this, I can boot it and logged in after that I see the desktop environment, and computer freeze. Have a great day.
P.S English is not my the first language, and I’m still learning it. If u can’t understand me please don’t hesitate to mention it. Then I try to explain it more clear and correct.
Hello guys, I tried something new since I use ventoy, I disabled secure boot. I enable it and when I try to boot 6.3.11 I saw a line booting kernel version line like output of uname -r. I didn’t use verbose mode so I don’t see lines when I boot my computer. Anyways It seems 6.3.11 works for now. ( I’m not an expert I do some kinda weird and not computer-scientific experiments so If it is not works for you I don’t know.)
I have a Lenovo Yoga 6 laptop with an AMD cpu/igpu that would boot then freeze after the 6.3.11 update. It was the first time I’d had a problem with a kernel update in quite some time.
I’m only popping into this thread just to say that I tried the quoted fix you suggesting to the OP. It worked like a charm for me.
Thanks to you and all the developers who dedicate your time to stay on top of matters such as this one when they arise!
Unfortunately, I cannot invest much time at the moment.
I think the possibility exists that the issue does not lie in the kernel but in your BIOS, given that the BIOS update itself caused a change in behavior. I suggest to also get in touch with Lenovo. I expect that the support will reject because Fedora is not officially supported on your machine (you can still try) but there is also a forum for Lenovo customers, which can be very helpful (I solved myself a BIOS issue there some time ago). Maybe someone there has experience with how to mitigate with your BIOS.
However, we don’t know for sure that the cause is in the BIOS:
I think you should file a bug: Log in to Red Hat Bugzilla → file against product “fedora” & component “kernel”. You will get a template in the description field. You should read that carefully and provide the requested information. I suggest to also add the information of the different outputs from here (use files as attachments). If you cannot provide a log from a failed boot, provide one from a working boot but make clear why you do that. Of course the screenshots are important. Detail your issue with all information. Also make clear that 6.4.2 does not solve the issue, too. The more details, the more it is likely that they can help you. But try to be brief because the developers there have much to do.
Once submitted, please add the link to the bugzilla report here in order to link the two tickets. You can also add the link to here in the report.
Keep watching both tickets and provide information if requested.
In the meantime, you can also search on your preferred search engine for issues with your BIOS version but also the errors in the screenshots (start with the lines that contain VFS). Maybe you will find something that helps.
6.3.8 is the latest kernel I can boot.