I have this same problem but even kernel 6.2.14-300.fc38.x86_64 won’t boot, neither rescue won’t work and rescue seems to be from fedora 34: time…
Rescue won’t work in this case means:
"Cannot open access to console, the root account is locked. See sulogin(8) man page for more details.
Press Enter to continue."
"Starting default.target
You are in emergency mode. After logging in, type “journalctl -xb” to view system logs, “systemctl default” or “exit” to boot into default mode.
Cannot open access to console, the root account is locked. See sulogin(8) man page for more details.
Press Enter to continue."
…
My machine is Thinkpad E14 Gen 2 Intel.
Could I somehow fix this issue by creating new fedora USB stick from another machine and using it?
This does not look like the same problem – others are able to boot 6.2 kernels. You need to start new thread with a title like “Fresh install of Fedora 38 Workstation FTB 6.2.14-300 on Thinkpad E14 Gen 2 Intel”, and provide enough detail to allow others to reproduce your problem.
Was F38 a fresh install or an upgrade? Is the system dual boot? Are yuo using BIOS or UEFI boot? Is your F38 the Gnome Workstation or something else? Can you boot the F38 Live ISO system (e.g, from a USB device)? If you can get a terminal or text console, please post the output of inxi -Fzx as searchable text so potential helpers can find it.
Try editing the kernel command line in grub2 (press “e” when you get the list of available kernels) to remove “rhdb quiet” so you can see kernel messages as the system boots. This often allows you to determine exactly why the system fails to boot.
I have far from fresh install. First install was Fedora 34 beta which has been updated to 38. Sure I can’t boot 6.2 kernel but issues started after update, this isn’t fresh install!
Also I have to write here using phone, cause can’t boot computer, so if I still start new thread even when only difference is that in my case even kernel 6.2 won’t boot, with same update got issues to system wide… then someone would just complain about not writing every little detail with my phone.
I wait a little if I get answer here, if not, I will make backups and switch back to debian based after these years of using fedora 34 beta-38. Several fedora updates have given problems to boot during this time, but with older kernel I was always able to boot until now.
@huuhaa : as @gnwiii has already noted, this is a different issue. This is why I have moved this into a separated topic.
This is default unless you have enabled the root account yourself. I am not sure if/how wheel members (users with sudo) can login here (but this would indeed be a good question on itself; I never had to manage that since I use root). However, if you are not able to login, check the next:
I think we should start with checking the logs.
Please boot with 6.3.6. I assume it enters emergency mode. Then reboot, boot with a working kernel, and then immediately get the output of sudo journalctl -r --boot=-1 (-1 outputs the then-last boot, which at this point should be the broken 6.3.6 boot with emergency mode).
Feel free to anonymize data you consider private (IP/MAC addresses, user names, …)
Just to be sure I understand your problem correct: you are still able to boot with 6.2.13, right? And all later kernels enter emergency mode including 6.3.6?
Some more details of what is when going on and what you did before the issue occurred first would be helpful, too.
None of kernel boots, all just show black screen, at left upper corner there’s
_
Also none of kernel boots to emergency mode, I can try select that at grub, but it won’t work either.
There is no way to enter any commands unless maybe with usb stick and chroot, but so far wasn’t able to mount any partition, I have btrfs filesystem and tried to follow this:
My device is shown as /dev/nvmeOn1 with biggest partition being /dev/nvmeOn1p3 but those can’t be mounted, no such device…
Anyway I haven’t done nothing yet by using chroot, but really don’t know any other way to continue. All kernels will just boot to black screen with top left showed
_
So to me this sounds same issue as the one I originally posted to. All started after update.
Only reason I described what emergency mode gives as output, was to show that I can’t use that either to fix. Can’t really understand how outcome could have been that every kernel boots to emergency mode.
Edit:
To make it more clear. There’s no place to enter single command, just black screen with
_
That’s why I have readed about chroot now, but won’t do anything before seeing replies here.
I thus assume some of the kernels in your grub list have worked before and now they work no longer. Thus I assume this is not a kernel issue. Something else has changed.
If you can boot other live systems but they are not able to mount existing file system(s), then there is an issue in the file system(s), and I would assume that one of the damaged file systems (or the one damaged file system) is your root partition.
I assume your boot partition is separated (which it is by default). This is why grub works (the kernels are also stored in the boot partition). So the issue is likely to appear when any kernel tries to mount the root file system, which is damaged. I still wonder a bit about the absence of a related error message, but if all kernels that are installed suddenly stop working, it cannot be a kernel issue.
Anyway, …
… given what you describe, I would first focus on the file systems.
I assume you know for sure you have btrfs as root partition?
Generally, you have to work with a live system obviously.
Before automatically assuming that this is intended to be the root partition: is that maybe free/non-allocated space, an extended partition or something else?
Can you boot with a live system and check parted /dev/nvme0n1 (be aware that this is likely the number 0 and not the letter O) and then enter print: what is the output? Just to have some rough overview of the table. Also, btrfs filesystem show
I have changed the title of the topic correspondingly to trigger btrfs and not kernel. I currently focus on the kernel topics. And I think we have people with more experience “off the cuff” in btrfs troubleshooting.
GNU Parted 3.5
Using /dev/nvme0n1
Welcome to GNU Parted! Type ‘help’ to view a list of commands.
(parted) print
Model: SAMSUNG MZALQ512HALU-000L1 (nvme)
Disk /dev/nvme0n1: 512GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 1049kB 630MB 629MB fat32 EFI System Partition boot, esp
2 630MB 1704MB 1074MB ext4
3 1704MB 512GB 510GB btrfs
And for this part… Yep I don’t remember selecting to update, but laprop was forgotten power on and without charger connected at evening. At morning when I noticed that lid wa open I went to check. Sure power was off, connecyed cable… At first syarted normally and started to install updates, was bit nervous what would happen case apparently power was turned off in the middle of updates… But since no error messages, thought I was lucky and everything went ok…But same black screen with _ mixed me to thing it was other issue…
Anyway I won’t blame fedora for that one, I just hope there’s still way to fix that issue… I have fixed manjaro in times using chroot but then I used ext4 and different machine… And one of my thoughts was Fedora could be nice semi rolling release. Well not rolling but packages quite fresh and hopefully more stable than those rolling realeases…
If grub works fine so that you can select all kernels, I assume number 1 and 2 work fine, and there is an issue with 3.
Please provide the …
… I asked for.
Beyond waiting for someone with hints, you can find with your preferred search engine much about btrfs repair that could be helpful. Also the related man pages (e.g., in any terminal of the live system man btrfs-check or man btrfs-recover) can be helpful.
I suggest to start with sudo btrfs-check --readonly <file system> (–readonly should be default anyway) while the paths can be seen in the btrfs filesystem show output (/dev/…).
Check all btrfs paths that are shown with btrfs filesystem show and provide both the btrfs filesystem show output and all btrfs-check outputs. Please provide all these outputs to foster support (and maybe also to make it a little palatable for our btrfs experts ).
Indeed something dangerous. However, I would first check the file systems anyway. Possibly the file system was corrupted if dnf was writing/working on it when powered off.
Label: ‘fedora_localhost-live’ uuid: 403402af-f52b-4935-8220-84f99a0ba6eb
Total devices 1 FS bytes used 419.26GiB
devid 1 size 475.35GiB used 429.02GiB path /dev/nvme0n1p3
btrfs check --readonly /dev/nvme0n1p3
Opening filesystem to check…
Checking filesystem on /dev/nvme0n1p3
UUID: 403402af-f52b-4935-8220-84f99a0ba6eb
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 450175451136 bytes used, no error found
total csum bytes: 389591156
total tree bytes: 2010431488
total fs tree bytes: 1393541120
total extent tree bytes: 150110208
btree space waste bytes: 388494002
file data blocks allocated: 559011770368
referenced 509933764608
Have you tried to mount this manually or in the live system’s GUI? If manually, the issue was likely the letter O, which should be the number 0.
In the live system: open the file manager and click on the left side on the volume so that the GUI mounts it. If it does not work, let us know the output/issue. Also, in this case try mount /dev/nvme0n1p3 /mnt (number 0, not letter O). The same: show us the output/issue if it doesn’t work.
If you get it mounted, please provide us the logs:
sudo chroot <path where the btrfs was mounted> (such as /mnt in the manually mounted case) and then sudo journalctl --boot=-1 and sudo journalctl --boot=0
Additionally, the dnf logs of the very day the issue happend, which can be found in ../var/log/dnf.log beginning at the very place the btrfs is mounted.
These are the three subvolumes in the btrfs. You have to add them (I forgot to mention that):
E.g., if manually, /mnt/root/… → the outcome for the dnf.log would be thus, e.g., /mnt/root/var/log/dnf.log ; for the chroot then /mnt/root/ (or the respective path set by the GUI, which should begin with /run/media/…
Well graphically it’s not possible to mount (only sees live stick), I umounted that previous I mounted manually but… gotta now read that fedora magazine article and your answer cause didn’t quite get it from your message what should manually do.
from article tested practical method for btrfs subvolumes…
mount /dev/nvme0n1p3 /mnt/newroot btrfs -o subvol=root
mount: bad usage
I’m wondering if there is any btrfs issue that the check doesn’t get. Maybe someone else has an idea. However, …
Not sure if I get your point, but it should also mount without separating subvolume.
You have to be more precise when you read/write commmands. You missed the -t option. If you want to try it the way from the article: sudo mount /dev/nvme0n1p3 /mnt -t btrfs -o subvol=root → formally, the options are set immediately after the “mount”, not at the end. Never tried if that works.
Also, if you want to use a subfolder, you have to create it first (I mean the “newroot” folder you tried to use).
Further, be careful with working with individual parts from articles without considering their context.
I assume I got it… At least it shows also yesterdays boottest with secureboot disabled, tried also that cause saw mentioned at some thread here or elsewhere…
Yhteenveto
Jun 09 23:21:08 fedora kernel: microcode: updated early: 0x66 → 0xaa, date = 2022-12-28
Jun 09 23:21:08 fedora kernel: Linux version 6.3.4-201.fc38.x86_64 (mockbuild@f05670b8769b4768acb2ba95ad39c164) (gcc (GCC) 13.1.1 20230511 (Red Hat 13.1.1-2), GNU ld version 2.39-9.fc38) #1 SMP PREEMPT_DYNAMIC Sat May 27 15:08:36 UTC 2023
Jun 09 23:21:08 fedora kernel: Command line: BOOT_IMAGE=(hd0,gpt2)/vmlinuz-6.3.4-201.fc38.x86_64 root=UUID=403402af-f52b-4935-8220-84f99a0ba6eb ro rootflags=subvol=root rhgb quiet
Jun 09 23:21:08 fedora kernel: x86/split lock detection: #AC: crashing the kernel on kernel split_locks and warning on user-space split_locks
Jun 09 23:21:08 fedora kernel: x86/fpu: Supporting XSAVE feature 0x001: ‘x87 floating point registers’
Jun 09 23:21:08 fedora kernel: x86/fpu: Supporting XSAVE feature 0x002: ‘SSE registers’
Jun 09 23:21:08 fedora kernel: x86/fpu: Supporting XSAVE feature 0x004: ‘AVX registers’
Jun 09 23:21:08 fedora kernel: x86/fpu: Supporting XSAVE feature 0x020: ‘AVX-512 opmask’
Jun 09 23:21:08 fedora kernel: x86/fpu: Supporting XSAVE feature 0x040: ‘AVX-512 Hi256’
Jun 09 23:21:08 fedora kernel: x86/fpu: Supporting XSAVE feature 0x080: ‘AVX-512 ZMM_Hi256’
Jun 09 23:21:08 fedora kernel: x86/fpu: Supporting XSAVE feature 0x200: ‘Protection Keys User registers’
Jun 09 23:21:08 fedora kernel: x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256
Jun 09 23:21:08 fedora kernel: x86/fpu: xstate_offset[5]: 832, xstate_sizes[5]: 64
Jun 09 23:21:08 fedora kernel: x86/fpu: xstate_offset[6]: 896, xstate_sizes[6]: 512
Jun 09 23:21:08 fedora kernel: x86/fpu: xstate_offset[7]: 1408, xstate_sizes[7]: 1024
Jun 09 23:21:08 fedora kernel: x86/fpu: xstate_offset[9]: 2432, xstate_sizes[9]: 8
Jun 09 23:21:08 fedora kernel: x86/fpu: Enabled xstate features 0x2e7, context size is 2440 bytes, using ‘compacted’ format.
Jun 09 23:21:08 fedora kernel: signal: max sigframe size: 3632
Jun 09 23:21:08 fedora kernel: BIOS-provided physical RAM map:
Jun 09 23:21:08 fedora kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009efff] usable
Jun 09 23:21:08 fedora kernel: BIOS-e820: [mem 0x000000000009f000-0x00000000000fffff] reserved
Jun 09 23:21:08 fedora kernel: BIOS-e820: [mem 0x0000000000100000-0x000000006da7dfff] usable
Jun 09 23:21:08 fedora kernel: BIOS-e820: [mem 0x000000006da7e000-0x0000000072f3bfff] reserved
Jun 09 23:21:08 fedora kernel: BIOS-e820: [mem 0x0000000072f3c000-0x0000000073b32fff] ACPI NVS
Jun 09 23:21:08 fedora kernel: BIOS-e820: [mem 0x0000000073b33000-0x0000000073bfefff] ACPI data
Jun 09 23:21:08 fedora kernel: BIOS-e820: [mem 0x0000000073bff000-0x0000000073bfffff] usable
Jun 09 23:21:08 fedora kernel: BIOS-e820: [mem 0x0000000073c00000-0x0000000077ffffff] reserved
Jun 09 23:21:08 fedora kernel: BIOS-e820: [mem 0x0000000078d00000-0x0000000078dfffff] reserved
Jun 09 23:21:08 fedora kernel: BIOS-e820: [mem 0x0000000079600000-0x00000000807fffff] reserved
Jun 09 23:21:08 fedora kernel: BIOS-e820: [mem 0x00000000fed20000-0x00000000fed7ffff] reserved
Jun 09 23:21:08 fedora kernel: BIOS-e820: [mem 0x0000000100000000-0x000000087f7fffff] usable
Jun 09 23:21:08 fedora kernel: NX (Execute Disable) protection: active
Jun 09 23:21:08 fedora kernel: efi: EFI v2.7 by Lenovo
Jun 09 23:21:08 fedora kernel: efi: ACPI=0x73bfe000 ACPI 2.0=0x73bfe014 SMBIOS=0x70374000 SMBIOS 3.0=0x70367000 TPMFinalLog=0x73994000 MEMATTR=0x6a862018 ESRT=0x6a505298 MOKvar=0x6e4dc000 RNG=0x73bfd018 TPMEventLog=0x543a4018
Jun 09 23:21:08 fedora kernel: random: crng init done
Jun 09 23:21:08 fedora kernel: secureboot: Secure boot disabled
Jun 09 23:21:08 fedora kernel: SMBIOS 3.2.0 present.
Jun 09 23:21:08 fedora kernel: DMI: LENOVO 20TA0028MX/20TA0028MX, BIOS R1EET32W(1.32 ) 01/05/2021
Jun 09 23:21:08 fedora kernel: tsc: Detected 2400.000 MHz processor
Jun 09 23:21:08 fedora kernel: tsc: Detected 2419.200 MHz TSC
Jun 09 23:21:08 fedora kernel: e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
Jun 09 23:21:08 fedora kernel: e820: remove [mem 0x000a0000-0x000fffff] usable
Jun 09 23:21:08 fedora kernel: last_pfn = 0x87f800 max_arch_pfn = 0x400000000
Jun 09 23:21:08 fedora kernel: x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT
Jun 09 23:21:08 fedora kernel: last_pfn = 0x73c00 max_arch_pfn = 0x400000000
Jun 09 23:21:08 fedora kernel: esrt: Reserving ESRT space from 0x000000006a505298 to 0x000000006a505348.
Jun 09 23:21:08 fedora kernel: e820: update [mem 0x6a505000-0x6a505fff] usable ==> reserved
Jun 09 23:21:08 fedora kernel: Using GB pages for direct mapping
Jun 09 23:21:08 fedora kernel: secureboot: Secure boot disabled
Jun 09 23:21:08 fedora kernel: RAMDISK: [mem 0x4fc9b000-0x52142fff]
Jun 09 23:21:08 fedora kernel: ACPI: Early table checksum verification disabled
Jun 09 23:21:08 fedora kernel: ACPI: RSDP 0x0000000073BFE014 000024 (v02 LENOVO)
Jun 09 23:21:08 fedora kernel: ACPI: XSDT 0x0000000073BFC188 00012C (v01 LENOVO TP-R1E 00001200 PTEC 00000002)
Jun 09 23:21:08 fedora kernel: ACPI: FACP 0x0000000070355000 000114 (v06 LENOVO TP-R1E 00001200 PTEC 00000002)
sudo journalctl --boot=0
Failed to get boot id: Function not implemented
Also, is the log in your post the whole output of journalctl --boot=-1? (is that the -k output?) If I got it right (so that this is not the boot of the issue but a later one), it seems root is able to indeed mount the root partition at startup. Is 6.3.4 the newest kernel you have installed?
It was all I could copy from terminal screen. I could make text file of those but… That wouln’t help anything… Since I opened dnf.log at gedit and copied everything from there as hidden block, but I’m not allowed to insert as long texts
Don’t mean to step on ones toes, but should not the mount command for a btrfs file system be trying to mount the subvolume and not the root volume?
i.e. mount -t btrfs -o subvol=root,compress=zstd:1 /dev/nvme0n1p3 /mnt
should work where mount -t btrfs /dev/nvme0n1p3 /mnt
should not.
Those 2 options (subvol=root,compress=zstd:1) are default in the /etd/fstab file for all fedora installs I have done since fedora chose to switch from ext4 to btrfs. Fedora uses 2 subvolumes, ‘root’ and ‘home’, by default with every workstation install I have done.
That’s indeed necessary, but I think mount can determine that one itself (not sure if mount determines it or if it just tries the default zstd if not set differently).
It’s definitely the best practice and if you make something permanent, you should do it that way. However, for temporary purposes like the one we have here, you can just mount the root volume. That should work as well. It then shows the subvolumes as folders within. But I could imagine that this gets deprecated at some point.
).