Emergency mode after hard shutdown (Tree root error)

Ask Fedora
, , , , ,
21

The mount command has two forms – a one argument form that looks up the mount options in /etc/fstab and a two (or more) argument form where you provide all the details. Since you are booted to a LiveOS, it is no surprise that the partitions of your local disk are not listed in /etc/fstab. You’ll need to figure out what parameters should be passed. At a minimum, it needs to know where to mount the filesystem. Linux has traditionally provided the directory /mnt as a place for users to mount filesystems temporarily. For example mount /dev/sda3 /mnt.

P.S. The command to undo the mount would be umount /mnt (yes, there is no n :person_shrugging:). Also, if you really want to dig into the details of how these commands work, they have man pages (e.g., man mount). But be ready for some information overload on that one. :slightly_smiling_face:

P.P.S. There is also a zero argument form for the mount command – it lists the currently mounted filesystems and what options they were mounted with.

22

I tried it and I think we’re getting somewhere. "Source write-protected, mount read-only. Ill try and find out how to read it.

1 Like
23

FWIW, someone just pointed out in one of the Fedora Matrix rooms that there is a known Btrfs bug in recent Linux kernels that can cause what you are seeing.

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=975f3b6da18020f1c8a7667ccb08fa542928ec03

So if you get your filesystem repaired, you might want to run an older kernel until the 6.10 kernel comes out.

Edit: I’ve been informed that the above bug is not supposed to be present in the stable kernels that end users are running. So maybe you don’t have/need to run an older kernel afterall.

24

I need to see logs somehow. The screenshots don’t really give me enough information, the real problem is almost certainly in the back scroll. I realize this can be difficult to do if the system won’t boot.

How was the Live USB created? If you use Fedora’s Media Writer tool, it’ll just write the Fedora image directly to the USB stick - and the behavior is non-persistent. So it shouldn’t ever fill up. There is a volatile write layer setup during boot, but therefore when you reboot, everything is lost. Nothing is written to the USB stick. Some other utilities might try to setup persistence and that could make it blow up eventually if it fills up with updates or files.

Anyway, if you get a Live USB booting again, try to mount the Btrfs file system normally. Use lsblk to find the largest partition, that’s also Btrfs.

mount /dev/sda3 /mnt

If this fails with an error, use journalctl -k -o short-monotonic > mountfail_dmesg.txt and post the file somewhere so we can see what the issue might be.

If it mounts OK then we actually need to redirect journalctl to look at the journal files on your drive instead of the currently running LiveOS. Something like

journalctl -D $pathtojournals -k -b -o short-monotonic > boot0.txt
journalctl -D $pathtojournals -k -b-1 -o short-monotonic > boot1.txt

-b is the most recent, -b-1 means the boot before that, and -b-2 would mean two boots before.

$pathtojournals starts at your mountpoint for root, then to the “root” subvolume since this is on btrfs, and then to where systemd stores journals, something like:

/mnt/root/var/log/journal/$machineid

$machineid - there will only be one directory in journal/ so if you just TAB after the /, the shell will autocomplete the name of the directory. Journals are in there. So the -D option just tells journalctl to use this location for commands instead of the location for the LiveOS you’re running.

Anyway, once I see some logs I’ll have a better idea what the problem is.

3 Likes
25

Some error logs from dracut:

IMG_20240713_093955624
IMG_20240713_0939556241920×2560 354 KB

IMG_20240713_093950898
IMG_20240713_0939508981920×2560 347 KB

IMG_20240713_093939912
IMG_20240713_0939399121920×2560 532 KB

26

IMG_20240713_093945834
IMG_20240713_0939458341920×2560 226 KB

IMG_20240713_093939912
IMG_20240713_0939399121920×2560 532 KB

IMG_20240713_093929116
IMG_20240713_0939291161920×2560 284 KB

27

IMG_20240713_093921403
IMG_20240713_0939214031920×2560 555 KB

IMG_20240713_093903146
IMG_20240713_0939031461920×2560 749 KB

IMG_20240713_093744644
IMG_20240713_0937446441920×2560 529 KB

28

IMG_20240713_093738919
IMG_20240713_0937389191920×2560 665 KB

IMG_20240713_093733259
IMG_20240713_0937332591920×2560 681 KB

IMG_20240713_093724074
IMG_20240713_0937240741920×2560 772 KB

29

IMG_20240713_093903146
IMG_20240713_0939031461920×2560 749 KB

IMG_20240713_093744644
IMG_20240713_0937446441920×2560 529 KB

I might have reposted a few. Just wanted to make sure I sent all the error logs. I also have the non-error logs if that provides more info.

30

@oxidize: Those screenshots are very hard to read. Does dracut not give you an option to sign in to emergency mode at the end? If it does, then you should be able to connect a USB thumb drive and copy the logs to that. Then use another computer (or maybe your smartphone/tablet) to upload the log files to somewhere like https://paste.centos.org. Then you can share a link to them here so we can access the “text” version of the logs for easier reading and searching.

TIP: If your secondary computer is running Fedora Linux, you can use fpaste <filename> to upload your log file to that website for sharing. (Or you could upload the log files from your LiveOS session.)

31

in the 3rd screenshot i see
nvme0: Device not ready...

And then a bunch of abort messages also by the nvme driver, And then
nvme0: Disabling device after reset failure

And then see the btrfs errors, all of which are write errors, which makes sense because btrfs wants to write, but can’t, because the device is disabled by the nvme driver.

I can’t tell why the nvme driver is having trouble with this drive. There should be much earlier nvme messages that we’re missing that could provide a hint at the trouble.

Thing is, you got Fedora installed to this NVMe drive. So at one time it was cooperative. But now it isn’t. And I’m not sure what changed. Maybe it’s dying?

It would be helpful to have you boot the Fedora installer USB, and get a clean copy of the boot log, see if there’s any nvme messages:

journalctl -k -o short-monotonic > dmesg.txt

And post that somewhere. This will be very tedious for you and us to read by cell phone photo so I recommend doing it in Terminal per the above command and posting it somewhere.

sudo nvme id-ctrl /dev/nvme0

This would also be helpful to have copy/pasted into a file and shared somewhere.

32

Another screenshot shows some nvme i/o errors with opcode 0x1 qid 3. A google search turns up only three results that don’t help understand what’s going on. Therefore I think it’s unlikely to be a bug because bugs tend to generate lots of search results. Plus you got Fedora installed and it was working OK until now.

Maybe the laptop took a tumble recently? It might be worth opening up the laptop and reseating the NVMe drive in the off chance it became loose?

1 Like
33

I don’t recall dracut ever giving a sign in option. I’m also not sure how I could transfer any data from dracut to a live USB or external device. I turn on my device, it gives 3 versions of fedora l, an edit mode, and access to a command line/grub. All 3 fedoras take 5 minutes to load them send me to dracut. The only thing I know how to do in dracut is type reboot which allows me to access the live fedora is from my usb. I don’t know where along this process id be able to sign in to emergency mode or transfer logs from dracut.

34

Uhhhh, I can try that. Never taken apart a computer before but as long as it doesn’t require tools I might not mess things up. Also, if there is a way for me to just replace my current root system with the live os that would be nice. Thanks for helping me though. As much as I’m trying to get this fixed, I have no idea what I’m doing.

35

Maybe you don’t have to sign in. It may have changed since I last messed with it. If you are at a prompt where you can enter “reboot”, then you should be where you would need to be to mount a USB drive and copy the logs to it. Instead of “reboot”, enter ls /dev/sd*[1] to see what drives are present/detected on your system. Then connect your USB device. Then re-run ls /dev/sd* to see what has appeared that wasn’t there before. The new entry will be your USB device which you can mount somewhere an save the logs to it.[2] Once that is done, then you can reboot back to the LiveOS to upload the log files somewhere.

  1. There may be better ways to see what drives are available on your system such as lsblk or dmesg, but I’m not sure what all is available in Dracut. ↩︎

  2. Or, if you don’t have a spare USB device available, you might save the log files to your ESP. The downside of the ESP is that it isn’t very large, but it should have more than enough free space to save off a copy of your boot log. ↩︎

36

Dracut: “no such file or directory”. Lsblk does not work in dracut. Dmesg works but I’m not sure how to scroll to see logs, or decipher what the logs mean. At the end of the logs from dmesg it says “maybe the USB cable is bad?”

IMG_20240716_122713049
IMG_20240716_1227130491920×2560 792 KB

37

I tested with and without drive, sda, sd*, and sda1-3. All have the same output

38

If I’m reading your screenshot correctly, it appears that your USB device is connecting and disconnecting repeatedly as if the connection is poor. Maybe your PC’s I/O controller is fried? I don’t know enough about the hardware to say what might cause that sort of behavior, but my guess at this point would be that something is very wrong with your hardware. It probably isn’t something we can help you with short of recommending that you replace the defective hardware.

39

Dang. I think I’ll have to get used to running fedora from a liveusb for a while. I’m broke and have 0 hardware knowledge.

40

Like I said, I don’t know enough about the hardware to say for sure what is wrong. Usually hardware failures are more of an “all or nothing” type deal (it either works or it doesn’t). If you could determine what exactly is wrong, there might be a way to fix it (e.g., a loose wire that could be reconnected or a bad battery that could be replaced). Sorry.