All Fedora versions fail to install

submariner · October 12, 2023, 11:08pm

Hi, I have tried to install various versions of Fedora (20,21, 34,35,36,37,38). I am using the Fedora Writer to download and write/check to the media (USB).
When attempting to ‘Install to Hard Drive’ (desktop gui), it either will ‘spin’ and then go away, or fail altogether.
On this latest, I ran from the terminal, with -d option. The results are similar to prior attempts. There are errors with specific scripts, e.g.

File “/usr/lib/python3.11/site-packages/blivet/populator/populator.py”, line 471, in _populate
new_devices = udev.get_devices()
I can’t upload logs, so this is the debug from liveinst command:
File “/usr/bin/anaconda-cleanup”, line 82, in
devicetree.populate(cleanup_only=True)
File “/usr/lib/python3.11/site-packages/blivet/threads.py”, line 53, in run_with_lock
return m(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/blivet/populator/populator.py”, line 446, in populate
self._populate()
File “/usr/lib/python3.11/site-packages/blivet/threads.py”, line 53, in run_with_lock
return m(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/blivet/populator/populator.py”, line 471, in _populate
new_devices = udev.get_devices()
^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/blivet/udev.py”, line 105, in get_devices
dev = device_to_dict(device)
^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/blivet/udev.py”, line 77, in device_to_dict
result = dict(device.properties)
^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/pyudev/device/_device.py”, line 1101, in getitem
return ensure_unicode_string(value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/pyudev/_util.py”, line 56, in ensure_unicode_string
value = value.decode(sys.getfilesystemencoding())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0xdb in position 1: unexpected end of data
Starting installer, one moment…
anaconda 38.23.4-2.fc38 for Fedora 38 started.

installation log files are stored in /tmp during the installation
shell is available on TTY2 and in second TMUX pane (ctrl+b, then press 2)
when reporting a bug add logs from /tmp as separate text/plain attachments

(anaconda:3878): Gtk-WARNING **: 18:26:29.157: Theme parsing error: gtk.css:1649:16: ‘-gtk-icon-size’ is not a valid property name

(anaconda:3878): Gtk-WARNING **: 18:26:29.157: Theme parsing error: gtk.css:1652:16: ‘-gtk-icon-size’ is not a valid property name
Traceback (most recent call last):
File “/sbin/anaconda”, line 478, in
ignore_oemdrv_disks()
File “/usr/lib64/python3.11/site-packages/pyanaconda/ui/lib/storage.py”, line 415, in ignore_oemdrv_disks
matched = device_matches(“LABEL=OEMDRV”, disks_only=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib64/python3.11/site-packages/pyanaconda/core/storage.py”, line 180, in device_matches
single_spec_matches = udev.resolve_glob(full_spec)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/blivet/udev.py”, line 192, in resolve_glob
for dev in get_devices():
^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/blivet/udev.py”, line 105, in get_devices
dev = device_to_dict(device)
^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/blivet/udev.py”, line 77, in device_to_dict
result = dict(device.properties)
^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/pyudev/device/_device.py”, line 1101, in getitem
return ensure_unicode_string(value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/site-packages/pyudev/_util.py”, line 56, in ensure_unicode_string
value = value.decode(sys.getfilesystemencoding())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0xdb in position 1: unexpected end of data
This machine is an MSI MaxX570 Tomahawk WiFI, with AMD Ryzen 5700 G, 64 GB Ram.

chrisawi · October 13, 2023, 1:36am

github.com/pyudev/pyudev

UnicodeDecodeError when reading GPT PartitionName

opened 08:10PM - 26 Apr 23 UTC

ogayot

Hello, In Subiquity (the Ubuntu Server installer), we are using pyudev to list …and retrieve information about the different devices that we can install on. We are iterating over all the properties of the block devices and exporting the data as JSON. Some users set a name to their GPT partitions (by setting the PartitionName field) and sometimes include accentuated characters in the name (or other special characters). My understanding of the PartitionName field is that it is UTF-16 encoded. When iterating over the properties of the block devices in pyudev (and specifically the PARTNAME property), accents or special characters cause pyudev to raise a UnicodeDecodeError exception: ```python import pyudev context = pyudev.Context() for device in context.list_devices(subsystem='block'): if 'PARTNAME' in device.properties: print(device.properties['PARTNAME']) ``` ``` Traceback (most recent call last): File "/home/olivier/file.py", line 6, in <module> if 'PARTNAME' in device.properties: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "<frozen _collections_abc>", line 780, in __contains__ File "/usr/lib/python3/dist-packages/pyudev/device/_device.py", line 1101, in __getitem__ return ensure_unicode_string(value) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pyudev/_util.py", line 56, in ensure_unicode_string value = value.decode(sys.getfilesystemencoding()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 4: invalid continuation byte ``` I have a partition where the name is set to "Données" (i.e., Data in French) For some reason, the PARTNAME property looks broken (i.e., it is not valid UTF-8 nor UTF-16). It is as if the MSB from all the UTF-16 characters got dropped: * letter A in UTF-16 is 0x0041. It becomes 0x41 (which is letter A is UTF-8 so it works) * e-ecute (é) in UTF-16 is 0x00E9. It becomes 0xE9 * euro sign (€) in UTF-16 is 0x20AC. It becomes 0xAC * pound sign (£) in UTF-16 is 0x00A3. It becomes 0xA3 Below are the relevant properties as shown by `udevadm info --export-db | less`: ``` E: PARTNAME=Donn<E9>es E: ID_PART_ENTRY_NAME=Données E: DEVLINKS=/dev/disk/by-partlabel/Données /dev/disk/by-partuuid/4f685f6a-606d-b845-8596-562dab728429 ``` The other properties (e.g., ID_PART_ENTRY_NAME and DEVLINKS) look okay, but PARTNAME does not. At this point, I'm not sure if this is purely a problem in pyudev. Maybe in udev or in lower levels? That said, I wonder if there is something we can do at pyudev level to, at least, work around the issue. Is there a known way to avoid this issue when iterating over the properties of devices? If not, I was thinking of adding a way to configure what `errors` parameter would be passed to [bytes.decode](https://docs.python.org/3/library/stdtypes.html#bytes.decode) in `ensure_unicode_string` when accessing a property of a Device. Maybe at the device level or at the context level. Does it sound like a viable solution? ### Steps to reproduce ```sh # Create a 100MiB file under /tmp dd if=/dev/zero of=/tmp/test.img bs=1M count=100 # Create a GPT partition table with one partition named "Données" (Data in French). # This sets the GPT PartitionName field. sfdisk /tmp/test.img <<EOF label: gpt label-id: CAEC69BF-F7D7-5F4E-9657-2A400D14F09A device: test.img unit: sectors first-lba: 2048 last-lba: 204766 sector-size: 512 test.img1 : start= 2048, size= 200704, type=0FC63DAF-8483-4772-8E79-3D69D8477DE4, uuid=4F685F6A-606D-B845-8596-562DAB728429, name="Donn\xc3\xa9es" EOF # Make this partition table visible system-wide by binding it to a /dev/loop device. sudo losetup -fP /tmp/test.img ``` Next, run the following code: ```python import pyudev context = pyudev.Context() for device in context.list_devices(subsystem='block'): if 'PARTNAME' in device.properties: print(device.properties['PARTNAME']) ``` output when I run it: ``` Microsoft reserved partition Traceback (most recent call last): File "/home/olivier/file.py", line 6, in <module> if 'PARTNAME' in device.properties: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "<frozen _collections_abc>", line 780, in __contains__ File "/usr/lib/python3/dist-packages/pyudev/device/_device.py", line 1101, in __getitem__ return ensure_unicode_string(value) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pyudev/_util.py", line 56, in ensure_unicode_string value = value.decode(sys.getfilesystemencoding()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 4: invalid continuation byte ``` Thank you! Olivier

You have a device property containing invalid UTF-8, which python infamously doesn’t tolerate well. From the comments on that issue, this may ultimately be caused by a kernel bug.

Can you try running this python code, tweaked slightly from the snippet in the issue?

#!/usr/bin/python3

import pyudev

context = pyudev.Context()

try:
    for device in context.list_devices(subsystem='block'):
        if 'PARTNAME' in device.properties:
            print(device.properties['PARTNAME'])

except UnicodeDecodeError:
    print(device)

This should help identify the device triggering the bug.

submariner · October 14, 2023, 6:05pm

Chris,

first, apologies for the delay, but work…

second, this worked to resolve. I don’t think it’s a kernel bug per se. The Windows drive has a 16 MB parition with no identifiable file system. I’m not sure if it’s needed by MS, but for now, I’ll let it be. The label/name was definitiely not UTF. I cleared that. After that Fedora installed without a hitch.

Perhaps, as an ?enhancement, if there could be, either (a) ignore it, or (b) offer to replace the value for the user, after informing the user what the issue is/appears to be. This would have been less frustrating if I had been smart enough, or if the error was not hidden down in the logs. Granted, there could have been other reasons for a failure, but this seemed to be the only one, in my case.

Much, much thanks.

John

chrisawi · October 14, 2023, 6:31pm

GPT partition names are encoded in UTF-16LE. The kernel bug is that it tries to convert that to ASCII, but improperly passes through bytes >127, inevitably making invalid UTF-8 multi-byte sequences: [PATCH v2 0/1] block: fix conversion of GPT partition name to 7-bit

I think it would be worth filing a bug for this in RHBZ. (I guess against the kernel? or maybe Anaconda?)

kparal · October 16, 2023, 10:30am

The upstream issue is already tracked in pyudev, as you already linked above. I don’t think we need a bug downstream as well, it won’t speed it up.

Topic		Replies	Views
Fedora not installing Project Discussion workstation-wg	0	642	May 1, 2020
Fedora 38 installation fails, "Failed to resolve the Live OS image." Ask Fedora f35 , f38 , kde , installation	1	1186	February 16, 2024
Fedora 40 installation doesn't start Ask Fedora installation , f40	4	419	August 17, 2024
Fedora 38 W/s fails to launch from liveCD - script errors Ask Fedora f38 , workstation , optical-media	7	481	July 26, 2023
The Live CD works; Anaconda doesn't Ask Fedora installation , intel , f39	4	937	February 23, 2024

All Fedora versions fail to install

Related topics