FCOS unusually slow on Raspberry Pi 3B

So I installed the UEFI firmware ( GitHub - pftf/RPi3: Raspberry Pi 3 UEFI Firmware Images ) to a MicroSD, used coreos-installer to install an aarch64 image to a USB flash drive… put that into a RPI3b… and it worked.

However, it’s strangely slow. The initial boot took about 30-60 minutes, mostly seemed taken by resizing the partitions in the USB drive. After that, I rebooted and I got to a login prompt in ~4 minutes, but it was pretty unresponsive (got tired of waiting for the password prompt).

This is a bit surprising because while it’s a random Kingston USB flash drive I got from Amazon, it’s a recent one and coreos-installer was reasonably fast writing to it (given how slow are USB flash drives).

Questions:

  1. Is there a better way to run FCOS on a Pi 3B? I just want to add an aarch64 node to a K8S cluster [to run CI builds for aarch64].

  2. Has anyone run FCOS on a Pi 3B and noticed such slowness? I’m beginning to think the USB flash drive is borked or somehow FCOS kernel is using the Pi’s USB ports as USB 1.0-mega-slow-edit-mode or something.

  3. Is there an easy way to get boot logs to see and spot something unusual? I took some photos of the boot process, but that’s not great. I wouldn’t also page up/page down through the boot log. I tried to plug the USB flash drive into my main laptop, but I only found an empty journal file and I couldn’t find logs anywhere.

Cheers,

Álex

2 Likes

Can this be related to Very(!) slowly booting Fedora CoreOS on intel i9 (Hetzner Box) - #4 by lucab / remove console=ttyS0 on metal · Issue #567 · coreos/fedora-coreos-tracker · GitHub ?

Well, --delete-karg ‘console=ttyS0,115200n8’ didn’t seem to change anything, so it’s likely not the serial console thing.

I haven’t tried to run FCOS on a Pi3. How much memory does the Pi3 have?

I would definitely try with a USB 3.0 HDD to see if that helps. Sometimes the I/O of the flash disks are really slow.

1 Like

It’s just 1gb, but I have VMs running FCOS and they sip memory.

As an experiment, I played with Fedora IOT and Fedora Workstation. Both gave me responsive tty prompts. Fedora Workstation didn’t try to resize the disk, but I could load Gnome (although it was very unresponsive), then switch to a tty and run commands normally.

The RPI3 doesn’t have USB 3, so I guess that doesn’t help- but really it seems that FCOS is especially slow for some reason. The disk doesn’t seem to be pathologically slow; I could dd the images at 11MB/s, which seems within what’s possible with USB 2… and that shouldn’t be that slow…

Thank you for doing those other benchmarks. It helps to narrow down the problem.

Perhaps this is related to the FCOS default to being more secure by setting the mitigations=auto,nosmt kernel argument. Could you try changing it to mitigations=off to see if that helps? See our documentation for how.

OK…

$ sudo coreos-installer install -f fedora-coreos-36.20220618.3.1-metal.aarch64.raw.xz --delete-karg=mitigations=auto,nosmt --append-karg=mitigations=off /dev/sdb
...
$ cat .../boot/loader.1/entries/ostree-1-fedora-coreos.conf 
...
options ignition.platform.id=metal $ignition_firstboot ostree=/ostree/boot.1/fedora-coreos/191b984648b4ecd074ebd8cad0f87ee0e7a24223bbcfa3b64c786b41989afbb4/0 mitigations=off

So I think that did “something”, because I think (but I didn’t record this) that previously it would take like 2-3 minutes to get to growfs, and now it took like 37s.

However, growfs is now at 10 minutes, and there’s been a kernel message “systemd-udev: sda2: Worker processing SEQNUM=2115 is taking a long time”.

So probably mitigations does have an effect, but growfs is still “slow”. But I don’t know, I don’t think I could boot Fedora Workstation (lagging, yes), and growing the partitions would be more painful…

I’m happy to try and collect more information- or even install some other OS to the MicroSD card and then run some benchmarks on my USB flash drive.

This may not be even an FCOS bug? It could be that the EFI boot thing is leaving the system in a bad state or something…

(small observation: so I thought I’d restart and also disable the console, as per the other thread, because who knows, maybe that’s also slowing me down… Well, now it took 95 seconds to start growfs, so this seems to be highly random [this suggests me that the flash drive performance is unstable]).

(also, it’s now 15 minutes inside growfs and there have been some messages about systemd-udevd killing some workers- but the growfs progress message is still updating…)

So last update for today :slight_smile:

I can do further testing if needed…

As FCOS next is now 37, I did some testing today- same results.

Saw this issue coreos-printk-quiet.service always fails after first boot. · Issue #1304 · coreos/fedora-coreos-tracker · GitHub where someone claimed to have FCOS running on a RPI3, asked for details there (trying now disabling SELinux and denylisting the VC4 module as that poster does, but without much hopes…).

edit: as expected SELinux/VC4 didn’t seem to make much of a difference.

Unfortunately I have also same problem with RPi3 + sdcard. But I thought coreos on sdcard not booting at all on my RPi3. I didn’t wait like you did. I just fed up and install it to USB with u-boot.

Just follow this official installation docs with u-boot but download bcm2835-firmware package too and put it to efi folder.

Soo this is how I did:

RELEASE=36 # The target Fedora Release. Use the same one that current FCOS is based on.
mkdir -p /tmp/RPi4boot/boot/efi/

# bcm2835-firmware package added for raspberry pi 3
# there is a missing package bcm2711-firmware on the official docs prevent me to boot without it.
# I open a PR for this package: https://github.com/coreos/fedora-coreos-docs/pull/457
sudo dnf install -y --downloadonly --release=$RELEASE --forcearch=aarch64 --destdir=/tmp/RPi4boot/  uboot-images-armv8 bcm283x-firmware bcm283x-overlays bcm2835-firmware bcm2711-firmware

for rpm in /tmp/RPi4boot/*rpm; do rpm2cpio $rpm | sudo cpio -idv -D /tmp/RPi4boot/; done
sudo mv /tmp/RPi4boot/usr/share/uboot/rpi_4/u-boot.bin /tmp/RPi4boot/boot/efi/rpi4-u-boot.bin
# this is for raspberry pi 3
sudo mv /tmp/RPi4boot/usr/share/uboot/rpi_3/u-boot.bin /tmp/RPi4boot/boot/efi/rpi3-u-boot.bin

# Now install it to USB drive with coreos-installer.
# I can not solve sdcard boot issue :( thats why use USB drive or disk.
# Change this command for your needs
FCOSDISK=/dev/sdX
sudo coreos-installer install --architecture=aarch64 -i config.ign $FCOSDISK

# Put u-boot files to efi partition
FCOSEFIPARTITION=$(lsblk $FCOSDISK -J -oLABEL,PATH  | jq -r '.blockdevices[] | select(.label == "EFI-SYSTEM")'.path)
mkdir /tmp/FCOSEFIpart
sudo mount $FCOSEFIPARTITION /tmp/FCOSEFIpart
sudo rsync -avh --ignore-existing /tmp/RPi4boot/boot/efi/ /tmp/FCOSEFIpart/
sudo umount $FCOSEFIPARTITION

That’s how I did it. After this you can use your USB disk or drive with both raspberry pi 3 & 4.

Great, thanks. I’m trying to encapsulate this in this script:

, but I’m not having a ton of success yet. I had to make some changes because I’m not running Fedora, so I had to use podman for some bits. Tried this with an SD card, but I get nothing on boot. Will continue working on that…

edit: wait! you’re using USB. What are you using to boot on the USB? GitHub - pftf/RPi3: Raspberry Pi 3 UEFI Firmware Images ?

edit 2: ah, maybe Booting Raspberry Pi 3 B With a USB Drive : 3 Steps - Instructables !

Yea I did that USB boot preparation for my RPi3 before. RPi3 need some love for USB boot.

I have also some difficulties about having required packages on Silverblue. This rpm download method is not very efficient for preparing coreos. They just documented for fedora user and if you are not a fedora user you need to use toolbox/distrobox/podman/docker solution for that. This need to change imo.

I hope they will prepare and release raspberry pi image.

About sdcard issue: I have no idea why my RPi3 not booting from sdcard.

From USB:

  • Pİ 3 boots OK
  • Pi 4 boots OK

From SDcard:

  • Pi 3 not booting
  • Pi 4 boots OK

I am trying to debug boot issues with uart cable but bootcode.bin stuck on the beginning faze. There is no output except Raspberry Pi Bootcode
Use this code to debug early boot: (you need uart cable)

  • sed -i -e "s/BOOT_UART=0/BOOT_UART=1/" bootcode.bin

I think there is an issue with coreos EFI partition on RPi3. Because Fedora IoT is using standart FAT-16 for EFI partition and It boots without any problem.

Well, I tried creating an USB drive from the script above and apparently it didn’t work. I came back from lunch, retried, and it booted (!).

But growfs is still terribly slow (5m now and running). So if it’s working well for you, then maybe it’s just that I picked a terrible USB flash drive :frowning:

edit: more updates, growfs eventually finished, apparently. But rebooting the Raspberry resulted in a borked system (systemd-fsck-root.service failed- apparently it couldn’t find a disk by uuid).