However, it’s strangely slow. The initial boot took about 30-60 minutes, mostly seemed taken by resizing the partitions in the USB drive. After that, I rebooted and I got to a login prompt in ~4 minutes, but it was pretty unresponsive (got tired of waiting for the password prompt).
This is a bit surprising because while it’s a random Kingston USB flash drive I got from Amazon, it’s a recent one and coreos-installer was reasonably fast writing to it (given how slow are USB flash drives).
Questions:
Is there a better way to run FCOS on a Pi 3B? I just want to add an aarch64 node to a K8S cluster [to run CI builds for aarch64].
Has anyone run FCOS on a Pi 3B and noticed such slowness? I’m beginning to think the USB flash drive is borked or somehow FCOS kernel is using the Pi’s USB ports as USB 1.0-mega-slow-edit-mode or something.
Is there an easy way to get boot logs to see and spot something unusual? I took some photos of the boot process, but that’s not great. I wouldn’t also page up/page down through the boot log. I tried to plug the USB flash drive into my main laptop, but I only found an empty journal file and I couldn’t find logs anywhere.
It’s just 1gb, but I have VMs running FCOS and they sip memory.
As an experiment, I played with Fedora IOT and Fedora Workstation. Both gave me responsive tty prompts. Fedora Workstation didn’t try to resize the disk, but I could load Gnome (although it was very unresponsive), then switch to a tty and run commands normally.
The RPI3 doesn’t have USB 3, so I guess that doesn’t help- but really it seems that FCOS is especially slow for some reason. The disk doesn’t seem to be pathologically slow; I could dd the images at 11MB/s, which seems within what’s possible with USB 2… and that shouldn’t be that slow…
Thank you for doing those other benchmarks. It helps to narrow down the problem.
Perhaps this is related to the FCOS default to being more secure by setting the mitigations=auto,nosmt kernel argument. Could you try changing it to mitigations=off to see if that helps? See our documentation for how.
So I think that did “something”, because I think (but I didn’t record this) that previously it would take like 2-3 minutes to get to growfs, and now it took like 37s.
However, growfs is now at 10 minutes, and there’s been a kernel message “systemd-udev: sda2: Worker processing SEQNUM=2115 is taking a long time”.
So probably mitigations does have an effect, but growfs is still “slow”. But I don’t know, I don’t think I could boot Fedora Workstation (lagging, yes), and growing the partitions would be more painful…
I’m happy to try and collect more information- or even install some other OS to the MicroSD card and then run some benchmarks on my USB flash drive.
This may not be even an FCOS bug? It could be that the EFI boot thing is leaving the system in a bad state or something…
(small observation: so I thought I’d restart and also disable the console, as per the other thread, because who knows, maybe that’s also slowing me down… Well, now it took 95 seconds to start growfs, so this seems to be highly random [this suggests me that the flash drive performance is unstable]).
(also, it’s now 15 minutes inside growfs and there have been some messages about systemd-udevd killing some workers- but the growfs progress message is still updating…)
Unfortunately I have also same problem with RPi3 + sdcard. But I thought coreos on sdcard not booting at all on my RPi3. I didn’t wait like you did. I just fed up and install it to USB with u-boot.
RELEASE=36 # The target Fedora Release. Use the same one that current FCOS is based on.
mkdir -p /tmp/RPi4boot/boot/efi/
# bcm2835-firmware package added for raspberry pi 3
# there is a missing package bcm2711-firmware on the official docs prevent me to boot without it.
# I open a PR for this package: https://github.com/coreos/fedora-coreos-docs/pull/457
sudo dnf install -y --downloadonly --release=$RELEASE --forcearch=aarch64 --destdir=/tmp/RPi4boot/ uboot-images-armv8 bcm283x-firmware bcm283x-overlays bcm2835-firmware bcm2711-firmware
for rpm in /tmp/RPi4boot/*rpm; do rpm2cpio $rpm | sudo cpio -idv -D /tmp/RPi4boot/; done
sudo mv /tmp/RPi4boot/usr/share/uboot/rpi_4/u-boot.bin /tmp/RPi4boot/boot/efi/rpi4-u-boot.bin
# this is for raspberry pi 3
sudo mv /tmp/RPi4boot/usr/share/uboot/rpi_3/u-boot.bin /tmp/RPi4boot/boot/efi/rpi3-u-boot.bin
# Now install it to USB drive with coreos-installer.
# I can not solve sdcard boot issue :( thats why use USB drive or disk.
# Change this command for your needs
FCOSDISK=/dev/sdX
sudo coreos-installer install --architecture=aarch64 -i config.ign $FCOSDISK
# Put u-boot files to efi partition
FCOSEFIPARTITION=$(lsblk $FCOSDISK -J -oLABEL,PATH | jq -r '.blockdevices[] | select(.label == "EFI-SYSTEM")'.path)
mkdir /tmp/FCOSEFIpart
sudo mount $FCOSEFIPARTITION /tmp/FCOSEFIpart
sudo rsync -avh --ignore-existing /tmp/RPi4boot/boot/efi/ /tmp/FCOSEFIpart/
sudo umount $FCOSEFIPARTITION
That’s how I did it. After this you can use your USB disk or drive with both raspberry pi 3 & 4.
Great, thanks. I’m trying to encapsulate this in this script:
, but I’m not having a ton of success yet. I had to make some changes because I’m not running Fedora, so I had to use podman for some bits. Tried this with an SD card, but I get nothing on boot. Will continue working on that…
Yea I did that USB boot preparation for my RPi3 before. RPi3 need some love for USB boot.
I have also some difficulties about having required packages on Silverblue. This rpm download method is not very efficient for preparing coreos. They just documented for fedora user and if you are not a fedora user you need to use toolbox/distrobox/podman/docker solution for that. This need to change imo.
I hope they will prepare and release raspberry pi image.
About sdcard issue: I have no idea why my RPi3 not booting from sdcard.
From USB:
Pİ 3 boots OK
Pi 4 boots OK
From SDcard:
Pi 3 not booting
Pi 4 boots OK
I am trying to debug boot issues with uart cable but bootcode.bin stuck on the beginning faze. There is no output except Raspberry Pi Bootcode
Use this code to debug early boot: (you need uart cable)
sed -i -e "s/BOOT_UART=0/BOOT_UART=1/" bootcode.bin
I think there is an issue with coreos EFI partition on RPi3. Because Fedora IoT is using standart FAT-16 for EFI partition and It boots without any problem.
Well, I tried creating an USB drive from the script above and apparently it didn’t work. I came back from lunch, retried, and it booted (!).
But growfs is still terribly slow (5m now and running). So if it’s working well for you, then maybe it’s just that I picked a terrible USB flash drive
edit: more updates, growfs eventually finished, apparently. But rebooting the Raspberry resulted in a borked system (systemd-fsck-root.service failed- apparently it couldn’t find a disk by uuid).
To close this thread- I got a cheap USB SSD, installed FCOS to it, used the MicroSD card to chainboot using the UEFI firmware (for some reason, I didn’t get the USB to boot directly, whereas I managed to do it previously)… and it worked pretty well. Growing the filesystems was quick.
So yeah, don’t get a USB flash drive to use as your FCOS boot drive, I guess. A cheap USB SSD works much better.
