So I want to run a cluster using bare k8s on a bunch of FCOS machines, basically to set up a personal CI infra. Basically, I’m using GH runners for other stuff, but they don’t have aarch64 runners, which I need (and emulation slows down my builds by x8). I could do with on-demand nodes on clouds (I don’t run so many aarch64 builds, or I could use Oracle’s free tier), but I thought I’d experiment and learn FCOS.
I assume FCOS devs test on RPI4s, but a) they are a bit hard to get hold off nowadays b) they don’t have realtime clocks c) storage is a bit of a headache- I’d rather get something with some builtin storage.
Is there any other good aarch64 alternative, I’m looking for:
Something that I can buy easily, preferrably from the EU
I think I can do with 2gb of RAM
FCOS is hassle-free
Hopefully it has some decent storage out of the box (eMMC would be fine, I guess?), so I don’t have to buy an expensive SSD, hunt for a USB flash drive that performs well, or buy spinning rust
So what is everyone using on their labs for ARM? Is the only viable option an RPI4 and hoping you buy the right USB flash model? (it seems to me that even if you see good benchmarks for a specific USB flash drive, you could “accidentally” buy a variation that doesn’t perform well).
In my experience, most of these little Arm devices are great if you like to tinker with getting little Arm devices going, and not so much if you just want it to do something. It’s my understanding that the next-gen Jetson Nano devices should Just Work — but those aren’t out until January 2023.
Apparently PI 4 supplies might improve “real soon now”, so if I can get one for a non-inflated price, I’m quite happy. I was just wondering if everyone else was using some specific SBC (Orange Pi? Odroids? …) that made things a little bit better
Oh, I want ARM (although I wouldn’t mind switching everything to RiscV, but I think that’s a bit too bleeding edge?). The HoneyCombs are nice, but starting at $800? The Khadas is much cheaper, but still a bit steep for hobbyist use.
(The RPIs create this “dissonance”. They’re so cheap they make reasonably priced stuff look expensive…)
Too true, but when you compare with the Khadas SBC’s say, there is so much more capability, especially with storage, and they offer the V4 with the GPIO accessible for hacking. But you’re correct when you point out the price difference, for hobbies it is much more appealing. As well, there is also much more ootb available with the plethora of shields.
I believe it is likely that FCOS will run just fine on a “random ARM64” box once the booting has been set up. I base this guesstimate on the fact that Fedora 36,37 can run (virtually) flawlessly on an Amlogic s922x TV-box such as the Ugoos Am6 plus or the Beelink GT king pro:
You can boot Fedora using e.g. a small “sandwitch” step between the native u-boot on the box and the UEFI+grub setup on the image, implemented via Petitboot or u-root, see https://gitlab.com/tripole-inc/bootdisk (tested versions are the raw images for Workstation and KDE plasma from alt.fedoraproject.org).
So, my feeling is that as long as booting can be set up, the OS itself should run fine. (Analogous conclusions hold for e.g. Debian, Ubuntu and openSuse; you can run vanilla versions of the OSes on e.g. ARM64 TV-boxes as long as you can boot the OS.)
Edit: Even more off-topic: I hope more people explore these alternate architectures with Fedora since Fedora is a great distro and is a really good option for such devices.