I’m using Fedora IoT, similar to Silverblue except no gnome, only minimal image. After did a small change on u-boot, I now have a working env. with Eth via USB hub to install software packages. I had tried to install weston(wayland), it was working fine. So IMO you can just try IoT as base, connect to network and install graphic on top of it then tweak. For custom kernel, just fork kernel repo from src.fedoraproject.org, adding custom patches, using rpm-ostree to overwrite default one.