I’m building a PXE environment that can headless provision new Raspberry Pi 3 B+ and 4 B boards as well as re-provision when the SD card needs swapping out.
I’ve managed to get a Fedora 31/32 PXE kickstart install working on the RPI3B+ with the following steps:
Get DHCP address
Pull bootcode.bin, fixupx.elf, startx.elf, upstream.dtbo and u-boot.bin from TFTP server assigned in DHCP lease
Start U-Boot
Pull boot.scr.uimg and execute
boot.scr.uimg makes U-Boot pull down grubaa64.efi as well as the DTB
Anaconda is loaded and installs Fedora 31 on SD card according to kickstart.
Hack 1: I did not get start.elf to load the Fedora PXE vmlinuz + initrd.img nor the GRUB2 EFI binary directly. It downloads the files but then all I get is a frozen UART console output. The only way I was able to run vmlinuz + initrd.img is to put U-Boot and the GRUB2 EFI binary in front of the kernel load.
Hack 2: When supplying install.img (stage2 image) via HTTP, the PXE needs to download it first and then expand it into memory. The RPI3B+ only has 1G of memory but the modern decompressed stage2 images seem to need more than that so I had to point the kernel to a NFS volume to read it from NFS instead which makes it not have to decompress it into memory. As I’m moving to RPI4B devices with 4G memory, this will be the past hopefully.
So my main question right now is: Why do the PXE vmlinuz + initrd.img images only load through U-Boot + GRUB? What am I missing that the RPI start.elf and start4.elf can’t load these PXE images?
Thanks to everyone taking the time to look into this.
I did make some progress by decompressing the vmlinuz kernel and loading the raw kernel binary.
I can now load the kernel and the initrd directly without U-Boot.
The next error I’m facing is that the kernel doesn’t see any UEFI from the RPI and can’t load the VL805 firmware and doesn’t see any UEFI environment as well as DMI:
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[ 0.000000] Linux version 5.6.0-300.fc32.aarch64 (mockbuild@buildvm-aarch64-08.arm.fedoraproject.org) (gcc version 10.0.1 20200311 (Red Hat 10.0.1-0.9) (GCC)) #1 SMP Mon Mar 30 15:34:01 UTC 2020
[ 0.000000] Machine model: Raspberry Pi 4 Model B Rev 1.2
[ 0.000000] efi: Getting EFI parameters from FDT:
[ 0.000000] efi: UEFI not found.
<... snip ...>
[ 0.000000] Kernel command line: coherent_pool=1M 8250.nr_uarts=1 cma=64M cma=96M smsc95xx.macaddr=DC:A6:32:80:95:FC vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000 console=ttyS0,115200 console=tty1
<... snip ...>
[ 0.069030] DMI not present or invalid.
<... snip ...>
[ 25.011544] uart-pl011 fe201000.serial: there is not valid maps for state default
[ 25.019400] fe201000.serial: ttyAMA1 at MMIO 0xfe201000 (irq = 20, base_baud = 0) is a PL011 rev2
[ 25.030823] raspberrypi-firmware soc:firmware: Request 0x00000001 returned status 0x00000000
[ 25.039448] raspberrypi-firmware soc:firmware: Request 0x00030046 returned status 0x00000000
[ 25.048365] raspberrypi-firmware soc:firmware: Request 0x00030007 returned status 0x00000000
[ 25.056938] raspberrypi-clk raspberrypi-clk: Failed to get pllb min freq: -22
[ 25.064206] raspberrypi-clk raspberrypi-clk: Failed to initialize pllb, -22
[ 25.071295] raspberrypi-clk: probe of raspberrypi-clk failed with error -22
[ 25.079778] raspberrypi-firmware soc:firmware: Request 0x00030030 returned status 0x00000000
<... snip ...>
[ 37.256920] raspberrypi-exp-gpio soc:firmware:gpio: Failed to get GPIO 0 config (-22 80)
[ 37.340183] raspberrypi-exp-gpio soc:firmware:gpio: Failed to get GPIO 1 config (-22 81)
[ 37.456938] raspberrypi-exp-gpio soc:firmware:gpio: Failed to get GPIO 2 config (-22 82)
[ 37.537775] raspberrypi-exp-gpio soc:firmware:gpio: Failed to get GPIO 3 config (-22 83)
[ 37.597253] raspberrypi-exp-gpio soc:firmware:gpio: Failed to get GPIO 6 config (-22 86)
[ 37.614486] raspberrypi-exp-gpio soc:firmware:gpio: Failed to get GPIO 4 config (-22 84)
[ 37.631423] raspberrypi-exp-gpio soc:firmware:gpio: Failed to get GPIO 6 config (-22 86)
[ 37.648358] reg-fixed-voltage: probe of sd_vcc_reg failed with error -22
[ 37.670269] raspberrypi-exp-gpio soc:firmware:gpio: Failed to get GPIO 4 config (-22 84)
[ 37.688453] pci 0000:01:00.0: Failed to load VL805's firmware: -22. Will continue to attempt to work, but bad things might happen. You should fix this...
[ 37.717228] raspberrypi-exp-gpio soc:firmware:gpio: Failed to get GPIO 5 config (-22 85)
[ 37.722993] raspberrypi-exp-gpio soc:firmware:gpio: Failed to get GPIO 4 config (-22 84)
[ 37.758343] gpio-regulator: probe of sd_io_1v8_reg failed with error -22
[ 37.784134] raspberrypi-exp-gpio soc:firmware:gpio: Failed to get GPIO 6 config (-22 86)
[ 37.820213] raspberrypi-exp-gpio soc:firmware:gpio: Failed to get GPIO 7 config (-22 87)
<... snip ...>
[ 37.950342] raspberrypi-firmware soc:firmware: Request 0x00028001 returned status 0x00000000
[ 37.959593] xhci_hcd 0000:01:00.0: xHCI Host Controller
[ 37.966309] xhci_hcd 0000:01:00.0: new USB bus registered, assigned bus number 2
[ 37.966753] raspberrypi-firmware soc:firmware: Request 0x00028001 returned status 0x00000000
[ 37.973884] xhci_hcd 0000:01:00.0: Host supports USB 3.0 SuperSpeed
[ 37.973908] xhci_hcd 0000:01:00.0: WARNING: Host System Error
<... snip ...>
[ 39.774086] random: crng init done
[ 39.777581] random: 7 urandom warning(s) missed due to ratelimiting
[ 207.308822] dracut-initqueue[980]: Warning: dracut-initqueue timeout - starting timeout scripts
[ 208.087573] dracut-initqueue[980]: Warning: dracut-initqueue timeout - starting timeout scripts
[ 208.788905] dracut-initqueue[980]: Warning: dracut-initqueue timeout - starting timeout scripts
[ 209.497727] dracut-initqueue[980]: Warning: dracut-initqueue timeout - starting timeout scripts
<.....snip.....>
[ 291.298611] dracut-initqueue[980]: Warning: Could not boot.
Starting Dracut Emergency Shell...
Generating "/run/initramfs/rdsosreport.txt"
Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report.
Press Enter for maintenance
(or press Control-D to continue):
When booting the Fedora-Minimal-32-20200403.n.0.aarch64.raw.xz image, it seems that U-Boot is providing the correct UEFI environment for the Kernel to be able to find things:
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[ 0.000000] Linux version 5.6.0-300.fc32.aarch64 (mockbuild@buildvm-aarch64-08.arm.fedoraproject.org) (gcc version 10.0.1 20200311 (Red Hat 10.0.1-0.9) (GCC)) #1 SMP Mon Mar 30 15:34:01 UTC 2020
[ 0.000000] Machine model: Raspberry Pi 4 Model B
[ 0.000000] efi: Getting EFI parameters from FDT:
[ 0.000000] efi: EFI v2.80 by Das U-Boot
[ 0.000000] efi: SMBIOS=0x3cb3a000 MEMRESERVE=0x3c8ae040
<... snip ...>
[ 0.066411] SMBIOS 3.0 present.
[ 0.066442] DMI: raspberrypi rpi/rpi, BIOS 2020.04-rc4 03/31/2020
<... snip ...>
[ 5.875064] fe201000.serial: ttyAMA0 at MMIO 0xfe201000 (irq = 16, base_baud = 0) is a PL011 rev2
[ 5.884335] serial serial0: tty port ttyAMA0 registered
[ 5.891764] raspberrypi-firmware soc:firmware: Attached to firmware from 2020-03-26 17:19
[ 5.900825] raspberrypi-clk raspberrypi-clk: CPU frequency range: min 600000000, max 1500000000
All of the DMI seems to be co ming from U-Boot here which brings me back to my initial question. Where does the dependence on U-Boot come from for the RPIs?