The packages that caused the problem are actually the kernel-debug-uki-virt and kernel-uki-virt. In a test I found out that removing these packages removes the wrong file and leaves the big file in /boot/efi.
The files in /boot/efi/EFI/Linux are all created as a result of installing or upgrading the uki packages. After that there is nothing that would remove them again when upgrading.
After running dnf remove kernel-uki-virt the initrd file is removed, potentially making the system unbootable.