I found the issue, it was indeed because of initramfs being too large. Apparently grub can’t deal with a 220 MB initramfs on a 64 GB machine from 2020 
I’ve managed to slim down the initramfs to about 50 MB by running make INSTALL_MOD_STRIP=1 modules_install before make install to have the modules stripped from any debug information, and that makes the kernel boot.
If anybody has edit access to Building a custom kernel - Fedora Project Wiki it might be worth pointing this out and saving someone 3 hours of troubleshooting.