Ignition failed when trying to install on Hetzner Cloud VPS

Ask Fedora
,
1

Hello,

I am using these instructions in order to deploy FCOS on a Hetzner VPS.

My butane file looks like this:

variant: fcos
version: 1.5.0
passwd:
  users:
    - name: core
      groups:
        - wheel
      ssh_authorized_keys:
        - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIArauL+P2Tu7Yi900y0YH/Z9Pf7hGI3ktIJYBsxpVglb

Which generates a pretty simple ignition file:

{"ignition":{"version":"3.4.0"},"passwd":{"users":[{"groups":["wheel"],"name":"core","sshAuthorizedKeys":["ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIArauL+P2Tu7Yi900y0YH/Z9Pf7hGI3ktIJYBsxpVglb"]}]}}

I was able to get this all working last week. However, since yesterday it is failing and I’m not sure what changed. I did make changes to my butane file, but I’ve reverted back to try the exact same thing I did last week.

I’m installing FCOS using:

export COREOS_DISK="https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/40.20240825.3.0/x86_64/fedora-coreos-40.20240825.3.0-metal.x86_64.raw.xz"
curl -sL $COREOS_DISK | xz -d | dd of=/dev/sda status=progress

There are two things that I notice are different now. One is that there are new lines in the output when I run fdisk -l /dev/sda in the recovery mode after I’ve installed FCOS.

The new lines are:

GPT PMBR size mismatch (5328895 != 80003071) will be corrected by write.
The backup GPT table is not on the end of the device.

And the full output is:

root@rescue ~ # fdisk -l /dev/sda
GPT PMBR size mismatch (5328895 != 80003071) will be corrected by write.
The backup GPT table is not on the end of the device.
Disk /dev/sda: 38.15 GiB, 40961572864 bytes, 80003072 sectors
Disk model: QEMU HARDDISK   
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 00000000-0000-4000-A000-000000000001

Device       Start     End Sectors  Size Type
/dev/sda1     2048    4095    2048    1M BIOS boot
/dev/sda2     4096  264191  260096  127M EFI System
/dev/sda3   264192 1050623  786432  384M Linux filesystem
/dev/sda4  1050624 5326847 4276224    2G Linux filesystem

I then create my ignition file like so:

mount /dev/sda3 /mnt
mkdir /mnt/ignition
vi /mnt/ignition/config.ign

I copy-paste the content of the ignition file in, save, and then power the server off.

After creating a snapshot of the powered-off server and provisioning a new VPS with it, I can not access the server running FCOS at all over SSH.

ssh: connect to host 5.78.122.84 port 22: Network is unreachable

When I log into Hetzner and view the cloud console, I see these errors:

image
image3374×1678 534 KB

Does anyone have any ideas on how to trouble shoot this?

2

You should take a look at Provisioning: Add Hetzner by travier · Pull Request #654 · coreos/fedora-coreos-docs · GitHub. It’s not fully ready yet and may change but it works well and should be easier to use, especially once we start producing those images directly.

1 Like
3

Those are really outdated instructions. You should use GitHub - coreos/coreos-installer: Installer for CoreOS disk images now and pass it your Ignition config.

4

Thanks for the prompt replies @siosm! I’ve been looking though those resources a bit and I’m finding it a bit confusing.

It looks like the hcloud-upload-image tool is a more streamlined way to make a temporary server and create a snapshot. I think I know how to do this manually using native hcloudcommands.

The instructions on this page suggest that the basic steps are:

  1. Download a metal image of FCOS
  2. Modify a the platform ID to be ‘hetzner’ instead of ‘metal’
  3. Create a snapshot of this image on Hetzner
  4. Provision a server and pass an ignition file in as user data.

However, you are also recommending that I use the coreos-installer tool to embed my ignition file into the FCOS image. Does that mean I should use something like coreos-installer iso ignition embed to put my ignition file into the image before I create the snapshot on Hetzner? Or, do I use coreos-installer after I have already provisioned a VPS with FCOS and have it add my ignition file after the fact?

Lastly, I have noticed that I can mount a FCOS image in Hetzner using their online portal:

image
image2966×1366 234 KB

However, beyond mounting this image, I haven’t figured out if I can do anything useful with it yet. Do you know if this is a viable way of installing FCOS?

Thanks!

5

It’s one or the other. Either you use the previous instructions from the post by Major Hayden but using coreos-installer, or you use the instructions from the docs PR that I linked above.

6

Looks like work is in progress to create special Hetzner ISOs. :tada:

Original comment:

I saw the same ISO in the online portal. I think this is the metal image of FCOS.

I successfully used it to create a new server and installing FCOS with the Ignition config in the user data with terraform. The only thing I had to change to make it work was to change the parameter ignition.platform.id to hetzner in grub during booting via the console in the web portal.

I contacted Hetzner, if they could correctly set the parameter in their provided ISO, but their response was, that the ISOs are not supported by them. I should update the ISO myself and submit a request to add it to my project.

Maybe it would be possible to add a Hetzner specific ISO to the Cloud images section on the download page of Fedora CoreOS? Then not everyone has to alter the image themselves, but could submit the official download link. Maybe Hetzner would even provide this ISO by default some day.

It would definitely be a lot less work, than all the custom snapshot or rescue system hacks that exist today.

7

Why would using an ISO be less work that using the snapshot as done in Provisioning: Add Hetzner by travier · Pull Request #654 · coreos/fedora-coreos-docs · GitHub?

Once you have a snapshot, you can immediately start systems and provision them with Ignition. If you boot an ISO then you need to install the image to the disk first and then reboot before the system effectively starts and provision itself.

8

Is there information somewhere about what the ignition.platform.id is and why it needs to be changed or what values are valid?

I got the ISO image feature to work as well, but I didn’t do anything about the ignition.platform.id.

What I did was simply:

  1. Provision a server with Fedora 40
  2. Use the online UI to mount the FCOS image. I think you could also use hcloud server attach-iso <server> <iso>
  3. Open the online console that shows the remote terminal
  4. Press the Ctrl + Alt + Del button in that terminal. The server then booted from the ISO image.
  5. FCOS recognized it wasn’t installed permanently and suggested running a coreos-installer command:
    image
    image2592×1872 451 KB

Overall, I found this pretty easy, but using that online console is not a great experience. Pasting my URL for my ignition file had to be done with a right-click and ‘https://’ was converted to ‘https;//’ for some bizarre reason. Also, I think that having to use the online console makes it so that this method can not be automated.

9

I think the online ISO is nice because Hetzner’s online console seems to do a lot for you, and every step can be completed using their online UI. For a complete beginner (like me), that was a bit more approachable.

However, I think I will prefer the snapshot method for the reasons you outlined and because that can be automated if desired. At the very least, I think avoiding that online console will ultimately be a better experience.

That said, I haven’t put these instructions into practice yet Provisioning: Add Hetzner by travier · Pull Request #654 · coreos/fedora-coreos-docs · GitHub.

The main hurdle is that I am on a Mac and most of the tooling required seems to run on Linux. So, I think I either need to set up a VM or use Podman to create an environment for me. Either way, that is one additional step I have to figure out how I want to manage, even though the end result will be better.

10

I found a list of values for it in the Afterburn pull request, that added support for hetzner.

In my opinion the ISO is easier, because I can mount it with terraform or using the hcloud API without me first creating the snapshot by hand or setting up an automation for it. Furthermore, I do not have to set any URLs like you stated, but just put my Ignition configuration in the user data. Ignition and Afterburn know about Hetzners metadata API and load it at boot.

With the ISO mounted I just have to start the server, it boots into the ISO, installs everything, reboots and is finished. Using a snapshot as the base would also require initialization and rebooting, so in terms of speed this should be pretty similar.

The only thing for that to work completely without touching the cloud console is an ISO, where the ignition.platform.id is set correctly.

With this, I can install FCOS on Hetzner without separate tools like Packer, coreos-installer, or hcloud-upload-image.

I can share my terraform script tomorrow, if you anyone is interested.

Edit: the script:

data "external" "butane" {
  program = ["${path.module}/generate-ignition.sh"]
}

resource "hcloud_server" "fcos1" {
  name = "prod-fcos-web-1"
  server_type = "cx22"
  image = "fedora-40"
  datacenter = "nbg1-dc3"
  iso = "fedora-coreos-40.20240709.3.1-live.x86_64.iso"
  backups = false
  user_data = data.external.butane.result.ign
}

generate-ignition.sh:

#!/bin/bash

# transpile to ignition config
ign=$(butane $BUTANE_FILE)

# prepare json for terraform
jq -n --arg ign "$ign" '{"ign":$ign}'
11

Yes, this should be easier once we start producing images.

Just to clarify, the “using the ISO” option is not wrong, it’s just more manual.

12

Maybe I misunderstand something fundamental about how FCOS works or is supposed to be used, as I am new to it and the whole immutable OS topic.

Could you please elaborate on why mounting a standard ISO is more manual, than first booting a recovery system to install the OS and make a snapshot?

You wrote:

Once you have a snapshot, you can immediately start systems and provision them with Ignition. If you boot an ISO then you need to install the image to the disk first and then reboot before the system effectively starts and provision itself.

Assuming we have an ISO with the boot parameter ignition.platform.id set to hetzner (which the FCOS team is planning to release) this will not be the case. The server boots the ISO, fetches the Ignition configuration and installs the fully provisioned OS. Only after that it reboots and is usable right away.

On the cloud console screenshot we see the orange message, that the ISO tried to fetch the Ignition configuration, but that it did not find any, so it could not provision the server automatically.

So what I am hoping for is, that this Hetzner specific ISO will be a standard base, we can use to provision new servers, instead of all of us creating snapshots ourselves.

Please correct me if I am wrong, but I think this is the closest to a 1-click install we will get to with Hetzner.

13

With the snapshot approach, once you’ve created a snapshot, you can provision as many server as you would like from it, and they directly boot, fetch their ignition config and work. No need to boot the ISO, install the OS, reboot before.

1 Like