Help beginnger for first RE-provisioning

Hi there,
using CoreOS for a longer time now but only did a

  • bare-metal install via live iso initially
  • did make manual changes since then and in parallel keep my stored config file up to date manually for next initial provisioning
  • wanting now to check out the re-provisioning of my running production system i am a bit scared to break things

Having read “Reinstalling Fedora CoreOS” at Installing CoreOS on Bare Metal :: Fedora Docs, i think i did miss to e.g. create a var-partition with wipe_filesystem: false option

This means not having a separate partition for my var folder would completely wipe away all of my data if would execute sudo coreos-installer install with my updated ign-file if im not wrong

  1. am i guessing right?
  2. What is the best way to get to the state where re-provisioning my bare-metal setup is as easy as possible

Below maybe some helpful snippets of my butane-config
Using ucore to automatically include nvidia-drivers

variant: fcos
version: 1.5.0
...
storage:
  disks: # Define Disk
    - device: /dev/nvme1n1
      wipe_table: true
      partitions:
        - label: nvme2
          number: 0 # No Explicit number
          wipe_partition_entry: true
          size_mib: 0 # Max Size
          start_mib: 0 # First Startingpoint
  filesystems: # Define Disk Filesystem
    - path: /var/mnt/nvme2
      device: /dev/disk/by-partlabel/nvme2
      format: ext4
      with_mount_unit: true
...
  directories:
    # For ucore rebase
    - path: /etc/ucore-autorebase
      mode: 0754
...
systemd:
  units:
    # Unit for auto-rebase
    - name: ucore-unsigned-autorebase.service
      enabled: true
      contents: |
        [Unit]
        Description=uCore autorebase to unsigned OCI and reboot
        ConditionPathExists=!/etc/ucore-autorebase/unverified
        ConditionPathExists=!/etc/ucore-autorebase/signed
        After=network-online.target
        Wants=network-online.target
        [Service]
        Type=oneshot
        StandardOutput=journal+console
        ExecStart=/usr/bin/rpm-ostree rebase --bypass-driver ostree-unverified-registry:ghcr.io/ublue-os/fedora-coreos:stable-nvidia
        ExecStart=/usr/bin/touch /etc/ucore-autorebase/unverified
        ExecStart=/usr/bin/systemctl disable ucore-unsigned-autorebase.service
        ExecStart=/usr/bin/systemctl reboot
        [Install]
        WantedBy=multi-user.target
...

Are you using Fedora CoreOS or uCore? Why do you want to reprovision the system, i.e. is it unstable or malfunctioning, etc.?

1 Like

hi @hricky
thank you for responding to my question

Now and then i need to make changes to my system for the services i run on it
e.g.

  • activate ipv6 for docker
  • activate WoL

I thought:

  • Its good practice to NOT adjust stuff manually because on next install i have to do it again (or even remember what i need to do to run my services)
  • Its good practice to try to always update and develop the butane-config as single source of truth and just re-provision the system

As mentioned i think my current workflow

  • change everything manually
  • try to update butane config “offline” in parallel and hope it works on next installation as expected

is not very good

Did i get something wrong?

If not, im searching for the path how to “re-configure” my already running system with all persistent files and docker volume mounts stored somewhere in var/home, without wiping/crashing/breaking my running system

PS:

EDIT:
I know why u asked, the post above was bad edit from my side! I did not accidentially install ucore, i am really running fedora-coreos:stable-nvidia → Edited initial post

Could you provide the output of the rpm-ostree status command?

If I understand you correctly, you are using FCOS (uCore) as a general purpose server operating system on a bare metal machine. In such a use case, re-provisioning (reinstallation) should rarely be necessary (especially if the system is running smoothly), just like with any other server operating system, i.e. Fedora Server.

I think it is good practice to keep the Butane config as a blueprint. Since I do various tests, I have a lot of Butane templates.

There’s nothing wrong with custom system configurations. Whether it’s manual or not is up to you. Personally, I would recommend using Ansible, in which case you will need to layer Python.

If you are running your applications in containers, which is the recommended way for FCOS, you may want to consider implementing some automation/orchestration, e.g. Docker/Podman Compose, Quadlets, or Kubernetes.

1 Like

If you want to be able to re-provision your system from an updated Butane config then you probably want to have /var on a separated partition so that you can keep your data as is.

See:

As Hristo said, you only need to re-provision if you want to use this mechanism to update your system config but you may also use other means to do so.

1 Like

Sure, see below. Also noticed a few days ago, that my zincati updates are broken. I have to solve this … :eyes:

[core@blackpearl ~]$ rpm-ostree status
State: idle
Deployments:
● ostree-unverified-registry:ghcr.io/ublue-os/fedora-coreos:stable-nvidia
                   Digest: sha256:c461022b0ecce29da4aec38dec5955ad7e57326ecee9fbe26d1717839c1263f8
                  Version: 40.20240416.3.1 (2024-05-07T23:43:23Z)
          LayeredPackages: btop docker-compose helix vim

  ostree-unverified-registry:ghcr.io/ublue-os/fedora-coreos:stable-nvidia
                   Digest: sha256:c461022b0ecce29da4aec38dec5955ad7e57326ecee9fbe26d1717839c1263f8
                  Version: 40.20240416.3.1 (2024-05-07T23:43:23Z)
          LayeredPackages: btop docker-compose helix vim
Jan 31 16:25:08 blackpearl systemd[1]: Starting zincati.service - Zincati Update Agent...
Jan 31 16:25:08 blackpearl zincati[1359901]: [INFO  zincati::cli::agent] starting update agent (zincati 0.0.27)
Jan 31 16:25:08 blackpearl systemd[1]: Started zincati.service - Zincati Update Agent.
Jan 31 16:25:08 blackpearl zincati[1359901]: [ERROR zincati] error: failed to assemble configuration settings
Jan 31 16:25:08 blackpearl zincati[1359901]: [ERROR zincati]  -> failed to validate agent identity configuration
Jan 31 16:25:08 blackpearl zincati[1359901]: [ERROR zincati]  -> failed to build default identity
Jan 31 16:25:08 blackpearl zincati[1359901]: [ERROR zincati]  -> failed to introspect booted OS image
Jan 31 16:25:08 blackpearl zincati[1359901]: [ERROR zincati]  -> Automatic updates disabled; booted into container image ostr>
Jan 31 16:25:08 blackpearl systemd[1]: zincati.service: Deactivated successfully.

Yes thats exactly what i do. Installed once and runs super smooth, efficient and solid. I absolutely love it!

Hmh. Maybe i am thinking too much about it in a NixOS way. I always thought i am just too lazy to do it right.

Thats what i already do
All of my docker volumes are either

  • bind mount to /var/home/core/dvol
  • nfs mount to NAS for bigger files like images, videos, documents etc.

This makes backup and reinstall easy for me

I just thought all system configurations can/should afterwards be done via butane anyways

Thank you very much for providing all of this information and the message that i did everything right until now :sweat_smile:

Thank you very much
So my guessing from the first post was right

Maybe im thinking about it on my next install :wink:

It looks like you are using uCore. At the end of the section from the link you provided in your previous post, it says:

Note

zincati fails to start on all systems with OCI based deployments (like uCore). Upstream efforts are active to develop an alternative.

Since I don’t have a uCore deployment at hand, I’m not sure if this is the cause of Zincati’s failures. I also noticed that your uCore version is outdated. I’m not sure why you’re still on version 40.20240416.3.1, but I would recommend that when the Zincati issue is resolved, you update your system to the latest release.