Zincati cannot to update to new image

Hello,

I normally have Zincati disabled, as I want complete control of when my PC updates and reboots. I am now trying to update my PC manually by starting the Zincati service, but am getting this error in the journal:

systemd[1]: Started zincati.service - Zincati Update Agent.
zincati[52316]: [INFO  zincati::cincinnati] current release detected as not a dead-end
zincati[52316]: [INFO  zincati::update_agent::actor] target release '39.20240104.3.0' selected, proceeding to stage it
zincati[52316]: [ERROR zincati::update_agent::actor] failed to stage deployment: rpm-ostree deploy failed:
zincati[52316]:     error: Cannot look up version while pinned to commit

Is this because I have a deployment pinned? Is this intended behavior? If so, is there another way for me to update the image and packages, but also have the option of reverting the PC back to its original state or one of several prior states if I need to?

Thank you

Added rpm-ostree

Added ostree

Added updates

Hello @anticut and welcome to :fedora: !

Not sure about the error, but generally you can disable automatic updates without disabling the Zincati service like this:

You can then upgrade manually:

rpm-ostree upgrade --bypass-driver

Thanks Hristo,

Yes this is the method I have used to disable Zincati, sorry for the poor wording in my initial post. In my attempts to update the system, I would change it to enabled = true and restart the service.

The manual upgrade through rpm-ostree does not seem to upgrade the image, but only the layered packages. Am I perhaps using it wrong? Is there a way to get it to update the image also?

We would need the output of rpm-ostree status to be able to help.

Thanks Timothée, here it is:

$ sudo rpm-ostree status -v
State: idle
AutomaticUpdatesDriver: Zincati (zincati.service)
  DriverState: active; initialization complete, auto-updates logic disabled by configuration
Deployments:
● 88bcf6cd3a7f00b67147afb2261025b910c87b8c06c8364e628ed3521979fae6 (index: 0)
                  Version: 39.20231119.3.0 (2023-12-04T16:21:28Z)
               BaseCommit: cd3ab5975ace83aa36f687d2f2d58ea59b4d1cceef6d0cd18a231248ce6ae207
                           └─ fedora-coreos-pool (2023-12-02T23:12:44Z)
                   Commit: 88bcf6cd3a7f00b67147afb2261025b910c87b8c06c8364e628ed3521979fae6
                   Staged: no
                StateRoot: fedora-coreos
      RemovedBasePackages: nfs-utils-coreos 1:2.6.3-2.rc3.fc39
          LayeredPackages: alsa-utils aria2 binutils bluez drm-utils dvb-firmware feh ffmpeg flatpak freeimage fzf git igt-gpu-tools intel-media-driver iotop kodi kodi-inputstream-adaptive kodi-pvr-hts lftp libdisplay-info libglvnd-gles
                           libva-utils lz4 man-db man-pages mesa-dri-drivers mesa-va-drivers mesa-vulkan-drivers moreutils mpv neovim netcat nfs-utils p7zip parallel parted pipewire pipewire-alsa pipewire-pulseaudio pipewire-utils
                           retroarch ripgrep tcpdump tinyxml2 tmux v4l-utils vlc-libs vulkan-tools wget wireplumber
                   Pinned: yes

  88bcf6cd3a7f00b67147afb2261025b910c87b8c06c8364e628ed3521979fae6 (index: 1)
                  Version: 39.20231119.3.0 (2023-12-04T16:21:28Z)
               BaseCommit: cd3ab5975ace83aa36f687d2f2d58ea59b4d1cceef6d0cd18a231248ce6ae207
                           └─ fedora-coreos-pool (2023-12-02T23:12:44Z)
                   Commit: 88bcf6cd3a7f00b67147afb2261025b910c87b8c06c8364e628ed3521979fae6
                StateRoot: fedora-coreos
      RemovedBasePackages: nfs-utils-coreos 1:2.6.3-2.rc3.fc39
          LayeredPackages: alsa-utils aria2 binutils bluez drm-utils dvb-firmware feh ffmpeg flatpak freeimage fzf git igt-gpu-tools intel-media-driver iotop kodi kodi-inputstream-adaptive kodi-pvr-hts lftp libdisplay-info libglvnd-gles
                           libva-utils lz4 man-db man-pages mesa-dri-drivers mesa-va-drivers mesa-vulkan-drivers moreutils mpv neovim netcat nfs-utils p7zip parallel parted pipewire pipewire-alsa pipewire-pulseaudio pipewire-utils
                           retroarch ripgrep tcpdump tinyxml2 tmux v4l-utils vlc-libs vulkan-tools wget wireplumber

  88bcf6cd3a7f00b67147afb2261025b910c87b8c06c8364e628ed3521979fae6 (index: 2)
                  Version: 39.20231119.3.0 (2023-12-04T16:21:28Z)
               BaseCommit: cd3ab5975ace83aa36f687d2f2d58ea59b4d1cceef6d0cd18a231248ce6ae207
                           └─ fedora-coreos-pool (2023-12-02T23:12:44Z)
                   Commit: 88bcf6cd3a7f00b67147afb2261025b910c87b8c06c8364e628ed3521979fae6
                StateRoot: fedora-coreos
      RemovedBasePackages: nfs-utils-coreos 1:2.6.3-2.rc3.fc39
          LayeredPackages: alsa-utils aria2 binutils bluez drm-utils dvb-firmware feh ffmpeg flatpak freeimage fzf git igt-gpu-tools intel-media-driver iotop kodi kodi-inputstream-adaptive kodi-pvr-hts lftp libdisplay-info libglvnd-gles
                           libva-utils lz4 man-db man-pages mesa-dri-drivers mesa-va-drivers mesa-vulkan-drivers moreutils mpv neovim netcat nfs-utils p7zip parallel parted pipewire pipewire-alsa pipewire-pulseaudio pipewire-utils
                           retroarch ripgrep tcpdump tinyxml2 tmux v4l-utils vlc-libs vulkan-tools wget wireplumber

Thanks Timothée, here it is (for some reason the forum software won’t let me embed the content): $ sudo rpm-ostree status -vState: idleAutomaticUpdatesDriver: Zincati (zinca - Pastebin.com

pinning a deployment here isn’t going to work out well because we have really limited space in the /boot filesystem and you’ll end up hitting other problems that will prevent upgrade if you try to keep 3 copies of the kernel and initramfs around. You need to unpin the deployment and let zincati work.

On any upgrade the currently running depoyment (the one being upgraded from) will never be deleted so you’ll always be able to roll back unless zincati does multiple updates in a row (which can happen if your system is out of date) . My suggestion here is for you to disable the zincati service by default so it won’t run on boot, then only start it when you want to update. Don’t pin deployments.

So the pinned deployment is what is ultimately causing the error and preventing me from updating the system? This is intended behavior?

That is really unfortunate, I had hoped that CoreOS would allow me to keep a “known good” state around until I can absolutely verify the update is working exactly as expected (which could take days or weeks). As you say, if multiple updates occur then that known good state is no longer available and I could end up with a broken system with no convenient methods to restore it.

but I told you exactly how to do that above. Just systemctl disable zincati.service so it won’t start by default on boot and then only systemctl start zincati.service when you want to do an upgrade.

The number of upgrades you ever have will correlate directly with the number of times you type systemctl start zincati.service.

or just rebase your system to a container registry endpoint you control and sudo rpm-ostree upgrade it whenever you want.

I think I understand, except for how this would work with multiple updates in a row.

Let’s say my PC is currently on deployment A, and I want to update to the most recent image, deployment C. There is also a deployment B, a less recent image that is a week older than deployment C.

I unpin A, allowing Zincati to update to deployment B, then C. Both deployments B and C have bugs for the software I’m running. Since I had to unpin A to perform the update and only 1 previous deployment is kept around (B), I am left with a broken system.

What could I do in this situation? I suppose I’m trying to determine whether pinning deployments is an unsupported use case in CoreOS, as it seems to prevent system updates from occurring?

In my proposal you’d validate everything worked on B fine before running sudo systemctl start zincati.service to progress to C.

What if my apps are definitely broken on B, but could potentially be fixed on C? Is there no supported method of reverting back to A if I update and find that C is still broken?

For some context, my PC is currently running an old image because when I updated to a newer image it was affected by a GPU driver bug. That bug might be fixed now, but I cannot update to the latest deployment without potentially losing the one that currently works.

Not really. No. You can always just re-deploy that old commit from A (i.e. pull the commit from the OSTree repo and apply it), but keeping it locally isn’t going to work well because having 3 deployments on the node at a given time is going to run into Boot partition can easily run out of space on upgrade · Issue #1247 · coreos/fedora-coreos-tracker · GitHub and increase size of our /boot partition for new installs · Issue #1465 · coreos/fedora-coreos-tracker · GitHub

Really though, it would be better if you had a second system (or a VM or something) where you can test things before deploying them. So you’ll know if C works before you even start to update that system that you really care about.

I don’t think we explicitly disable upgrading systems with pinned deployments (i.e. zincati doesn’t prevent it) but clearly you are hitting a piece of code in the rpm-ostree code base that fails with that error. See Handle "pinned commits" specifically · coreos/rpm-ostree@27bd7b9 · GitHub

I’m not sure of the exact reasoning but you could maybe ask a question there or look deeper in the code to try to understand.