Fixing rpm-ostree automatic updates!

Project Discussion
, ,
1

I think I have found the solution for automatic rpm-ostree updates!

Note that ublue fixes most of these points too, since quite a while, using their own updating service ublue-update. It uses topgrade which is an abstraction layer automatically updating everything it detects to support it (rpm-ostree, flatpak, distrobox, toolbox, …).

What is the problem?

“Traditional” Distros prompt users to update their stuff. This is annoying, unneeded and will lead to users not updating their system.

Meanwhile that may be needed, as “traditional” distros may have package conflicts and more, that need manual intervention.

Atomic Fedora doesnt need this, as it is just so much better.

But for some reason it has no automatic updates.

The updates are integrated in the software stores, great work @siosm but this integration has 2 problems:

  • the stores start slower and stay in the background, annoying users.
  • they arent even updating but just fetching updates, and still take so much time and resources. Stopping it leads to crashes on Plasma6 currently, 50% of the time.
  • it makes them really unpleasing to use for searching and installing Flatpak apps, which should be their only use case (apart from firmware updates)
  • also, the integration shows 2 rpm software sources, which is bad UX, confusing and not practical (it makes installing multiple RPMs nearly impossible) and also leads people to think layering was a supported use case which it is not

So, how to solve this?

I thought it wasnt possible to fulfull the requirements that make automatic rpm-ostree updates practical, while it is not!

  • only update on power (AC) connected (optional but may be required)
  • only update when the battery is over a certain percentage
  • do not update when the network is metered, i.e. a phone hotspot or USB-tethered network.
  • only run when the machine is at idle, as rpm-ostree needs lots of power (at least when layering)

All these are crucial as not having them could cause data loss or completely consumed cell data plans.

Not having auto-updates makes atomic desktops currently not fool-proof.

(On a Desktop PC you can just comment out the lines to remove the checks)

But all these are possible using systemd, we can add requirements to be fulfilled for the service to run, and easily check if AC is connected, battery is over x % or if the currently active network is not metered.

The idea is

  • have a service that checks these and repeats every 15min if it failed
  • have a seperate timer that activates the service. That timer actually gives the interval you want your updates to happen

Have a look at my way, I have the feeling there are some hacky parts (e.g. getting the metered state of the currently active network).

I have included a logging service, so that I can try and report if it is practical.

The defaults are sane, but a GUI in systemsettings/gnome settings for changing

  • only on AC
  • only with battery percentage over 40
  • only on idle 15

(With the possibility to change the values) would also be cool.

This will make automatic updates possible and decouple the need to integrate them in the software stores, making them quick again.

(As a workaround, rpm-ostree override remove plasma-discover-rpm-ostree or gnome-software-rpm-ostree works)

2 Likes
2

How would the idle rule work? Maybe it’s unlikely, but I pretty much turn on my computer use it, and then close the lid or shut it off. If I do that on a regularly basis I guess it would be longer before my automatic update goes off? Maybe in practice I am more idle than I realize.

Otherwise this seems great to me.

2 Likes
3

So I can’t speak to Fedora Atomic and rpm-ostree specifically, but on our side of things, on MicroOS/Aeon/Kalpa, we fire off our updates by default, via a systemd timer, during a maintenance window, every night at a random time between 00:00 and 02:00 Localtime.

If you’re on a laptop or a system that doesn’t stay powered on 24/7, The default is a randomized time, sometime between boot and 2 Hours after boot, if you’ve missed the previous maintenance window, and are on AC Power.

Just one way of doing it.

2 Likes
4

No. I use Kinoite and I like the current system. I want to be the arbiter on when to apply updates. If an update is borked and I’m in the middle of a time-sensitive project, the last thing I need is a non-functioning computer.

A setting in the Fedora welcome screen that allows you to enable automatic updates and briefly describes the tradeoffs is good enough, in my opinion.

2 Likes
5

I have to fully agree with @runiqo on this suggestion. Don’t want to force auto updates on users period. Let’s remember where Fedora exists, at the near bleeding edge for rawhide, and sometimes right at it. This can and almost always will lead to unforeseen errors that our testing won’t catch.

6

systemctl disable --now rpm-ostree-automatic

This is a fair point but

  • it doesnt apply to all users
  • what do you get from manual updates that a good “x was updated” message doesnt give you?
  • this updater can be opt-out and the GUI store integration can be used instead. Even though I would prefer to remove it, this could be layered or kept also on autoupdates.

@jakfrost I dont see the problem, users can stay at the old supported version. Also there are rollbacks for a reason, (I think flickerless boot hiding the rollbacks is the problem)

7

Yes this needs to be tested and could be opt-in

@sfaulken how do you find if it is a laptop or 24/7 on Desktop? And what device is on 24/7 at all?

8

The decision to not do the update prior to finding it has introioduced a regression (although unintentionally).

The problem is, this is not Windoze or MacOS, which are closed ecosystems. Fedora has four founding principals, First being one important one that brings the newest tech to us pretty early in the use cycle. This leads to a near bleeding edge life for those who run rawhide perpetually, speaking from experience. This does lead to breakage that a casual Linux adopter would find intolerable, in most cases. Since there are many devices that Fedora can be installed on, and there is a vast array of technological differences and use cases, an auto update feature is not preferable just on that basis alone IMHO. Not to mention the fact there are limited dedicated resources in the project to begin with, this is all voluntary, even RH employees from what I can ascertain, are volunteering their time on this project. That goes also for the community “support” that some provide here or under ask.fp.o, which would be adversely affected by something like this. With limited resources like the project has, I would prefer to (see us) spend energy on things that can help in areas like … boot loaders, installation/recovery tools, infra-engineering needs, documentation, system hardening, QA Testing, the list goes on

9

Hm for sure. But we have full rollback support.

I think users knowing when a regression will come are very rare. Most will just press update, there is no warning about regressions. Manual updates dont help here.

This would also have no impact on our limited support or how well it runs on various devices, the spins/versions handle that and users will always update their systems and have breakages. I dont know how manual updates should help here, as I said therenis no way to see if there is a regression.

I would instead propose to use a nice Fedora grub theme and always display grub, making the rollback easier. Because as I understood auto-rollback will only happen if it doesnt boot at all.

A scenario could be to choose to not update for a while, until it is fixed. This would make sense but could be done via a systemd disable (or stop?) And a reminder after a week or so. This would be possible and is a good point to add.

This would need an interface which in general sounds like it all should be implemented into rpm-ostree, but this seems like making stuff more complicated and take longer (until then it would be hacky).

And as I said users can and probably should stay on the old supported version if stability is a priority. Fedora 38 is perfectly usable and afaik doesnt ship many feature updates.

But needing to manual pin deployments (and find the number to unpin) or rollback on every boot are for sure not good options.

10

There are a multitude of varied use cases and users in the Fedora community. To assume on automatic updates is a disservice to a larger user base than is being given credit for on this topic. Server admins are one such, developers are another, designers another, etc… There is a whole cohort of people using Fedora who never used Windows or MacOS as their personal PC OS from the beginning, not everyone chose the path of least resistance. The release notes do a pretty good job of letting anyone know whats changed with the latest update, and some of us pay attention to those. Another really bothersome item for updates, which is chronic until the OEM does some linux kernel lovin, is nvidia. This is currently a big problem for automatic updates, and due to it’s prolific use in pre-built systems being sold by the larger hardware OEM’s plus it’s very long historical relationship of ignoring the linux community in preference for closed ecosystem OS it will be for some time IMO.

11

I’m not sure I understand the question. If you’re using a laptop in a desktop role, that’s plugged into AC power 24/7 why would need to check for it?

That’s actually my usecase, my main workstation is a laptop plugged into a USB-C Dock, that only gets rebooted for updates, or if I’m leaving the house and need my laptop. So for the timers purposes, why would it care whether the hardware is a laptop or a desktop? The default just checks for the presence of AC Power as a condition of updating.

12

I don’t use Fedora Atomic everyday, but I’m pretty sure the updates aren’t applied until you reboot the machine, and boot into the new ostree deployment (or perhaps manually switch with apply-live to the new deployment)

How exactly would this cause your machine to mess with the running system? It’s part of how rpm-ostree works, this just isn’t a behavior that should generally be possible with an Atomic Distribution. That’s a huge part of why you run one, because the updates don’t touch your running system until such time as the user decides to reboot.

If you’re in the middle of something critically important, you just don’t reboot, until you’re in a spot where if something does happen to break, you have the time to deal with it.

3 Likes
13

I guess we can back up a step further and ask the question of whether Fedora Atomic Desktops should have automatic updates. On that topic, Silverblue and Budgie Atomic (I think) already have automatic updates enabled. Kinoite will soon as well. So from this perspective @boredsquirrel is suggesting a better way to handle this than the current way it’s done. I haven’t really been hearing complaints on that front so much so that I didn’t know it was already a thing.

If we want to discuss whether automatic updates should be enabled by default, I think it would be better to make a new topic on that since this topic is about a specific implementation.

I’ll also emphasize what @sfaulken shared about how atomic updates work. The changes are updated in the background to a new version of the image that you will then boot into next time. Nothing is changing live in your system. If someone was in the middle of working or left their computer on for days at a time, that update will not apply at all during that time. That’s part of the benefit of this style of updating rather than how updates work now. Even then, depending on how you do it even mainline Fedora will not apply certain updates until you reboot because it’s the recommended way to handle those updates. People shouldn’t experience breakage often, but if they do they can rollback. That is the way you cope with breakage on an atomic system rather than just waiting longer to update. And that’s on top of the fact that you can disable automatic updates to get even more control over how your PC is managed.

3 Likes
14

I echo this statement, Now with a caveat. The rpm-ostree does offer some stability to the underlying system, but if the productivity apps change constantly, it’s more of a burden for Creators/Creatives who need to stay on a version until a project is done.

Also, people who are QA/QC, it’s mandatory to QA your software against the build you started the process with. Changes to packages can cause irrational behavior in a QA process.

I’m not on an OS-Tree OS, because I have been doing more and more Creative work, so i am already out of the conversation. Interoperability with Contained apps is broken and will be until Portals is 100%.

1 Like
15

@jakfrost yes there are many user types and we have no idea if they would want this or not. So I think we can’t say anything here. We need more telemitry or surveys that pop up on the running systems to get neutral data.

to the NVIDIA point, for sure I may implement a mechanism that automatically pins the current deployment when there where NVIDIA package changes.

I added a notify-send notification displaying the package changes, that never disappears until manually closed (found that rpm-ostree command while translating docs lol).

This should already be very good for NVIDIA users. Adding a button to automatically pin the current deployment, and after a reboot adding a dialog to unpin it if everything works could be done, but that would be DE-dependend with kdialog or zenity (or just add zenity on all variants, but that Kinoite issue was closed pretty quickly…)

@sfaulken yes updates are not applied until rebooting but after a reboot they will be there and may cause trouble.

But please dont forget that… we have rollbacks, thats the point of the atomic distros.

They just need to be made more reliable (pinning deployments on critical changes) and accessible (not hiding grub)

So two additional change ideas:

  • add a button to the update message to pin the current deployment
  • AND somehow allow to unpin that deployment afterwards
  • dont hide grub but add a nice fedora theme

@joseph thanks, 100%. This is the point I am at, and the fact that some variants already have automatic updates is pretty shocking tbh. Not updating is not a solution too though, Fedora has no security backports so updates are also always security updates.

Tbh a general button for rpm-ostree rollback could be made more accessible. But I am not sure if the next update wouldnt just ignore that and you would need to repeat it all the time.

So having a dialog to disable auto updates for some time would be good. This would disable that service and

  • A: remind after x days to enable it again
  • B: enable it again after x days and display a message

I am not sure how to set such a timer though, that survives reboots.

16

I didnt know “staying on an app version until a project is done” was a thing. It seems rpm-ostree update --exclude APP doesnt exist.

But what app does this, arent normally used apps backwards compatible?

you can install apps in a toolbox/distrobox or just layer them, so I dont see the problem. It will cause longer updates… but these are now taken care of :smiley:

17

I am currently experimenting on a solution to pause automatic updates. This will need kdialog or zenity so it needs 2 solutions, as some people dont want to include zenity in kinoite…

18

Sometimes I do work in VMs (building test clusters, etc.), and I need to reboot because I have set a new sysctl or similar and want that to have an effect. If my system has had updates applied in the meantime, this reboot may cause a regression.

If you’re in the middle of something critically important, you just don’t reboot, until you’re in a spot where if something does happen to break, you have the time to deal with it.

Maybe that’s just me, but it’s much easier to do “I haven’t run updates in a hot minute, I should definitely do after finishing this bit of work” instead of “I may have updates applied in the background, under no circumstances should I reboot right now.” This is kind of stressing me out just thinking about it :slight_smile:

19

Ha, I wish. :slight_smile: I’ve had containers not working because Podman was convinced I need to do yet another podman system migrate or similar. There are times when we must sacrifice backwards compatibility on the altar of security.

On the other hand—I could be convinced to have backwards compatible security fixes applied automatically, but let’s be honest, this is Fedora and the edge is always bleeding. Asking package maintainers to find out if a package update is purely a backwards compatible security fix puts even more of a burden on them, and the poor folks are stretched thin enough as is.

20

Ah, I indeed wasn’t aware of that! Okay, if that is going to be the case, then maybe I’m just the exception.

That’s true then, @boredsquirrel what you’re describing sounds sensible. “Only run when idle” could just be done with a high nice/ionice value. That’d be one fewer setting, and fewer settings are usually good IMHO.

I’m still wary about the “shutting the lid” scenario, though. What would take priority: Finishing the update or suspending the system? I was under the impression updates cannot easily be paused and resumed.

1 Like