Objective Review: Fedora is a popular source for containers and Flatpaks

We’re working on Fedora Strategy 2028 — our next five-year plan. We are now reviewing those Objectives and their associated Impact. Read this guide for details on the current planning phase.

This Objective is part of the Theme “Fedora leads in Linux distribution development” and the Focus Area Technology Innovation & Leadership. For general discussion of this focus area, please see the topic Fedora Strategy 2028: Focus area review (Technology Innovation & Leadership).

Objective and Impact

Objective: Fedora is a popular source for containers and Flatpaks.
Impact: Fedora is a trusted source of software beyond our own base OS, reaching more users and potential contributors.

Let’s start this off with a comment on the overall introduction of this strategy draft from @howekif647:

That’s exactly what this is about. We’re pretty clearly backing away from Modularity as an approach to solving this problem, but it’s still a real one. It will also help solve the “too fast / too slow” problem — everyone wants different parts of the system to be either newer or older, faster or slower — but no two people have the same needs.

So, we need these things so we can make better Fedora Linux variants. Modularity intentionally did not depend on container technology — it seemed to early. But I don’t think we’re too early now. (If anything, we’re late.)

We could focus on just immutable variants just ignore everything but the base — stop packaging anything above that. We could send people to Flathub and Docker Hub and wherever else for the actual applications. But… we actually have a lot to give here.

I think it’s safe to say that there has been some tendency to use container formats as “skip the distro, dump it all in here!” or “direct from upstream is clearly best”. And, those are fine in some situations. But we know that distribution projects like Fedora provide significant value to end users and to upstreams — better security, better reliability, better integration, and more. We need to clearly show that to the world.

Our goal now

For this Objective and related Impact, validate that:

  1. If the Impact is achieved, it’s reasonable to expect an increase in active Fedora contributors.
  2. Success in the Objective logically results in the intended Impact.
  3. That link is reasonably sufficient — that is, it represents everything needed to have the Impact.
  4. While there might be other ways to have similar Impact, the chosen Objective is the right one for Fedora right now.
  5. The wording is precise and clear. The Objective is concrete, and the Impact is (at least a little bit) inspirational. Together, they fit into this Focus Area.

Bonus. If you can improve the longer explanatory paragraphs at the top of this post, that’s helpful too!

As outlined in the roadmap, this post will close in one month.

2 Likes

This one’s very important. If we win the “what should we use as the basis for our containers” battle, that makes everything else we do for doubling our active contributors much easier. Becoming the default means every* demo and instruction you see for third-party software uses Fedora Linux, which has a natural effect of growing our contributor base in addition to users.

2 Likes

I have a though about “skip the distro dump it all here” with containers. Not every time the apps devs react as fast as we do with security patches or even when just normal newer versions of dependencies, and I face that recently with an app: the rpm of a dependency was released fast and the container of that dependency was released too, but the app doesn’t work with that lates version. Me, being a “tech guy” and being part of the project, understand and know that this is not fedora’s fault, but a regular user just will see that the app doesn’t work in fedora, which is not true, ir doesn’t work anywhere. So, if we rely only in containers we are going to face this all the time. And I think the flatpaks suffer from the same, if the flatpaks is packaged with an old dependency it will suffer from their vulnerabilities.

So my question/thought is how we can address that as a project?

Yeah, to be clear: the proposal is that we generate flatpaks and containers that are entirely Fedora content — either from RPMs or directly to containers but in a way that gives the same advantages we have through building RPMs.

How are Fedora Flatpaks packaged differently from Flathub? Why should an end user prefer the Fedora Flatpak over the Flathub version, especially if the Flathub version comes straight from the developer? Even if Fedora’s process is different, how different is it from how Flathub does things? Does it add enough value to justify itself or is it only marginally better?

I don’t know the answers to these questions, but if the answers are not satisfying enough to Fedora and Linux users, we could see our efforts toward this objective be snubbed in favor of the promise of Flathub, which is the One True App Store /s. I’ve already heard people question why we package our own flatpaks at all, so that opinion already exists. Fedora has its own validation process, but it is that much better than what Flathub does?

I guess I want to be educated on why Fedora should be packaging its own flatpaks in five years.

The Fedora Flatpak repository definitely confuses people, it’s not uncommon on other platforms such as Reddit to see people ask similar questions as you wrote and for others to (perhaps reflexively) suggest the Flathub version. It’s just my opinion, but while a Flatpak generated from a RPM or even a manual wrapping is better than none, ultimately all the benefits of Flatpak won’t be realized by users until the source project embraces that model.

This goal regroups two different things that might appear similar from a high level but that are fundamentally different in usage and place in the ecosystem.

Containers

Disclaimer: I maintain Fedora (and non Fedora) based containers. None of them are hosted in Fedora infra

As far as know, the only currently maintained Fedora container images produced in and distributed by the Fedora infra are the fedora base image, the minimal image and the toolbox image.

All other images are unmaintained. All other official (as in under the Fedora namespace on Quay.io) Fedora based container images are apparently maintained in GitHub (Software Collections · GitHub) and pushed to Quay.io (Quay). The status of those images is not clear for me, and apparently not even for their maintainers: Are those Fedora containers official? · Issue #10 · sclorg/welcome · GitHub.

This is unfortunately not explained in our docs right now. I’ve started to update Fedora Container & Tools Documentation :: Fedora Docs but it’s still not accurate.

The infrastructure to build container in Fedora infrastructure is also only vaguely documented, just barely maintained and completely different from everything else available right now (GitHub Actions, GitLab CI, DockerHub, etc.) which makes it really hard to understand and use.

Now with that said, where should we aim Fedora to be in 5 years regarding containers?

Today there are a lot of options available to users to build their own containers (GitHub Actions, GitLab.com, etc.) and host them (Quay.io, DockerHub). It does not really make sense for Fedora to invest in an infrastructure that no one is going to use.

I’d argue that the Fedora Project should focus on building and testing the base and base-minimal images and stop there. All other images should be maintained, built and pushed via GitLab CI.

Flatpaks

Disclaimer: I maintain KDE Apps packaged as Flatpaks on Flathub and KDE upstream CI (GitLab based).

Fedora needs Flatpaks to be able to pre-install them and builds them from the same source as RPMs. We notably wrote that in SIGs/Flatpak - Fedora Project Wiki.

The infrastructure to build Flatpaks in Fedora is mostly undocumented and barely maintained, which makes it very developer hostile (and I’ve tried). The differences with the Flathub workflow also make it yet another hurdle to overcome.

I see no benefit for an upstream developer to work on Flatpak’ing their app in Fedora. Doing it in Flathub is much easier for them.

My perspective is that we should focus on a core set of apps that we need installed on our desktop variants and work with Flathub to improve their process (legal and licenses checks, updates, etc.) for everything else.

yeah, our container build pipeline is… a trip. ;(

FWIW, I agree we should focus on base/minimal, and let users build on those. The hyper specalized, single purpose images always seemed odd to me. I’m much more likely to use a base image and install say nginx in it than try and use a ‘fedora nginx’ image. That might be because I’m old though. :wink:

For flatpaks, the new flatpak sig has done a lot of things… if we can figure out a better/more sane way to build them that would sure be good.

Can we help improve the flatpak default configs so that they’re more secure and actually take advantage of the sandboxing? I’m not sure if there is a way to encourage this, but I know that’s been a common complaint about flatpak that it doesn’t go far enough in it’s expectation of sandboxing.

These options presented could be good alternatives so that we keep contains and flatpaks in focus, just with a change to the objectives.

1 Like

This is mostly FUD spread by websites such as flatkill. There might indeed be some apps with “too much access” on Flathub but most of the time, the access is there to either let the app work as intended or to workaround some design limitations that would require significant work such as the introduction of a new desktop portal.

I’m not saying Flatpaks are perfectly sandboxed (they are not) but the point is that they are not installed as root on the system and don’t run random scripts as root during updates. This already makes them hundreds times more secure and sandboxed than regular packages that you would download from a random repository.

In some cases in Kinoite for example, we’re keeping some apps in the image because it’s too hard to make them run sandboxed (Dolphin (file manager), Filelight (file size explorer)).

If Fedora Flatpaks have stricter configs than Flathub ones, then it’s likely that some things won’t work in Fedora Flatpaks and that will create another reason making them less compelling compared to Flathub ones.

Work to improve how apps work with Flatpak is however a good idea and will benefit everyone.

I’d say there is value in having well made images (not running as root, using a configuration that works by default in a container, etc.) for popular applications such as nginx, postgresql, etc.

The main difference with the rest of the artifacts that Fedora produces is that their life cycle is not synced to Fedora releases. You want a PostgreSQL 13, 14 or 15 image, not an image with “the version of PostgreSQL included in Fedora 38”.

Thus why I think it makes sense to have it as a separated / autonomous thing that is not tied to our release cycles, which is kind of the state it’s in right now.

1 Like

This alone feels like it would be a huge win for both this Objective and possibly other strategy themes (and a load off of the Fedora Infra folks).

Adopting a more well known and documented way of building container images via GitLab CI would open up the ability for more of the community to help curate/support the “official” Fedora container images.

I disagree — I think we should ship server applications in containers by default, which means we need some way to produce them for ourselves.

I agree more with this, but the release boundaries are also where we change compilers, introduce security hardening features and policy changes, etc., and so it’s actually useful to have both things. (I’d also like to see us produce EPEL containers on UBI.)

This goes right back to what I was saying in the first post in this topic — I hope we can bring some of what we’d wanted to get from Modularity to our users, so they can get the version of PostgreSQL you need, on whatever base they like.

What about a compromise where the base images are built in the existing pipeline, but we make it easier for contributors to maintain and build the server application containers via Gitlab? Thankfully there is already a Fedora org in place on Gitlab, so it seems possible to create a Containers project under there and let folks contribute.

Kevin agrees that the existing container build pipeline is not ideal and limiting the support for that pipeline to just base images would reduce the burden on the Fedora Infra team.

The existing pipeline needs to be replaced. Gitlab CI might be the answer.

I’m quite possibly ‘old man yelling at containers’ here, but for me (and lots and lots of other SRE/admin types I interact with), I don’t understand where you would seek out and use a ‘apache container’ or a ‘nginx container’ instead of using a base image and layering the packages/application you need + all the config you want to do in it. I mean, I can see a case for complex to setup applications to have it all setup (so, say a synapse-matrix container makes sense to me), but do people really choose specific app containers like this over layering?

And yes, I would be happy to replace our current pipeline, but I don’t think we should focus on the technical replacement/that here…

I can think of three reasons quickly:

  1. It’s nice to have it all (no pun intended) contained. More lightweight (and energy-efficient) than starting a VM for each service. (And you can manage the configuration on the main host and mount it into the container.)
  2. Addresses “too fast + too slow” by decoupling the base OS from the applications and the applications from each other.
  3. If you get used to working this way, it’s easy to move things to kubernetes / openshift if you want to do that.

Evidence suggests “yes”. :slight_smile:

I am not suggesting using VM’s. I mean the difference between:

  1. pull fedora:38/nginx and then add your config to it (I guess via passing options or variables to the container?)
    vs
  2. pull fedora:38 base container and ‘dnf -y install nginx’ as the first part of your configuration, then add your config, then use that.

The second one is what I do and see others doing. It means you don’t have to learn what way the person who setup the container has to configure things or setup things, you can just install and configure like you always would.

The ‘too fast + too slow’ issue I don’t see being solved by this either. We have to make the containers from packages, and if we have postgresql-10 and postgresql-11 packages, they could just install the one they want in the base container.

You can totally use a base container + installing what you need + config in openshift, this is what we do in fedora. This also allows you to install N things if you need them instead of running seperate containers for each.

But anyhow, I’ll shutup now… perhaps I just need to adjust to learning how ‘specialized’ app containers are configured.

Oh, I see! Sorry for misunderstanding. Please disregard my entire reply above. :slight_smile:

Ngnix is easily configured with one simple file, so I just do -volume=~/etc/nginx.conf:/etc/nginx/nginx.conf:Z.

For the build-on-the-base-image approach, how do you deal with updates? Rebuild periodically? (Genuine question — I am, after all, only a dilettante sysadmin these days!) If, instead, there’s an Official Image, I just use podman auto-update.

Yeah, sorry if I wasn’t clear there at first.

Yeah, rebuild the image… Then if some update causes pain you can rollback to the old one, etc (which I imagine podman auto-update also lets you do).