Fedora Cloud — Edition, not an Edition, and the future

I’m not sure how much I can add to this, but as the person who first proposed the “Fedora Products” split, I can at least try to provide a bit of context.

Back around Fedora 19, the Project was suffering. We were hemorrhaging users to other distributions (in part because of the not-at-all-controversial switches to systemd and GNOME 3). Beyond that, we were getting pressure from inside Red Hat about the value of Fedora as it skewed more and more heavily into a desktop-focused distribution and was therefore looking less valuable as a basis for RHEL.

So I proposed the idea that we would take the idea of Spins and go a bit further with it. The idea was to build the distribution in such a way that we could produce deliverables to address specific needs in the marketplace, rather than being a one-size-fits-few “bag of Lego™ bricks” (as Matthew liked to describe it). In particular, the big change we needed to make (beyond the ability to compose multiple trees) was the ability for different Fedora “flavors” to have different default setups. For example, Workstation and Server groups in Fedora had wildly conflicting perspectives on the default filesystem, SSH server and firewall state (among other things).

I pitched the topic at the first Flock conference in Charlotte, North Carolina in one of the early main-room sessions and it continued into numerous hallway track conversations from there. One of the most important points that was raised was that for this to work, we’d need for the Products (later renamed “Editions”) to serve a specific need. This had to be a meaningful chunk of a marketplace. So we initially decided that we wanted to focus on three main areas, with the option to expand or contract the number of Editions as needed:

  1. The desktop use case - We wanted to ensure that we didn’t lose the loyal users that had stayed with us up to this point, but we also wanted to pick something more focused than “general desktop user”. We settled on the idea of a Workstation Edition that was designed around some common user personas to do work (initially we focused on software development, but later branched out into a somewhat more general space).
  2. The “pets” use case: In the old pets-vs-cattle metaphor, you had some systems (“pets”) that you always care for and keep running for a long time. These would be fundamental to your environment (Domain controllers, Storage servers, databases, etc.) that care deeply about “uptime” and “reliability”. For this, we decided that we would create a Product called “Fedora Server” to address those needs.
  3. The “cattle” use case: These were the short-lived, throwaway instances that could be used for rapid scaling of services. As such use-cases were generally handled in “the Cloud”, we opted to create a Cloud Product that would focus on making Fedora a player in that space.

Over time, the VM-based nature of Cloud computing gave way to containers and Kubernetes, and the perceived value of having a primary deliverable focused on a dwindling use-case was dropping. That was about the time that rpm-ostree was maturing to a point where it could be relied upon for a complete distribution and we opted to switch the “cattle” focus from VMs over to containers and container hosts. Initially, this was Project Atomic until that eventually merged with CoreOS to become Fedora CoreOS.

All of this has been a somewhat long-winded way of saying that the Editions have always been very targeted at filling a specific niche. “Cloud” is too broad of a concept, and it’s part of why we’ve considered numerous times that “cloud images” should just be a delivery artifact of Fedora Server and Fedora CoreOS rather than their own separate Edition.

Cloud images are very valuable to the Fedora Project! We do not want to lose them, nor do I want to suggest that not calling them an Edition means they are unimportant. From my perspective (and how I originally envisioned things), an Edition needs to solve a particular problem. Server for example solves the “I need a secure-by-default platform for running persistent services in my environment”. Workstation solves the “I need a platform that allows me to do my job with a minimum of hassle and a maximum of efficiency” case. But Cloud really only ever existed to provide a minimal base image for building upon.

To use a slightly different metaphor, I think this lines up with how the US Supreme Court decided some important patent cases in the early 2000s. Essentially, they ruled that “you can’t patent something that would not be patentable simply by adding the words ‘on a computer’ at the end of the description.” Essentially, they ruled that the novelty (and therefore patentability) came from the solution that the device/process presented, irrespective of the delivery mechanism. To my mind, this is exactly what Cloud should be: We should have a cloud image for Fedora Server for those who want to run their essential services on a private or public cloud. We should have a cloud image for Fedora CoreOS for people to build their container hosts in a cloud. Maybe we should even have a Cloud Workstation for people who want a centrally managed virtual desktop setup.

I just don’t think that “Fedora Cloud” in and of itself is an Edition. Find a use-case that is exclusively about the cloud (for example, managing and monitoring public, private and hybrid clouds) and build THAT, rather than simply offering a minimal base image. Maybe create and ship Fedora OpenStack, a tool to easily set up your own private cloud. Or Fedora OpenShift: manage containers in the public cloud using Fedora’s exciting new container tools! These ideas are, to me, better examples of what an Edition would be.

Over time, the VM-based nature of Cloud computing gave way to containers and Kubernetes, and the perceived value of having a primary deliverable focused on a dwindling use-case was dropping.

This is a bit of a reductive framing, and one I specifically disagree with. Containers are an important usecase for sure, but one that’s orthogonal to VM-based Cloud, which is still actively used, growing and isn’t going anywhere. There’s plenty of room for both to coexist.

We should have a cloud image for Fedora Server for those who want to run their essential services on a private or public cloud.

But “a cloud image for Fedora Server” is a fundamentally different product from “Fedora Cloud”. Cloud environments have different technical constraints (e.g. around storage and networking), different deployment mechanics, different management tooling and ecosystem integration points and generally different classes of users from traditional servers. This naturally results in a product that makes different choices to better address them, which aren’t necessarily the same choices you’d make if you were optimizing for traditional servers. Why woudn’t that be an Edition in its own right?

2 Likes

I’m suggesting action is what makes or unmakes an edition. A formal process was followed to make Cloud working group and edition. No formal process was followed to unwind them. Editions exist by Council mandate. Marketing strategy, or lack thereof, is not what defines whether something is an edition.

So I am currently stuck on: “cloud is not an edition because the folks involved intended it to no longer be an edition, but cloud cannot now be called an edition either even though folks involved intend it to be an edition.” It’s a double standard. And the inevitable confusion from being loosey goosey about process and setting expectations properly has now ensued, to the point that there isn’t agreement on whether Cloud is or is not an edition.

My view is, since Cloud Edition is being produced and Council hasn’t formally unmade what it made, Cloud Edition exists.

Considering that it’s been 4 years of assumption without process being pointed to why it’s not an edition, on the one hand; and two change proposals following process to evolve, maintain, and reference to the product as Cloud Edition on the other - it’s very curious to me that a bug asserting the edition is not an edition and therefore a blocker is filed, discussed, and voted on, inside of 20 minutes. What is the backstory here?

Per rules of order, the question should not have been asked because it was out of scope for the meeting. The presumption that Cloud edition is not an edition is in dispute and only Council can address the question.

For what it’s worth, I like the name “Fedora Flavors”. It’s nice, alliterative, and catchy! :stuck_out_tongue_winking_eye:

I think we might be talking past each other a bit, here. I’m suggesting that “Fedora Cloud” is too broad and that it might actually be better if we produced something specifically tailored to the VM-based cloud. My view is that it would probably not actually be called “Fedora Cloud”, because that’s both broad and ambiguous. Instead it could be things like: “Fedora for SaaS Infrastructure” and “Fedora MicroVM”.

I think that underestimates the requirements around running in the various environments. Minimizing the effort and activities of the very capable people who work on it. I can run FCOS and K8s all day, but Fedora Cloud is a better starting point for efforts around VDI on Amazon and next generation components for efforts there and I know that others feel so as well because they don’t ask for Workstation or Server.

Cloud gets to deal with platform issues, like the use of swapfiles and power management in opportunistic compute power. Is that an issue for Fedora Server? It might be a problem you want to resolve on workstation, but using a swapfile would be a strong deviation from the best practices – I would call it misleading and out of scope. It’s a single example. I can go on, but I think the point is clear. Cloud enables a lot of enterprise level support that doesn’t fit the server or other domains, per se.

I think you miss the (conceptual) difference between an ‘action outcome’ (to avoid product) and an Edition.

As I understand it, an ‘Edition’ adds various properties and (long term) goals, and the ‘formal process’ defines the rules how to decide if these additions are fulfilled. And yes, ‘marketing’ is one those - not as commercial Blabla but technical properties and use cases.

Following Davides (and others previously in the thread) hints, cloud may meet these additions.

On the other hand: CoreOS provides a ‘Cloud Launable’ and a ‘BareMetal / virtualized’ version.

I am really curious why that model should not fit for Fedora Server as well.

This is starting to hit purely semantic concerns at this point. You are addressing a bias on the word Cloud here and while that might be a real concern in this edge meets car, meets platform computing world, we might want to make up a new term like we did when Distributed Computing became Cloud I see that as potentially valuable in the face of portfolio marketing, but it’s marketing.

Cloud is a big deal and there are some interesting problems to resolve there, like DNS and long disk timeouts that just can’t get the right kind of attention when they are divided across multiple editions. that would just be frenetic.

3 Likes

Oh! I see. That actually did happen. It’s recorded in Issue #119: Approve Atomic PRD and Lifecycle Plan - tickets - Pagure.io. I thought I linked that above but I guess not. Atomic went through the same PRD process. It doesn’t spell out “formally wind down Cloud Edition” in the ticket, but that was definitely the understanding we had at the time of what was happening. As I said, at the time, there was no active separate-from-Atomic functioning Cloud WG. The lack of action here wasn’t that we didn’t formally make the decision, but that we didn’t follow up on it in updating the fedora-release package.

Arguably, we should have also updated the Cloud wiki docs and so on, but in practice Cloud and Atomic were all mixed together because they were the same group.

You can see even today on the Cloud wiki page:

Quorum Voting Policy

At minimum 51% of the voting members of the Atomic WG must be in attendance in order to make decisions on meeting items. The definition of voting members is that of those who have self identified themselves as participants of the Working Group by adding their name here. A quorum vote is required to approve a decision on behalf of the working group, not simply the majority vote of those in attendance of any particular meeting.

… emphasis added. Basically, what happened was:

  1. Cloud WG started making Atomic Image (not an Edition) in addition to Cloud Base Image (Edition). That was distributed initially from that same Cloud wiki page. See for example the Fedora 22 release announcement, where Fedora Atomic Host is listed as part of the Cloud Edition.
  2. The “Cloud WG” just rolled into “Atomic WG”. You can see this in meetbot here and here, where the last “Cloud” meeting is October 2016 and the first “Atomic” one November 2016. Note same people in attendance under either name.
  3. That group worked on the PRD for Atomic, which was approved by Council
  4. That group made Atomic the “Edition” deliverable, as per that Council approval.
  5. The Cloud Base Image almost stopped getting made entirely, except Dusty kept doing it. Thank you Dusty!
  6. Red Hat bought CoreOS and canceled Project Atomic. Uncertainty abounds.
  7. Fedora CoreOS replaces Atomic Host, but edition status kind of wobbly.
  8. Fedora IoT exposes the fact that we didn’t really have a good formal promotion process so we developed that (thank you Adam!). See Issue #296: Decide, plan for and clearly communicate any changes to primary Editions and/or release-blocking deliverables for Fedora 33 - tickets - Pagure.io and the result Process for promoting a Fedora deliverable to Edition :: Fedora Docs
  9. CoreOS Edition status still pending (see incomplete Change Changes/FedoraCoreOS - Fedora Project Wiki), so we’ve been calling it an “Emerging Edition”.
  10. Meanwhile somewhere in there, new interest in original Fedora Cloud has (obviously) resurged and awesome stuff is happening.

Details may be a bit fuzzy, really. But that’s the overview. And, I think more important than the details, here we are now. I admit I find it fun to spelunk through the past and discover what things we’d written down in a useful way and what things I just remember and I’m going to have to ask you to trust me on. But, really, I think there’s only so much to be gained there, and a lot more to be gained by figuring out what we want to do next and how to get there.

1 Like

Define formal. Because I don’t see a formal vote by Council in that ticket to either name change Cloud WG into Atomic WG or dissolve Cloud WG. The formal action, i.e. what the ticket says, looks indistinguishable to me from a new Atomic WG with imprimatur from Council.

Council needs to provide clarity, without delay. If only Council could act on this inside of 20 minutes, seeing as after 4 years it was suddenly so urgent to figure out as a Fedora 35 blocker. :confused:

Chris, I don’t think that’s necessary. Nothing in Fedora has “imprimatur”, really. (As Adam Williamson once famously told me, I am not the Pope.) We, Fedora, area almost never that formal. We’re getting better at recording important decisions in a way that we can look back on and say “yeah, that official” (or, really, more importantly I think, where we can look at the reasoning), but there are hundreds of important decisions about both project structure and engineering that are at best in some meeting log somewhere.

When something is contentious, the bar is higher. We try to make sure to have all sides heard and document the whole thing more clearly. That’s a crucial part of Fedora decision making. We don’t always do that perfectly, but I’m very conscious of the need to continually improve at it. But at the time here, there weren’t any sides. Just the one group, and the Council validating the decision. No one then anticipated the situation we have now, so there was no real reason to do more.

And, to be clear, miscommunication aside, I think the situation now is good. Back then, basically no one cared about the non-container-focused Cloud Image enough to do more than keep it on life support. But now! We have so many people here who clearly care and have big plans. It’s awesome! Let’s focus on that.

I’m absolutely certain I could get the current Council to look at this and say, yes, sure, here’s an official seal. But I’m opposed to doing that, because I really, really, really do not see any value it would add. I don’t see how it matters either way except to fill some theoretical requirement for extra process. Let’s figure out what we want now.

I’m honestly not sure what you’re getting at here. As the proposed blocker bug noted, the Cloud image had incorrectly identified itself as an Edition for a long time. That is factually correct; there was nothing to figure out. That’s also a violation of the given release criteria; also a plain fact with nothing to figure out. The question was “should we block the whole Fedora Linux release to change this, or should we waive the blocker”… and that’s what needed urgent response since the blocker was proposed the morning of Go/No-Go day. And that was a pretty easy waiver: it’s been fine with nobody really noticing for years, so seems pretty silly to hold up anything for. No change necessary for the release.

But, it does highlight that we should formally plan for what we want to do with Fedora Cloud in the future. That’s what this thread is about!

At least for me it is a valuable information. It provides a solid basis for decision making and courses of action. Unfortunately, the discussion around and about cloud is not always strictly rational, as I had to experience myself in the past.

And, in the spirit of following my own advice: I think, at this point, there is a quite compelling case to make sure Fedora Cloud image in its traditional form is well-represented, well-maintained, and well-presented to users. That’s because:

… it represents 15.2% of all persistent (installed for more than one week) Fedora Linux 34 systems observed to date, and …

… a whopping 30.5% of the “ephemeral” systems. (Those that show up initially but never report as being installed for more than a week.)

So, that’s a lot! It’s important! I mean, it’s obviously greater than any number of systems from any other deliverable we have, but it’s also greater than all of the non-Workstation desktop releases put together, even for persistent installs.

(Note that CoreOS, Silverblue, and IoT numbers are low because they just started getting the countme feature. They will be higher for F35 numbers.)

The huge difference in persistent vs. ephemeral for Cloud and Server does speak to a different use case for sure. But that 15.2% persistent systems makes me wonder still. Is it really better for those to be Cloud, or should we bring some of that cloud knowledge and expertise into the Server Edition? (Likewise, 5.7% of ephemeral systems are Fedora Server Edition? That’s a lot. Might those be better steered towards Cloud?)

Here is where I really need your expertise. I was really deep in doing cool stuff with Cloud at my previous non-RH job. When I joined the company, I was, like, OMG all these people who have worked at RH for 10 years have no idea how the computing landscape has changed. Now, I am those people. So, I need you to help me understand how the persistent systems installed with Fedora Cloud represent use cases different from those outlined for Fedora Server. (Which, really, is no slouch at these levels; Fedora Server is always going to be in a unique position against LTS/LTM distros and that’s fine.)

Or to put it another way, I actually fairly frequently get the question: “I don’t get it — why isn’t Fedora Cloud just, like, Fedora Server available to me in Amazon / Azure / GCE / DigitalOcean?” Help me answer that question with confidence.

As I read through this thread, it appears to me there are two main ideas presented:

  1. A very detailed history lesson of how we got here.
  2. Those who work on and use the cloud build artifacts do not see Fedora valuing/promoting that work in the same way as other build artifacts.

Just scanning our web sites and documentation, anything we label is Cloud is inconsistent when compared to Server or Workstation. And we know how we got here. To me this shows that we have learned a lot since Fedora.next was first announced and we as a project are in a much better position to define use cases and build artifacts.

Speaking as a Council member, I do not understand what a Council vote would gain us now except to say “yes, it’s time to look at the Cloud builds again”. We already know that, so let’s make that a focus for Fedora 36 rather than rehashing the past.

What I would like to see for the Fedora 36 cycle is for the Cloud group to:

  • Evaluate the group’s name and change it if needed.
  • Evaluate the group’s scope and targeted use cases, revise those if needed.
  • Define the non-Server and non-Workstation builds to deliver in Fedora 36.
  • Work with infra, docs, and other teams to get consistent naming and terminology across Fedora. Just like you can say Server and Workstation now, we need new well-defined terms for anything from the Cloud group.

There is probably more here too, but I guess my main thinking here is let’s make the changes we want to see based on what we have learned.

3 Likes

Oh, this reminds me – maybe too deep of a tangent, but: at some point, in the very distant past, the Cloud SG agreed* to call the cloud image officially Fedora Cloud Base image. Since then, Edition or no, we have been horribly inconsistent with “Fedora Cloud image”, “Fedora Cloud Base image”, “Fedora Cloud Base Image” (capital i), “Fedora Cloud Base” (no “image”), and in at least one unfortunate instance, “Cloud Based Image”.

We should definitely take the opportunity to clean that up — pick and actually consistently use a standard name. At the time, we were contemplating a lot of “cloudy” images, and we wanted something specific, and we felt that “Base image” conjured up the desired idea of “this is a minimal image for building on”. I am not at all attached to this nomenclature.

* note: URL contains “atomic-wg”, but at the time it was actually “cloud”; the pagure repo was renamed as part of all of the above.

Re: history

I definitely think there is value in understanding how we got here and why, both to ensure everyone’s on the same page, and to troubleshoot what went wrong so we can strive for improving process in the future, and hopefully avoid ending up in similar situations. Thanks Matthew (and everybody else) for taking the time to dig up references and provide context around past decisions. I do agree that at this point we should focus on the future, and that is Fedora 36.

Re: naming

I’m partial to keeping Fedora Cloud called as it is now, mostly because I don’t really see what the benefit would be in changing the name. We should definitely clean up and update the documentation so that it’s used consistently everywhere and strive to clarify possible ambiguities.

The way I see it, Fedora Cloud is the Edition and the Fedora Cloud Base image is one of our deliverables. There’s plenty of other deliverables we could produce to cover more specialized usecases and demonstrate best practices if there’s interest.

Re: why Cloud and not just Server

I touched on this before, but fundamentally Cloud and Server are different products targeting different usecases. We’ll put together a proper writeup on this, given that it seems one of the main points of confusion.

1 Like

Yeah, cool.

I’m open to some format other than “PRD” for this, although there are some essential elements of that I’d like to keep. I’m going to create a new thread for anyone interested in diving into the details of what this should look like going forward.

And here is that thread:

Hmmm. I either don’t understand this, or I disagree, or both. :slight_smile:

To me, an Edition is a deliverable. To resort to the dictionary, “an edition = a particular version of a product”. An edition with multiple deliverables doesn’t make sense to me. A Team can have multiple deliverable, one of which might be an Edition.

You said earlier “I see an Edition as a primary deliverable for Fedora that addresses a board market segment.”, which agrees with my understanding.

But I am not sure how to put the two together.

Ah, so in this example, the Server WG produces the Server Edition ISO, as well as the Server netinstall ISO, and the latter is not an edition?

I’m not sure if this strict “an edition is exactly one deliverable” make sense for the Cloud team (WG/SIG) though since there could be several very closely related images just for targeting different cloud providers?