Fedora Hummingbird: Taking the Hummingbird model to the full operating system

News & Announcements Fedora Magazine
41

In addition to the missing bib-config.toml file mentioned in the Getting Started section of the article, I also see that the steps have changed slightly. It looks like the image builder no longer pulls images for you when you execute the first step:

bootc-image-builder no longer pulls images, make sure to pull it before running bootc-image-builder:
    sudo podman pull quay.io/hummingbird-community/bootc-os:latest

At least the error message explains how to correct it but, it may be good to update the “Getting Started” section.

42

@nhr: Would you like me to add sudo podman pull quay.io/hummingbird-community/bootc-os:latest to the first step in the Getting Started section of your article?

43

If it means anything, I had initially assumed this blog post was a mistake (see upthread where I kind of made a fool of myself) since Council usually is transparent with its decision-making and I did not see a public topic about it.

I don’t have mistrust toward anyone involved with Hummingbird. I like Fedora and they want to make Fedora better, so I’m happy to collaborate, figure out any issues that may come up, review the eventual Change Proposal, etc. It doesn’t have to be that deep :slight_smile: [1].

  1. Okay, maybe it’s a little deep :wink: —[2] I still wish the decision-making process was more transparent here, and I understand there were $REASONS for that not happening, but that doesn’t have to taint the Hummingbird+Fedora integration work itself. ↩︎

  2. This em dash was also typed by a human and not an LLM (Ctrl+Shift+U 2014). ↩︎

2 Likes
44

Since the folks working on this likely have their hands full until Thursday, I think it is safe to say, yes please! :folded_hands:t2: I don’t think that’s controversial since it is in the warning message anyways. We can always change if it should be something else.

Hey, no, that’s not on you. If I were in your shoes, I’d probably feel the same way. We have a process for a reason and I think it is totally reasonable to have assumed there was some issue here. In an ideal world, everything would have been public from the outset. But now, things are public and open, and as with any effort, the usual rules apply from here on out.

I’m thinking we could probably review the trademark policy to at least make it more obvious that not everything will be public at the start. I was thinking about this today, and if I remember right, the Framework Computer folks asked us privately for a trademark approval on an upcoming pre-release product they were working on at the time. I could be misremembering, but I think they opened a private Fedora Council ticket which we later made public when the embargo was cleared.

Thanks @gotmax23! I appreciate that a lot. :folded_hands: I fully expect there will be future work for FESCo to review from the Hummingbird folks, as things integrate more with Fedora proper instead of being a periphery thing. We’ll get there! :flexed_biceps:

2 Likes
45

Thanks Justin,

I agree with everything Justin has said here. No notes from me.

I’m looking forward to the Hummingbird SIG meeting at the end of the month.

3 Likes
46

Isn’t this entire thread proof that an “open, public way” is exactly what has not happened here?

It’s less than three weeks since the story being presented in email to the devel list was that they would like to “explore bringing this work more formally into the Fedora
community” and now it’s a done deal?

In reality of course there’s no way they weren’t already planning the big summit announcement when they sent that email so presumably it was just an attempt to create some sort of fig leaf of community consultation without blowing the “big bang” announcement.

3 Likes
47

I agree with @tomh . It was a minor case and without bad intention, but effectively, the community has been misled, including those who had been involved in getting this forward with the assumption this is just to get the SIG started. Even if it was just 10 minutes or so someone is involved to create a tag in a category - minor thing, minimal work - but it’s still making the community work for what still is a RH project. Community channels were only used to implement a decision done effectively by RH. This is not a way to start gaining trust.

It is hard for a big hegemon (which RH is in this community) to avoid imposing its preferences intuitively on the community by accident: it needs careful consideration of implications and precise planning and communication to ensure such an accident does not happen. The Council is likely to be crucial in this process: translating, explaining, filtering, feedbacking and “firewalling” (not in a hostile sense!) between RH and the Community: yet, as far as I have information (which is not much: thus this is only an assumption!), the Council seems to have acted as a passthrough entity in this case, and did so with one-way information provisioning most of the time. When information provisioning takes place matters.

I have not sufficient information to say what and where an error has happened (which in a community that lives from transparency is already a flaw in itself), but at first glance, I am not sure if RH has failed, or the Council?

The major problem: if that is a new norm, the Council became unpredictable for the community members who ain’t involved in internal RH information flows, and predictability is the core of governance towards those who are governed (at least if they want to be accepted+treated as the governors). Where is the boundary, the limits? Referencing the rules doesn’t help, as they give the Council more or less “unpredictable amounts” of power, that used to be balanced by transparency except in security/privacy-sensitive cases. That does no longer apply?

Again, this is a minor case, and I do not expect any bad faith or bad intention on any side, and I like the Hummingbird project to be integrated as far as I see it at first glance. It is more an underlying issue that I think should not be “just dismissed”, also for the sake of the Council.

4 Likes
48

I still got the same errors today (If i used the link in a private new windows, I had the choice to accept a ton of cookies), and the link seamed to be the same. As already said, bad prepared announcement with a lot of “yes, but …”

The foundation for Fedora Hummingbird already ships today from the Hummingbird containers repository. You can pull and boot it right now.

I fixed the link direct in the Magazine on my own :roll_eyes: .

2 Likes
50

I’d say this is sort of a “different perspectives” thing.

You’re looking at it from a “the alternative was everything being discussed out in the open from day one” perspective. Justin is looking at it from a “the alternative was Red Hat deciding this didn’t need to happen in Fedora at all and doing an end run around us” perspective.

That’s why to Justin (and, honestly, me) this is on the good end of the possibility spectrum. I’m happier that RH is trying to do Hummingbird via Fedora - yes, even with the initial, probably-unnecessary secrecy - than just doing it independently (which runs the risk that, in a few years, RH turns around and goes “and why are we even funding this Fedora thing any more?”)

From inside Red Hat, especially if you don’t work on Fedora, it is possible/tempting to look at Fedora as a source of strife. It’s got all these people in it, with opinions, who aren’t on the payroll! You can’t tell them what to do or think or complain to their manager! They have eternal arguments about everything! You have to write a wiki page and convince some person you’ve never heard of that your idea is good! Who needs this?

So we’re kind of constantly fighting a tendency for RH to just spin stuff up in channels it 100% controls, which tends to seem easiest at first then turn out to be a mess after a few years.

I agree that ideally this should’ve just been spun up openly from day one, but I also get where Justin’s coming from in thinking it’s a lot better than it could have been (it could have been RH announcing “OpenBird” or something and dumping some repos on GitLab).

5 Likes
51

The stuff from the devel list discussion is still all true, really. Ultimately all that’s been “approved” is that there will be a thing called Fedora Hummingbird (exact shape TBD). It doesn’t get to be release-blocking or an Edition or anything like that without going through the usual processes. Right now it has no more significant status than many of the other dozens of “Fedora somethings” listed by @augenauf above.

The idea here is genuinely to try and build a thing that people will find useful/valuable. It’s not just openwashing for a “RH project”. As per my other post, RH could easily do that itself if it wanted to - RH is behind plenty of upstreams that are entirely independent of Fedora already, e.g. OpenShift and Foreman. From inside RH it can be seen as easier to do that than go through Fedora processes.

52

I think everyone made their points and it might be useful to avoid ending in circles. I would like to add a perspective on one of your justifications though. Not to disprove your point, but maybe highlight how dangerous its justification is, and that it maybe could achieve the opposite of what it was actually intended for.

In the context in which you made it, doesn’t this justification inevitably lead to Fedora being step by step transformed into a corporate unit of RH and behave as such?

I mean, if RH no longer finds it useful to keep funding Fedora and compares it, as you indicate, to its integrated alternatives (in which people “can be told what to do”), and uses the performance indicators that are usual in integrated alternatives, then Fedora cannot compete until it reached the point of being identical, right?

This would mean, that the transparency institutions of Fedora (to stick with a simple isolated example) must be step by step replaced by secrecy institutions in order to protect the transparency, which at some point will no longer exist anyway.

At the same time, the Council would more and more decrease the likelihood that any other organization would take over, as when it becomes apparent that we need to find alternatives, the Fedora governance is already so unpredictable from the outside and integrated into RH that no other organization would touch it.

With this in mind, would it be not more useful to let RH cut funding early and when it is rationalized as such (in contrast to a subtle non-rationalized non-discussed process that might became irreversible when it becomes apparent), to find alternative funding to take over? I admit, I am not sure if the Council as it is would be qualified to do so, as I am not sure if they are already too deeply integrated in RH to be considered neutral by others (this is not accusation, but might be also the result of circumstances they are not in full control of, and this is just a weak assumption based on many assumptions and limited availability of facts).

IF Fedora is important for this eco-system, one can trust that the Linux community as such - as with Alma and Rocky back then - finds a way to mitigate and fill the gap. Maybe someone takes over funding (maybe even more than 1), maybe it is split among other communities, maybe something new rises, maybe something else. Otherwise, then so be it, but then it is at least a rationalized process that everyone can respond to. IF not, the world will survive too I expect :classic_smiley:

This also implies that your justification enters the realm of “what if” → it can effectively justify anything. But immediately, it has made it unclear to the community when the Council acts as agent of RH and when as agent of the community. This can be a serious issue, and already questions what the community is, and if and what is the actual leadership.

I make these points not with relative ethics in mind but static correlations: communities in the Linux world are key facilitators of knowledge creation and transfer. This massive problem solving capability usually correlates with certain institutions. And if the Council replaces them, the actual value Fedora used to create will cease to occur anyway, long before a funding ceases.

Also, I agree that this project must be a synergy, and offer a well return on investment for both sides: if this is no longer fulfilled for RH, I find it ok to leave, but making it clear to do so and seeking a handover (even if it is just to retain their reputation for OS projects), rather then to end up in subtle developments in which the community is not even involved when it gets step by step transformed…

This is just a (partially quite radical & surely massively simplified) perspective that considers a potential development, not arguing this must apply just because of this one case, and it is only one interpretation of the reality, next to other realistic interpretations. But maybe worth to be kept in mind when evaluating cases like this or using justifications like that … :classic_smiley:

1 Like
53

I’m fairly confident OpenBird was not under consideration. But yeah… this.

54

A post was split to a new topic: Problems with setting up Hummingbird on Fedora

55

I don’t think it needed to be open from the beginning if there were reasons not to have it open.

What was needed was that Fedora announces Fedora Hummingbird, at least in parallel with RedHat’s announcement. After all, Fedora was involved through the Council who can act on Fedora’s behalf. RedHat cannot.

What threw me (and probably others) off was that RedHat announced what Fedora was going to do. The info about Fedora’s involvement in Fedora Hummingbird came only after the big backlash, which is too late.

So, communication timing (and wording) was really unfortunate here.

Plus that devel-list post which must look strange in retrorespect.

In that context, I found it interesting to listen in on bits from the RedHat summit. It became crystal clear that “RHEL as platform for everything” (in particular AI; AI; have I mentioned AI?) and “zero CVE” (which can never be true, unless every CVE comes with a mitigation) are the two main buzz word thrusts for selling the platform, and that explains two of the current initiatives. (Hummingbird looks cobbled together in dire need for something covering the “zero CVE” buzz, IMHO.)

There’s a third one that’s being emphasized, though: sovereignty. No, not Fedora’s sovereignty … but hearing Nvidia talk about the importance of open source for sovereignty over the platform was interesting, too.

2 Likes
56

I get your point, but from my perspective it doesn’t feel like exactly this. This isn’t a new dynamic - it’s been going on nearly as long as Fedora’s been around. I’d say it’s more like an ever-present tension than a slippery slope.

The way Fedora can “compete” is not by turning into the thing some parts of RH might superficially think they want - an upstream project that’s open source in license terms but closely-controlled in governance - but by being the thing they need - a loosely-coupled upstream project with passionate and involved people who will make things awkward and uncomfortable at times but usually produce a better end result over the long term, and act as a useful check on RH’s perceptions at times.

We (RHers who care about Fedora) have to keep selling this vision and making sure it actually works, but there isn’t an inevitable one-way slide into the abyss, it’s a more complex dynamic.

2 Likes
58

I’ll let someone else speak to how Hummingbird is built (I think the actual build pipelines are 100% deterministic, but IMBW). But for classic Fedora, you can in fact see how Fedora is built. It’s all open source. You can’t shell into the actual build systems unless you’re in the infrastructure team, but that’s all.

The best entry point for understanding how Fedora images (except CoreOS) are built is https://forge.fedoraproject.org/releng/pungi-fedora . That’s the repo that contains Fedora-specific configuration for the Pungi tool that runs Fedora composes. Mostly what Pungi does is create jobs in Koji.

There’s a shell script in the pungi-fedora repo which is run every day (by this cron job) and runs Pungi to create the day’s Rawhide compose. Those composes wind up here; there’s a symlink for the most recent one. In each compose there is a global log file which logs Pungi’s overall execution and from where you can track most of the individual tasks it kicks off. You can track from there to the individual Koji jobs. There are also various other log files you can poke at, I won’t try and explain them all right now.

The individual image build jobs in Koji mostly use Kiwi these days. Fedora’s Kiwi job templates are here. Some still use Lorax; the configuration for that is kinda spread around the stack. Koji’s source is here.

Basically all of this stuff is open, you just have to know/figure out where to look. :smiley: You can, if you figure it all out, stand up your own Koji and duplicate the entire compose process. It’s somewhat easier to replicate individual image builds, especially with Kiwi.

8 Likes
59

Is there a doc anywhere that can get an interested user started on this?

If not, then just pasting your post into a doc would be a great start!

60

This might help to import it this way: https://discussion.fedoraproject.org/raw/191184/58

62

Unfortunately not really, no. It’s difficult because it changes quite constantly. A couple of cycles ago I’d have had to explain livemedia-creator and the spin-kickstarts repo, for instance, and a few cycles before that the word “Kiwi” would not have appeared at all.

The skeleton of a doc that does exist is an excellent illustration, because it is wildly outdated…that’s sort of the inevitable fate of such docs :confused: I can try and give it a quick glow-up, though.

3 Likes
Custom live iso F44?
63

There is no one doc, but RelEng has a set of SOPs that describe each step during the release. It describes each step in chronological order and changes to the configs that Adam mentioned in his post.

It might be a good idea to redo the process landing page to describe the dependencies between the tooling and the different steps a bit more clearly. WDYT?@jnsamyak

2 Likes