Are the plans for CI documented anywhere? I.e. will Fedora CI be migrating to Forgejo actions, and will build status be available through the actions UI? If forgejo actions is chosen, will a lack of native support for K8s be an issue, or is dind (Docker-in-Docker) suitable?
I also noticed a portion of the user stories were answered with renovate as a solution, but ad far as I can tell, only their CLI is open source, while their on-prem solution is not. Does this pose any problems? Will packagers need to introduce CI scripts to run renovate on every repo, as forgejo themselves currently are? On this topic, are there any plans to further reduce maintenance burden by providing .spec file support into renovate, thereby elimating the need for the new hotness?
Also for Packit there is a fedora change proposal coming up, which should document these plans.
As for Fedora CI we are dedicated to have at least the experience there is now for pagure, ideally better with better integration support for third party CI apps.
“CI” in Fedora is a lot of different things — Testing Farm, OpenQA, the zuul gating. I don’t think we’d want to try to migrate all of those things.
I don’t think that we’ll have resources for heavyweight tasks as Forgejo actions. Instead, actions should be small, with any serious work farmed out to another place.
What I would like is for coordinated packaging and rel-eng workflows where all of these different tools interact with pull requests in a consistent and helpful way. Packit is a very likely part of this.
I was wondering the same thing, as for the Fedora Websites project on Gitlab we used Gitlab Pipelines heavily and for everything. Project Bluefin uses Github Actions for everything.
This is apparently a pretty common question they get. On their FAQ, they say it kind of like for-jay-oh.
I don’t really know a lot about Fedora’s infrastructure, but from what I can tell, their actions runners do not natively execute on k8s. That is, they run by connecting to an instance of Docker or Podman, instead of orchestrating a set of k8s pods like GitLab does.
I really hope a way is found to reduce the amount of Fedora-specific tools (and tools in general!). There’s so many moving parts already. I know there was some discussion about it, but I’d love to see it expanded on a little more. I mean, just look at this discussion. We’re 7 messages in, and we’re up to 3 different git forges used within the project!
I am also interested in having some sort of Github Actions– or Gitlab CI–like CI implementation in the new forge. The Fedora Go SIG has multiple projects hosted on gitlab.com that use Gitlab CI that I would like to move over to the new forge instead of relying on a not-so-open-source offering. I’d also like to move fedrq and forge-srpm-macros (other Fedora projects that I maintain) to our new forge, but I cannot do that either without a solid CI offering to run things unit tests, integration tests. linters, and other CI jobs that are not provided by Packit.
I am also interested in having some sort of Github Actions– or Gitlab CI–like CI implementation in the new forge. The Fedora Go SIG has multiple projects hosted on gitlab.com that use Gitlab CI that I would like to move over to the new forge instead of relying on a not-so-open-source offering. I’d also like to move fedrq and forge-srpm-macros (other Fedora projects that I maintain) to our new forge, but I cannot do that either without a solid CI offering to run things unit tests, integration tests. linters, and other CI jobs that are not provided by Packit.
@gotmax23 we will be able to run all of that via Packit + Testing Farm. I think we should document a bit how to migrate from Gitlab CI jobs to tmt plans and tests. But basically, it should be fairly straightforward, maybe except dependant jobs. Is your setup somehow complex? Maybe we could show you how would that look like if you point us to your setup. In general you can execute any script as part of Testing Farm, we use it as a general job executor in various places, not just to run some tests. Note that Packit and Testing Farm do not provide tests, it is all defined by you similarly to how you define GitLab CI jobs.
In the current pagure deployment, we do similar things with Zuul + Testing Farm, but Packit should improve that and make that on par what we do on GitHub for the projects.
I would like to be the voice of the lazy developer here:
Personally, I’m less motivated to learn Fedora-specific frameworks in order to contribute. I want to contribute, not learn a new technology to do so, THEN contribute.
A lot of developers already know standard CIs/action runners, of which Forgejo has a pretty good version IMO. I’m not super motivated to learn Packit/Zuul + Testing Farm, same as I didn’t love having to learn the idiosyncrasies of Pagure before it, because this knowledge is not transferable.
Nothing against niche and/or custom tools, my point is just that standard contemporary tooling reduces friction and would (for me) increase contributions as it did on the Fedora projects that run on gitlab and github (websites, mobility, atomics, etc).
Sorry to be a bother and thank you for the consideration!
I do think large parts of the custom Fedora tooling are worthwhile, like everything that has to do with gating/updates is rather nice. I think the test stuff, like tmt, is niche but not necessarily Fedora-specific. I think it’s nice, too.
I definitely believe things are a bit too scattered though. There’s projects everywhere across so many forges (in fact, it’s hard to tell what belongs to Fedora and what doesn’t), although moving to forgejo is definitely a great start!
I agree. GitHub action-like stuff (in GitHub or Forgejo) is well-documented, relatively straightforward to set up, can reuse existing pipeline components in a modular fashion, and is familiar to anyone who has previously used GitHub or Forgejo actions. The zuul / Fedora CI / etc. NIH implementations are none of those things. It would be a shame to move from a custom forge to one that is more popular while simultaneously adopting a custom CI instead of the “native” one.
WARNING: this is alpha release quality code and should not be considered secure enough to deploy in production.
), so we might want to consider Woodpecker CI which is what Codeberg (the largest public Forgejo instance) has deployed in production. It integrates well with Forgejo, and I think it has more deployment options than Forgejo Runner.
Indeed. TMT has its own unfamiliar constructs (tests, plans, tiers, discover, etc.) that I still barely understand after having added distgit integration tests to some of the Ansible ecosystem packages. Some of this could probably be solved with documentation improvements and more configuration examples, but I feel Github/Forgejo Actions’ (and Woodpecker’s) syntax is more accessible and also supports additional features like matrices and secrets.
That’s good to know. OTOH, RH uses Jira and GitLab (I think), and both are ruled out for Fedora, which limits the argument to some extent.
I’d say that CI hookup/config is necessarily forge specific in order to integrate well; while config of the actual tests being run should be about as flexible as %check in spec.
well, it rather enhances the argument, I would say. if we used the same forge, arguably something like tmt wouldn’t be necessary as we could just duplicate the CI test config. but since we don’t use the same forge, we need something like TMT in order to share tests. otherwise RH would be building all its tests as Gitlab CI…whatever Gitlab CI calls them, and we’d have to somehow convert that to Forgejo actions if we wanted to use them.
this is all still under a lot of discussion, though, I don’t know if anyone has a totally clear idea of how we expect this to wind up looking once all the dust settles…
Yeah, after playing with forgejo actions a bit, I still run into cryptic errors, although I can’t tell if it’s because of their act fork, or if it’s a general problem with docker in docker (or I guess docker on podman in my case). More importantly with the “alpha” label though is the fact that it is pending auditing.
There’s also open discussions on “how to make forgejo durable for the next 10 years”, as well as how to do STS/LTS releases.
In general, it seems like forgejo itself is still trying to figure out what exactly they’re doing with actions. Now looks like the best possible time for feedback.
I don’t care which CI it is as long as its runner can run natively on local PC. Duplicating CI setup in docker/podman for local development and vice versa is a waste of time.
Let me share my thoughts on the CI workflow. Basically the technologies should be ultimately used together as each one cover a different aspect
Forgejo actions/woodpecker CI, etc.
Scope: Control the overall CI workflows:
steps used in the workflow: build rpms, run tmt tests, run general tests like installability
define the dependencies of each step (early exit if rpm build fails for example)
define the overall pass/fail state
connect inter-dependent PRs to build in a temporary copr repo or side-tag after merge
Limitations:
Forge dependent so it cannot be shared across other backends like RH’s Gitlab CI or upstream’s forge
Locally built resources like an RPM is not easily™ available for external tools like testing-farm, bodhi, etc. or cross-PR
Packit
Scope: Glue services together and provide a shared build resource (copr/koji artifacts)
Communicate with koji or copr to request builds and report back in the forge as an action/workflow status
Report back the built resources for testing-farm or other tests to run on top of it
Allow upstream to replicate and integrate their workflow to the packit service
Limitations:
No inter-PR support, particularly for ephemeral (one-off) side-tag build. This could be supported though with a zuul-like interface of Depends-on comment text
Does not do any CI steps itself, it just orchestrates with other services to request, wait and report back build/test results
Testing-farm/tmt
Scope: Define and run actual tests, e.g. installability, rpminspect, etc.
Common test definition for upstream, forge, RH, etc.
Limitations:
testing-farm/tmt does not do any workflow gating like having an experimental/flaky test that is allowed to fail
it does not build the artifacts, it just consumes them
Technically, sure we can implement one service like Forgejo Action to handle all of these steps, but it would be hard to share the build results and workflows. This also goes for packit and tmt projects as well, e.g. packit should not define a Github workflow-like interface to define dependent steps so that it can exit early. Instead we should try to work out how to best combine the current strengths of each of these projects and patch up any missing functionalities, like the suggestion to duplicate the CI for local development.
This should also not limit the ability for developers to tinker with tests in the CI system that they are familiar with. E.g. writing the initial tmt test can be unintuitive, but writing the equivalent in Forgejo action is accessible to others. So if calling packit is implement as a Forgejo workflow, then the developer can simply reuse the artifacts there and implement their own steps like dnf install, pytest, etc. This could of course mean that something that is easily done in tmt is reproduced with more steps in Forgejo Action, but this would also serve as examples of what workflows are needed to support, what to document in tmt as equivalent Forgejo workflows for reference, what are some missing features in Forgejo, packit or tmt.
