Proposal: Build ROCm Test Subpackages by Default

This came up in the review for rocFFT but I probably should have started a conversation here first.

For most of the ROCm packages, we have the upstream test suite in a subpackage, disabled by default. As the title of this says, I want to start changing that to building the test subpackages by default for ROCm packages.

My argument for this is mostly in the realm of efficiency:

  • building test suites generally requires rebuilding the entire project which can take over an hour
    • extracting just the tests doesn’t seem to help due to dependencies back into the main codebase
  • packaging the tests @ build time in koji doesn’t add much to the overall build time
  • adding build time on to every test we run is going to add up very quickly
  • running tests can be a useful diagnostic tool for users

I’ve spent the most time with rocFFT where all of these are tru but from what I’ve seen, the other ROCm packages are similar in behavior for most of the things I listed. If you know of exceptions, please let me know.

What do others think? Are there reasons to not start working towards this?

1 Like

Most/all of my ROCm packages have an optional -test package. So we have a good start. My concern is the set of people that could use -test are those with ROCm hw. If we want -test to be a default koji option, koji needs to use the -test packages. That will not happen until Fedora’s infra has a set of ROCm hw for the GPU’s we build for and the c/i infra to use it. Just installing the tests is also tricky, because running the tests sometimes assumes you are running from a magic build or source directory. Making this work robustly across the growing set of ROCm packages is a lot of work that I do not think is worth doing if in the end it will be only 1 or 2 people running these -test rpms.

I don’t understand - why would koji have to use the -test packages if those -test packages were built in koji? I can’t see GPUs being added to koji builders any time in the near future, if ever. As far as I know, building tests for ROCm components doesn’t require a GPU; you only need a GPU to run the tests and the time required to run a build with tests enabled isn’t meaningfully more than just running the build.

I recognize that the -test packages will have a limited audience and it’s possible that there will be ROCm packages which have so many problems with their tests that it’s not worth enabling in koji. However, I fail to see how that invalidates any of my reasons for proposing this. If individual components end up being a problem, they can be special-cased and worked around.

I’m trying to put together a setup to informally test the ROCm pacakges in a ci-ish fashion. Ideally, automated tests will be run whenever there are changes in related packages (kernel being the first one that comes to mind). While your proposal of “don’t enable -test packages until there is a CI system using them” is one approach, not having the -test packages enabled makes it harder to set up a CI system so we end up with a bit of a chicken-and-egg problem.

I’m setting up the testing stuff either way once I have access to more hardware. If I have to set up a COPR or local repo to build with the test packages enabled and test against those builds instead of what’s coming out of koji, so be it - that’s still better than no testing at all.

To be clear, I’m not demanding that someone else do the work. I’m well aware that I’m probably going to end up submitting PRs myself if I want to see the -test packages built in koji. However, I did want to talk about this before just starting to submit changes.