Sorry, coming late to the party but I’m not using Discourse myself (so just discovering this thread as Adam mentioned it to me today )
We can eventually discuss the implementation detail in the Jira ticket ? That would be better to track there.
Can you confirm though that we can do something like this :
jenkins provisions a VM (let’s start with a x86_64 VM first)
once installed we just install tmt inside
it’s then pointing to a git repository containing the test and we should just know how to launch the test from inside the provisioned {VM,bare-metal} host.
Once I’ll have confirmation, I can easily plumb a parallel test in testing.stream.centos.org and we’ll have automatic run after each new Stream compose.
One thing: We need to agree from what repo we’ll pull the test suite. Should I make one in GitLab under CentOS Stream? Or do we want to use a repo in the Integration SIG namespace?
If @carlosrodrifernandez agree, I’d move the repo to the Integration SIG GitLab space.
I gave Carlos permissions to manage/merge MR in it as a member of the sig-integration group. And think it would be easier for us to collaborate there.
seeing this run, did you consider using Testing Farm for execution of the tmt tests? I believe it could spare you some plumbing code. If yes, was there some obstacles you encountered that you decided to run tmt directly?
In Fedora CI we have a quite solid scalable Jenkins setup using webhook_step Jenkins plugin.
@mvadkert , I don’t believe there is a technical obstacle forcing the local provision in the tmt run. The local run is how the existing t_functional infrastructure is setup, which involves all the mechanism for the promotion of composes. So I started the PoC assuming local would be the easier route to test it out, but I may be wrong since I don’t know the details how things are hooked together after the tests are run.
I think using one of the remote tmt provisioners for the run, leveraging that infrastructure might be better for test isolation, simplification, and less dependencies to be installed in the tested system.
My knowledge is limited on how things are setup beyond the tests themselves, though, but just wanted to give you some background about the local run thing.
@carlosrodrifernandez I see, so this is like a minimal change to switch to executing tests in tmt format in your current setup. Understood.
In case you would want to delegate that execution in the future to Testing Farm from your CI, we are here for you. Basically, all major users of tmt run tests in scale via Testing Farm. They do not run directly tmt.
As I understand it, Testing Farm only supports x86_64 runners, am I right? Our current setup covers aarch64, ppc64le, and x86_64 in VMs, and we have an x86_64 bare metal runner as well.
If we want to get this in production properly, effectively replacing t_functional, I think we should:
Let the PoC run for some time to see how it does over time
In the mean time, work on contribution guidelines and things around it like:
who owns the tests - is it the CentOS Stream team in Red Hat (my team)? is it the Integration SIG? My goal here is a smooth way for people to contribute, and a reliable way to get tested composes out. I’m open to both.
what are the criteria of getting a test in?
how would we treat test failures? Would we block the release? Would we re-evaluate the test? etc.
Agree on the repo location — in my head that’s a new repo in GitLab under the CentOS Stream name space. But I’m open to other proposals.
Ensure it properly covers CentOS Linux 7 and CentOS Stream 8 and 9 — so we don’t lose functionality copared to t_functional
For ppc64le the only possibility is internal testing (triggered from public), we do not have access to ppc64le in public.
As for bare-metal support, same story with ppc64le. We plan to enable bare-metal support on aarch64 (as it is fairly cheap in AWS), but x86_64 is very pricey, but we can talk about it. I expect that testing is limited?
Of course, if we could have access to your infrastructure, we could implement support for everything you need. Can you share light on the infrastructure you use now for bare-metal and ppc64le?
We should be able to run it in those as well by installing tmt via pip as @mvadkert mentioned in the ticket. I’ll run the tests in 7 and stream 8 to validate them.
Actually, it requires dataclasses which it is only available in python 3.7 or greater. CentOS 7 only has python up to 3.6. But, if we can install python 3.9 on it…
I ran the tests from a Fedora machine against a CentOS 7 machine and they were passing, except one, but I had to kill it to give more power to the machine. This method didn’t need python 3.9 or tmt installed to start. I did install beakerlib ahead of time.
I also installed Python 3.9 in the same CentOS 7 to run them with the local provision, and it is doing the same, that one test is failing but the rest are passing. I’ll post back with the result later.
execute
queued execute task #1: default-0 on default-0
execute task #1: default-0 on default-0
how: tmt
exit-first: false
test: imported 0_common test set
cmd: ./test.sh
00:00:00 errr /tests/legacy/0_common (on default-0) [1/137]
test: imported p_abrt test set
cmd: ./test.sh
00:00:00 pass /tests/legacy/p_abrt (on default-0) [2/137]
test: imported p_abrt-cli test set
cmd: ./test.sh
00:00:04 pass /tests/legacy/p_abrt-cli (on default-0) [3/137]
test: imported p_acl test set
cmd: ./test.sh
00:00:05 pass /tests/legacy/p_acl (on default-0) [4/137]
test: imported p_amanda test set
cmd: ./test.sh
00:00:05 pass /tests/legacy/p_amanda (on default-0) [5/137]
test: imported p_anaconda test set
cmd: ./test.sh
00:00:03 pass /tests/legacy/p_anaconda (on default-0) [6/137]
test: imported p_annobin test set
cmd: ./test.sh
00:00:00 pass /tests/legacy/p_annobin (on default-0) [7/137]
test: imported p_arpwatch test set
cmd: ./test.sh
00:00:03 pass /tests/legacy/p_arpwatch (on default-0) [8/137]
tmt -vv -c distro=centos-7 run -a provision --how=local
The result:
total: 134 tests passed, 1 test failed and 2 errors
failed:
[+] Tue Nov 21 15:48:25 UTC 2023 -> Running ./1-test-XmlTextReader.sh - test XmlTextReader of libxml2-python
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'libxml2'
errors:
p_cron test:
Redirecting to /bin/systemctl start crond.service
[+] Tue Nov 21 15:38:09 UTC 2023 -> Running ./cron_crontab_daily_test.sh - crontab will run daily jobs.
Maximum test time '5m' exceeded.
Adjust the test 'duration' attribute if necessary
0_common tests:
python: can't open file '/var/tmp/tmt/run-003/plans/legacy/discover/default-0/tests/tests/legacy/0_common/tests/0_common/000_centos_default_repos.py': [Errno 2] No such file or directory
It may need some tweaks to get it to work. I can dig into more. I have also updated the ticket.
)