Badges for bodhi: new badges for test cases and BZ#* - and change the source for kernel regression badges

Project Discussion
,
1

This topic is a bit motivated by Badges for Ask Fedora (and Fedora Discussion overall)

I observed in bodhi that many users contribute by testing if packages work in general. For adding “karma”, they receive badges to reward their contribution.

However, I observed over time that it is much less common that users also do tests that go beyond karma. One example for other types of bodhi tests is Firefox, where users have the opportunity to additionally test some “test cases” with separated thumbs up/down for each test case (although Firefox and the kernel are the two packages I know off my mind where contribution to test cases/regression tests occurs comparatively often).

Further, we have regularly updates where users have the opportunity to test/verify if bugs or CVE-security-issues have been solved: these have additional BZ#* (with * being a number), also with each BZ#* having a separated thumbs up/down.

So both in the BZ#* cases and in the test cases users can give additional thumbs up or down.

Generally, we have to be clear that these are often much more time intensive activities than general testing for karma. But what do you think of adding some incentives for users to also contribute more often to these issues by creating badges for BZ# and also badges for “test cases”? (May it be a common badge, or separated badges)

Independent of the above points: maybe it can create some simplification/harmonization by also allocating kernel regression badges by the thumbs up/down in bodhi instead of using the kernel regression scripts for that. This would reduce the users’ efforts to get their badges because it would be one login less for them (and one step less in setting up the regression test environment). From a security point of view, it would also reduce the attack vectors :wink: I guess the contained little simplification of setting up the regression test environment is not noteworthy, but users would be no longer confronted by what and how to write stuff in the regression script’s config (except nvidia-users).

Given the fact that we already allocate badges by thumbs up/down of karma, I guess we have everything in place to easily use the regressions’ thumbs up/down as the source for kernel regression test badge allocation and to add additional badges for the other thumbgs up/down activities.

What do you think?

1 Like
2

I have neutral feelings about this idea. I grouped my feedback into two parts:

  1. Is it technically feasible?
  2. Do we risk gaming of the system?

Is it technically feasible?

The pre-requisite for any automated badge to be awarded is the Fedora Messaging Bus message. So, I looked at this Cockpit update on Bodhi…

https://bodhi.fedoraproject.org/updates/FEDORA-2023-2e47853ed4

The corresponding message can be found here in Datagrepper:

https://apps.fedoraproject.org/datagrepper/v2/id?id=0f168ff0-4333-4d3b-91ad-599acd68e56e&is_raw=true&size=extra-large

The useful bit of the query for the test case looks like it is here:

      "testcase_feedback": [
        {
          "comment_id": 2939409,
          "karma": 0,
          "testcase": {
            "id": 122,
            "name": "QA:Testcase realmd join cockpit"
          },
          "testcase_id": 122
        }

So, it looks like we can get some data about test cases and and regression tests from the Bodhi messages in the messaging bus. I think we could determine whether a user gives karma on a specific test case, but knowing whether it is a specific test case might be harder. What could be easier is determining whether it is a test case or regression test of a specific package though, e.g. the kernel or Firefox.

Do we risk gaming of the system?

I guess this can be said of many things that we already have badges for. But I wonder if we risk people randomly choosing karma on a test case just to earn a badge. Do we have any way of truly validating whether someone has completed the test case other than whether they click a radio button?

So, I like the idea of giving a badge for this kind of activity, but I feel like we need a stronger way to validate that the person has actually done it. Then I don’t mind giving a badge for it, and then I think the badge would actually be more meaningful as an incentive for getting people to run test cases.

I know that the QA team uses some kind of application for test cases. I can’t remember the name of it or how exactly it works. Does it even emit Fedora Messaging messages? I’m not sure. Maybe @adamwill or @sumantrom have thoughts here.

In the event that there is a way of capturing this kind of activity in a more meaningful way than a Bodhi message, we should probably capture this feedback in some of the recently-created GitLab epics for the new Badges system, most likely this one in specific:

More context on the GitLab epics:

3

I agree on that, it applies to all posts on bodhi at all, including the karma. But it is less about validating that the person has done the activity but what was done and how (which obviously includes the first): everyone can do it, but maybe it makes sense to add some input that can be validated by third parties, indicating what the person has done and how. Even if a person did something, it does not necessarily mean it was done in a way that fulfills the goal.

The kernel regression test might fulfill this role for the regression tests, but I see these scripts a bit critical from a security perspective when used with account credentials (also, do they work with 2FA?). It might be nice to integrate this in bodhi somehow? This would also allow others/everyone to be able to verify or compare results as long, as it is anonymous. However, integrating the kernel regression test results might be hard to implement. Maybe more abstract data can be used in Bodhi instead of the whole logs (at least for successful tests). And maybe something comparable for other tests?

4

I can be a dreamer, but I always hope in good wills of people. And if the real intent is randomly choosing karma on a test case just to earn a badge… ¯\_(ツ)_/¯

For sure @sumantrom is more well versed.
There is the kernel regression tool. (It used to grant a badge, but right now it doesn’t).
There is relval, to compile the wiki for validation testing. And there is fedora-easy-karma (and fedora-update-feedback) to interact with bodhi from the command line. But they are indeed command line tools that avoid the need to use a browser, they don’t force you to run test cases adn I don’t think they emit any message (messages are still emitted by bodhi).