I think we have the tools to count contributions (using the definition I gave, which so far wins by default) that represent a broad enough sample that we can use it as at least a proxy metric. That is, we can have reasonably-good confidence that doubling that number means we’ve won.
In fact, I’d go so far as to say we probably can get that from the message bus now. We have:
git commits that represent package changes
git commits that represent docs
git commits that represent infrastructure / releng / tooling
pagure and gitlab tickets representing team work (like Design)
wiki edits that represent testing (would be nice to better capture test days, though)
wiki edits to change proposals
bodhi comments (also testing)
badges (as noted above, not a great metric alone, but should catch some things that might otherwise be missed)
posts here (which could be further analyzed using Data Explorer, so we get different team activities + support provided in Ask Fedora)
commblog posts
Plus maybe meeting logs — there are message bus notifications but I don’t think that includes list of users. Meetbot, however, lists participants, and assuming people have their IRC nicks in FAS, we can map those too.
To get the key number, extract the FAS username from each of the above for a given week, and count distinct occurrences.
I am still thinking about how to do the wide brainstorming (kinda thinking LimeSurvey?) Cali suggests, so without going to very specific things, are there broad areas that the above list misses?
Events. Specifically giving talks at conferences and staffing booths. Those are both important outreach activities that may only be partially captured. For example, not everyone who is staffing a booth comments on the ticket requesting funding. Some events have badges, but I’d argue those don’t represent a contribution in the sense we mean.
It may be that the imperfect metrics I mention are good enough, but in either case it’s not reflected in your list.
meetbot does emit user who started/ended meetings, but yeah, would need to scrape the list of ‘people who said something’ from it.
bugzilla? (tricky though to know whats a ‘contribution’ there… is it filing a bug? filing a bug that has a lot of comments? filing a bug thats resolved? resolving a bug?)
Some of these do generate tickets (through funding requests) and wiki edits (this used to be a more consistent practice). But, yeah, this is something we might need to be conscious about doing differently in order to measure better. (A separate “booth staff” or “speaker” badge for each event? making a SOP of recording names somewhere?)
On the one hand, I so badly want to not count filing bugs as contributions on principle. On the other hand, that might defeat my own definition. (It’s tricky, though, because of my Contributor Criterion #3 — many bug filers do not consider themselves part of the project, just communicating to the project.) I think, though,that most resolution of bugs will result in some other activity that does get measured (like a git commit). (If we had an active Bugzappers group that did triage and stuff, that’d be more difficult.)
So I’ve gone on the record in conference talks as saying bug reports are contributions. I stand by that, but I also don’t think all bug reporters are contributors. I think part 3 of your definition represents this distinction well. I don’t want to get us too off into this tangent, so I’ll add it to my Blog Backlog of Shame™.
Agreed. And this brings up another point that I’ve talked about in my “Your Bug Tracker and You” talk. The person or people who fixed the bug aren’t necessarily the assignee of the bug. So, for the most part, I don’t think there’s anything in the bug tracker that we care about for this purpose that we can’t already get somewhere else.
Many times the helpers on Ask are the real contributors. They do recognize errors and do report them on the right place or ask the person who related the error to do so. But unfortunately they not really get the credit for it because the user who related was just an normal linux user who not intended to contribute just wanted to make something work.
Absolutely. But actually, I think the “rule 3” distinction[1] is useful to preserve here — the change we want is that these helpers feel like contributors (and from there, an easier step to broader and deeper involvement).
Without going into examples of contributions, I find Fedora Project Contributor Agreement a starting point of our discussion.
I overlooked what the FPCA states when I signed up for the Fedora Accounts, but it entails overarching definitions of contributions. I recite here.
“Contribution” means a Work that You created, excluding any portion
that was created by someone else. (For example, if You Submit a
package to Fedora, the spec file You write may be a Contribution, but
all the upstream code in the associated SRPM that You did not write is
not a Contribution for purposes of this FPCA.) A Contribution
consists either of Code or of Content.
“Content” means any copyrightable material that is not Code, such as,
without limitation, (i) non-functional data sets, (ii) documentation,
(iii) wiki edits, (iv) music files, (v) graphic image files, (vi) help
files, and (vii) other kinds of copyrightable material that the Fedora
Council has classified as “content” rather than “code”.
This is tough, because filling or contributing to a bug is a great point of entry into the project, and it is an important part of the development effort.
Sure, but I would highlight Matthew’s “rule 3” distinction[1]
At times I report bugs I find in software I use. And… problem solved? Goodbye. Yes, I somewhat contributed to the software. But honestly, I don’t feel part in such software community and I wouldn’t describe myself as a contributor of such software even if I contributed
The last time I came into this thread, I was feeling jaded. So I am trying again with a fresh perspective.
TL;DNR: Use the number of unique users active on the Fedora Messaging Bus each week as our baseline metric.
Why getting a number down right now matters
The above resonated with me a lot! @smooge and @cdolfi are encouraging us to think in a way where we will know whether we achieved our end goal by 2028 (and perhaps earlier than 2028 too). There is a risk that if we go too far down this 5-year path without knowing how to measure what we are trying to double, we could burn out people along the way.
Burning out people could happen because we need to make sure we are building validation and recognition into this 5-year plan as we go. We might get to 2028 and still be asking ourselves whether we accomplished our north/guiding star. Maybe we get to 2028 and the results are so overwhelmingly obvious that we don’t have to spend a significant amount of time trying to measure whether our theory of change worked or not.
But building in easier ways to measure our success as we go is helpful so we can validate the things that are working as we go, and stop doing the things that may not be contributing to us achieving our goal. I want people to feel recognized and enabled for doing good work. We need to be good cheerleaders in our efforts as we go. This way, contributors to our strategy can feel recognized that they made a valid contribution(s), and we don’t risk people feeling invisible and unrecognized for contributions that might actually be significant in achieving our goal.
So, to summarize, it is important that we come up with some method for measuring contributions right now. It will be an imperfect measure. But we could also think of it this way: the way we measure contributions today does not have to mean we measure contributions the same way in 2028[1]. We may find new ways of measuring “invisible” contributions, and in a way, this might also mean we “double” contributions by 2028.
With the above in mind, why don’t we count the number of unique users active on the Fedora Messaging Bus each week?
This is data that we have right now. We could pull this data tomorrow. As everyone has enumerated in great detail above, there are lots of caveats and we miss a lot of things by only looking at the Messaging Bus. But this gives us somewhere to start.
Along the way, we can continue this conversation about how to improve our counting. And maybe even think of better ways of capturing contributions on the messaging bus.
Thoughts?
Actually I would be very sad if the way we measure contributions today did not change at all by 2028. ↩︎
We’d want to make sure to count only active usernames — that is, those where the message is triggered by that person’s activity. (As opposed to that person’s username being involved — for example, I add or remove you from a group, that doesn’t indicate that you are actually active.)
We also might want to, early on, filter out some specific things which we are excluding from the working definition:
Ask Fedora initial posts (and self-replies to those questions), based on comments above
Repositories which happen to be hosted on pagure.io but aren’t Fedora-related (maybe should be an allowlist of known team trackers and repos?)
I’m not sure about Copr. There’s some gray area there, and moving people from being “I build stuff in copr” to greater direct involvement seems meaningful.
I also think we should add meeting logs — even if we have to map IRC nicks to FAS accounts. (Zodbot already does send message bus messages, but doesn’t do the username mapping.)
I agree in theory, but this may be more trouble than it’s worth. Manually maintaining lists (either allow or deny) means we’re going to drift from reality very quickly. Are there that many repos on Pagure.io that aren’t Fedora-related in some way? We’re going to have noise no matter what we do, and I think this would be an acceptable level.
Two months later, I’ve written a longer version of my thoughts on the matter. The short version is that bug filing is neither necessary nor sufficient for being a contributor. Matthew’s four-point definition is super valuable in this context.
There is an ongoing conversation with me, @mattdm, @cdolfi, CPE folks, and some others in Red Hat OSPO about this definition of a contributor. I think this definition is acceptable. I like it.
To keep pushing the needle forward, I suggest we review CHAOSS metrics and look at mapping a metrics model that makes the most sense for the Fedora 5-year strategy:
If we can frame our measurements around CHAOSS metrics, we have a wider pool of open source tools and communities that we could leverage to make the challenge before us a little easier.
