My first question is ‘what is a contribution’? I was going to start off with ‘what is a contributor’ but then I realized that contributions had not been defined in any way which was measurable. My second question goes from ‘what is a contributor’ and how do we properly measure that they exist?
These are important to figure out because over time there will be questions of ‘are we really seeing the outcomes we wanted’
This is an excellent question, and I agree we need to address it early. I don’t think we’ve ever made an official, formal definition. I’m not sure we should have an overall strict definition, but clearly we need to have a common understanding of the working definition we’re using in the plan.
So, here’s a starting point — with the second question first.
A Fedora Project contributor is anyone who:
Undertakes activities
which sustain or advance the project towards our mission and vision
intentionally as part of the Project,
and as part of our community in line with our shared values.
(And “a contribution”, then, is any product of such activities.)
Explanation:
It has to be, you know, actually doing something. And, I like this to be plural — one drive-through might be a contribution, but to be (present tense) a contributor, it should be an ongoing thing.
And not doing just anything, but something that goes in the direction our project is all about.
An activity which is incidentally or accidentally a benefit to Fedora doesn’t count.
We’re not just a free-and-open-source-software-licensed project, but an intentional community with shared foundations.
I recognize that the 4th point is a little redundant with the 3rd — but think of it as emphasis.
What do you think? Is this missing anything? Does is it exclude something it shouldn’t? Does it get near the mark?
Then, to the pragmatic point: this is hard to measure. But, I think we can introduce ways of measuring that will be meaningful enough to track relative growth even if we can’t get to an exact number. And, we can refine some of these over time — we can start with something more rough like the fedmsg-based statistics I’ve presented before (see my 2016 DevConf.cz metrics presentation), and then develop more sophisticated approaches (I hope @cdolfi can help!) and apply those retroactively as we go.
For someone to not count as a “drive through” contributor, what kind of term / time period would you expect them to make a contribution by? Is one contribution a year enough? One every 6 months? Etc.
Also I wanna just point out that often our metrics for counting a contribution don’t consider design tasks!
I don’t like the idea of a formal definition for the contribution, as it becomes a limiting factor.
When we talk about metrics we should definitely discuss what we are measuring with them.
But I think we should be clear that there is no one good single metric to measure it all. We are not robots
I would say the goal of the metrics effort is not to cover all contributions, but to have some types of contributions measured and compared within their definition. While we should leave the room for contributions, which we do not measure yet.
We can have contribution by a fedora-devel Mailing list, contribution by Ask site, contribution by package update, contribution by Project Discussion on Discourse, contribution by comments on Pagure tickets…
They are all different aspects of contribution. They don’t provide complete coverage, but they provide some insights.
And then we can consider trends within each category, and may some conclusions for the state of a project based on that.
That’s why I think what Matthew wrote is a good definition. How we measure it is, to some degree, an implementation detail. We’ll have to acknowledge that our metrics will be an imperfect measure of what contributions (and thus contributors) exist. But let’s not confuse contributions with measured contributions, because that’s an important distinction.
I think I agree with Matthew’s take, but I am afraid that trying to formalize it this way can lead to people who contribute, but have impostor syndrome, always questions themselves on whether their “activity” is activity enough to qualify.
So my question is - do we really need this generic definition for anything, or is it just the natural desire to define everything with no practical purpose. Because while I can totally understand the latter, I think that it will be better to not start this conversation at all.
Can we just leave it undefined, focus on measurable contributions, and tell people to come up with additional ideas for additional metrics the same way how we accept proposals for new badges?
I am not looking for a definition to cover all contributions. I don’t think that is realistic either. However, if a goal is ‘double’ something, then we need to know where and what that thing is we doubled.
It may be that this goal is the problem in the first place because measuring it is not ‘possible’. That is good to know and time to come up with a better goal which we can ‘measure’.
I completely understand the concerns about making a “formal” definition for contributor or contribution. With that, if you want to be able to measure (even as a proxy) the goal of double the number contributors who are active every week there needs to be atleast a “metrics” definition. A good place I could see starting on this discussion is listing everything the community can think of that is considered contribution. Then from there determining every location (what specific repos, discussion forms, mailing list) these contributions could be. From there, I feel like solid definitions could be formed, and this is where I see the “metrics” definition coming in. We can determine what in that set is measurable, how we would would want to do that, and consider the impacts of this proxy as it will not be able to count everything.
Outreachy has a very specific definition of what consists of a contribution… you basically complete a task, usually a ticket, defined by the project you’re working with.
I think the GNOME Foundation to consider contributors active checks every 2 years to see if you’re active and if not you get put in the alumni group and lose voting rights? I am fuzzy on this but they have A Process
My gut answer is: I think one a year could be enough, if it is year after year. It’s hard to draw a line. My intention for the “Double the Number!” north star is that we’ll measure the number of people per week that do whatever it is we count, and use that number (Ben pointed out to me that “Weekly Active Contributors” is similar to the “Monthly Active Users” metric common for apps and websites[1]). For that, the contribution pattern of a given person doesn’t really matter, just the aggregate.
As I did with the fedmsg metrics, we can do some analysis to find interesting groupings and patterns and maybe learn more stuff — and the patterns that emerge there may show some natural lines.
Yeah. My best thought for including these is by including ticket activity from particular team trackers we’ve identified. I think we could probably reasonably-enough count even comments in design tickets as “contribution” for the metric purpose. But I can see trackers which need further distinction (where there might be, say, support tickets). I’m open to ideas for other things! For example, we could count comments in design by people in the design team group[2] as contributions.
And maybe generally: comments in Project Discussion in the tags that match groups they’re in. That might exclude some valid contributions, but seems like it might be more meaningful than just counting all messages here.
Interesting — thanks. I think the Outreachy one might be stricter than I’d like. And on the latter… we kind of have something similar with Ambassadors, and with packagers too. But not generally.
Actually, I’m open to changing our counting to month-based rather than week-based, if someone has a strong argument! ↩︎
we are really close to having FAS groups synced here! ↩︎
On the format, I dont think that matters too much. It really just matters where you think that document will get visibility, it does not need to be more complicated (at least at first) than 2 lists. I usually find that if you get something started and put it out there and say “this is the list” people will quickly start to tell you what your missing
Whilst contributions can be of various kinds, it is important to understand that there would be certain kinds of contributions that are tangible and can be quantified (say commits to a repository, number of responses in a mailing thread or discussions thread, list of packages maintained etc.) and here would be those that are not as tangible and quantifiable (say helping with fetching the swags from one location for a Fedora Project booth on an event). Does that mean that we do not value and/or appreciate the latter kind of contribution? Absolutely not. In fact, they are just as, if not more, important as the former kinds.
What we risk with being very specific with our definitions of “contributions” and “contributors” is that the folks around us would, in one way or another, be more inclined to make contributions that are tangible and quantifiable to help affect the metrics. It would certainly give us an illusion that the metrics of community health (and the number of active contributors at any given point in time) are improving but at the expense of severely ignoring other kinds of contributions that do not add to the metrics. How do we ensure that we are specific with our definition all while, being encouraging to all kinds of contributions?
The swag transfer example might be a good one to explore. What kind of digital trail might such activity leave? A ticket somewhere? A note in an event report? An inventory database change?
Kevin mentioned that in chat yesterday, but I don’t think it works well for this purpose. I’m in the top 25 all-time and I’ve averaged roughly one badge every 4.5 weeks in my time with Fedora. I suspect most people have a similar or lower average (excluding the very-newly joined who have a lot of low-hanging fruit to collect). Over a longer time scale (say 6–12 months), this is a good proxy. At the very least, it’s pretty trivial to earn two badges a year by casting a ballot in the elections. It can be used to measure how many active contributors we have year-over-year, but not more granularly.
So this raises a point that’s implicit in the “doubling the weekly active contributors” (or however we end up phrasing it). As I understand it, we want to increase our “sticky” contributors — the people who we see around regularly. That’s what gives us a growing, robust community. That’s not to say we don’t value the people who make one or two contributions a year. We very much do. But that’s a different goal.
There might be some areas where contributor activity might not leave much of a digital trail but where there’s a manually-assigned badge. Not sure. I think in general it’s better if we can link badges to some kind of automatically-measured activity, which brings us back to the same problem.
I think part of going through this exercise of ‘what is a contribution’ and ‘how do we count a contribution’ is important fundamental test of the hypothesis: ‘Double the number of weekly contributors’.
It may be that the intended goal is not something which can fit directly into the ‘Theory of Change’ using the available tools. It doesn’t mean that the goal is a bad one OR impossible to actually meet. It just means that the chosen framework (Theory of Change) is not able to be used with the chosen question. It may mean that the question needs to be changed to something which can be used within the framework. You may still reach the original goal but you do so in a way which can be better seen.
