@jflory7 following on from a chat in Matrix, I would like to share my ideas on tracking contributor journey.
A few weeks ago, with a help of marketing team and my own efforts to reach new Docs contributors, invitation to writing workshop was promoted to Fosstodon (Fedora Official account), Floss.social (my personal account, and reddit technical writing groups.
I wonder if there are ways to track contributors by traffic source using the link pointing to Fedocal (which I share agenda and details of workshop) or discussion.fedoraproject.org (also repost the details here).
If our attempt to reach new contributors is measurable, we could justify more efforts to use more social channels with well crafted marketing campaign.
An example of contributor journey is like this.
Viewed campaign/invitation
Clicked the link pointing to Fedora sites (Fedocal or discussion)
Visited Fedora account creation (if new comers)
Wrote a post in Discussion
Logged into Pagure or GitLab
Created issue ticket or made pull request
Is it feasible in terms of data privacy and practicalities? Or âitâs a long shot, but well worth tryingâ?
There are a couple of Badges that should be given by doing some of those actions, but most of them seem to be broken right now. If the Badges team manages to fix them they could be used to gather most of this data, I believe, as weâd only need to count how many Badges were given in a release cycle.
I love this idea, by the way, we really should have some more consistent metrics to work with.
I donât want to bring a negative point so early, but I would like to raise awareness on the privacy aspect of gathering metrics. For some that can seem like crossing a line where weâre not just tracking the information to provide a service, but also to track contributors. Then depending on where we land we may want to have a simple plan for managing expectations on what weâre following and why. Thatâs the unfortunate allure of metrics. Itâs great to have the data, but it comes at the cost of tracking.
Personally, the first pass at the idea gives me heebie jeebies, but Iâll step aside to see what you come up with.
Thanks @hankuoffroad for kickstarting this conversation. I am also taking a think at this, because in general, I do think we want to know more about our contributors in a measured, consistent way. We need to know the health of the community and understand how we can keep Fedora both a fun and productive place to contribute in the Open Source space.
For finding our footing, I am not sure the source of where people come from is that useful. Sure, it could tell us where people are finding our content, but the important part to me is what comes after people find our content. Are we onboarding people successfully as new contributors? After showing up in the community, do they continue and keep going, or do they engage with us and then drop off? The engagement in the community is more important to me right now than finding out where people are coming from.
So, instead, the question might be: how good of a job are we doing at keeping newcomers involved and active with Fedora?
I think Badges could be a part of this, and we might be able to do it without the broken badges. For example, say that we launched a campaign for driving new contributors to Fedora Docs. Every contributor (new and old) who participates with the campaign, we could give them a badge for it (e.g. attending a virtual workshop, solving a specially-marked good-first-issue, etc.). The badge would be awarded manually.
Then, we could simply collect the list of names from people who claimed the badge and consult our existing data to ask: what happened to these people after a month? Do they show up anywhere in the Fedora messaging bus? Did they continue contributing? Or did they drop off? If they started something and then dropped off, where are we seeing the most friction? The badge data would give us important clues on where to improve the onboarding process.
I think this is key and that it is also a conversation that we can manage. We have to be transparent, honest, and clear about why we would use this data. In many cases, we are already collecting it, but we donât do much with it right now. My intent of using and studying this data is so that we contribute to Fedoraâs vision and mission statements:
Vision: The Fedora Project envisions a world where everyone benefits from free and open source software built by inclusive, welcoming, and open-minded communities.
Mission: Fedora creates an innovative platform for hardware, clouds, and containers that enables software developers and community members to build tailored solutions for their users.
We are not trying to sell ads. We are not trying to sell anything, actually. We are trying to understand our vast and sprawling community better, in order to make our community more inclusive, welcoming, and open-minded.
Beyond just the Fedora community / perhaps as a side benefit related to the âFirstâ goal of FedoraâŚestablishing a really good way of measuring, analyzing and presenting the health of the projectâs contribution model could be shared out to anyone else who wants to use it.
I would suspect that many upstream projects, but also even other distributions, would benefit from having a clear best practice laid out for understanding where their own projectsâ contributor strengths and gaps are and sharing that back out to the community to help coordinate efforts.
(The things that keep popping in my head are the xkcd of the guy in Nebraska keeping the entire internet afloat, and the stat that something like 25% of all of Arch Linux is maintained by one person)
Iâd say that this primarily lands in the side of feeling creepy depending on how itâs done. For example, if I come to a forum I understand that anything I post will be public. I also understand that private information I give to the forum is technically available to whoever manages the forum. By contrast, if I click a link to go somewhere in Fedora and by that link Fedora now knows where I came from, that kind of information is not something I expect for Fedora or anyone to be tracking.
That concern on its own is creepy, but thatâs not the end of it. The next concern after the tracking is how that information is managed, what information is collected, and who has access to that information. For Fedora I trust that we would be responsible with how we do this, but for some people outside of the project they may not trust Fedora or want Fedora to have this information even if they trusted us. The telemetry proposal from this summer is a great example of a bunch of people who presumably use Fedora who do not want Fedora tracking anyone without clear consent.
And thatâs the bigger picture that weâre playing in. For as much as I trust that Fedora will be responsible, we would have to manage the optics of how things look to outside observers who would see Fedora taking the same steps that look to be heading in a privacy invasive direction. Almost every company with a web presence has some form of digital tracking and says theyâre handling it responsibly. Yet weâre plagued with weekly news about how that information, which did not need to be collected, was managed irresponsibly and led to something like a data breach. Those brands are the ones we would be getting compared to and every day it would be a matter of trust on the part of the community that weâre not doing the same. It can be argued that someone who uses Fedora already trusts Fedora in important ways, but the counter argument is that you donât want to break that trust.
I know that privacy is not a core value of Fedora technically, but you do see the expectation of privacy permeate the entire project. Itâs the reason why the Linux community caught fire at the idea of telemetry in Workstation. Itâs the reason why we donât have trackers on any Fedora websites currently (according to my uBlock Origin extension at least). Itâs the reason why many in the last two years have come to Fedora - to use one of the most secure Linux distros that also comes with the privacy that is expected of an open source project. Like it or not, many consider open source to be synonymous with privacy.
If we already go this far to maintain user privacy, my personal opinion would be to not edge away from that unless there was a clear and digestible method, reason, and opt-in/out that aligns with the expectations people have of being private. It would be great to track every move that users or contributors make as they move through the conversion pipeline from user to contributor. However, that is the value that has lead businesses and marketing departments to justify being extremely privacy invasive. If youâre tracking techniques get more effective without the consent of the people youâre tracking, thatâs just inherently invasive.
Solution Zone
âJoseph, do you not have any solutions?â Iâd have to think about it more, but here are the two main principles I would keep in mind if I had to get more data.
If I need more information on something:
Ask for it
Grab it from places where the information was given in a public context
Doâs:
Do ask people to take a survey at the end of a workshop
Do watch what folks have to say in comments
Do track public metrics like boosts and favorites that people gave knowing it would be public
Donâts
Donât learn information about users without them knowing that youâre collecting that information
Donât collect personal information and tie it to identifiable information in ways users donât expect
Donât use redirect links that give unclear information about the person visiting a site
Avoid depending on systems that are effective due to invasive tracking
Avoid looking like you are doing invasive tracking
TL;DR - Spying on someone will always get more information on them than what we would get by just asking. That doesnât mean we have a right to that information just because it would be more effective for us. Resigning ourselves to asking for information is the tradeoff we have to make if we want to respect digital privacy.
I could go on because thereâs tons of nuance to unpack, but Iâll cut myself off there.
I suspect one of the ways to responsibly mitigate this ethical dilemma is to better use the data that we already have available from public sources instead of trying to collect more data. We can learn a lot just by watching GitLab, Pagure, Matrix, and Fedora Discussion. These are all public platforms with public activity that is reflected on the Fedora messaging bus.
Register Fedora account: this happens after understanding why I need it and agree to Fedora Project Contributor Agreement.
After becoming a regular participant for a group (or first contribution), interactions, engagement and relationship are more meaningful than git activities or badges.
Q: Are we reaching out to right audiences?
A: Check views and interactions in each social platform.
Q: Are messages clear enough like âDownload nowâ?
A: Examine call to action message in âContributionâ menu in the home page.
Q: Is onboarding effective in my team? When does onboarding start?
A: Onboarding can start before a person joins weekly meeting for the first time or stay for a few weeks in a group asking for onboading process.
Q: I donât want to commit much. Can I contribute like âgigsâ for s specific project? How does Fedora acknowledge casual contributions?
A: Donât know. Not sure contributor survey tells much for us to take action (timely and corrective).
Does the new launch of badges have something for docs contributions? badges
Leaderboard is a popular mechanism in any community projects.
Top PRs created for a month: the number of PRs and lines per PR
Top review comments for a month
Although each Git forge has leaderboard stats, badges will bring more motivation for sustained contributions.
At the moment, even if a docs contributor changes metadata on 200 files, or rewrites contributor guide (over 2000 lines and xx hours of edits), there is no recognition or badges.
I am thinking about launching a new survey that we could use for this effort as well as any other team. This survey would be a âcontributor journey experienceâ survey. The survey would be filled by someone right when they start contributing to Fedora, while they are contributing, and at their option, when they stop contributing to Fedora.
This starts to build up a library of information that we can act on. When we run one of these contributor events or virtual workshops, we could ask people to complete the survey as part of the session. The idea is that this helps us determine a baseline when we do these outreach events. Are we effective at them? Where are our strengths and weaknesses? Where are gaps and how could they be filled? We could design this survey for the Fedora Docs use case specifically first, but I think the survey should be designed so that any team could use it too.
While the Badges 2.0 system is under development, we could also take a different approach. We can get this data in various forms with tools like Cauldron or platform-specific metrics. While we could still use a badge, we could also try recognizing contributors in other ways. The Fedora Quality team used to have a blog series called Heroes of Fedora (shout-out to @coremodule!) This was a report shared every release by the Quality team recognizing the top contributors from the release cycle.
We could do something like this with Fedora Docs. I could help out with even mailing a swag thank-you package to the person too.
I wholeheartedly agree with a survey tailored to contributor journey experience. If the survey is sent at the right time (moment of truth), it will be effective for us to follow up with a participant of the survey.
Moment of truth is something like;
within x weeks after a new joiner made an introduction at Fedora Join/Welcome to Fedora
within x weeks after first PR was made
Sure, we need to start with other ways to recognize people who made huge efforts to maintain/improve Fedora. Some working groups have good track record of rewarding contributors. We could do something like QA team on every release cycle. We just need to decide on criteria to acknowledge top contributors.
Are you and the Docs Team interested in trying this out? I think a good first step is figuring out the questions we want to answer. In the interest of getting more feedback collected in frequent intervals, the survey should be very brief. Maybe no more than five questions.
Things Iâd like to know:
Did you interact with the Fedora Join SIG when starting to contribute to Fedora?
Have you faced an issue that is blocking you from contributing further?
What team(s) have you participated in so far?
Do you intend to continue contributing to Fedora?
What is something that would help you contribute further to Fedora?
This is just to get ideas going.
Starting with one specific area might be good. For example, âQuick Docs Heroes of F39â could be a good start. It limits the scope to one repo instead of the entire universe of Fedora Docs repos and projects. This lets us try out a new idea gradually and see how it goes.
Many thanks for the suggestion. Iâve been pondering about this. Iâd say it would be worth sending a mini survey after the writing workshop. I guess I need to open a ticket to use Limesurvey.
Post-event survey for the workshop can be;
Did the Fedora Docs workshop meet your expectations?
What topics would you want to see more of at next Docs workshops?
Please share any additional comments, thoughts, suggestions for next workshops.
This is a project-wide questionnaire that requires consensus with other working groups and Fedora Join SIG with @ankursinha . Iâm happy to engage with other working groups to agree on survey questions and practicalities.
p.s.) I noticed we have a project-wide initiative already. Iâll watch a development from the thread.