@jflory7 following on from a chat in Matrix, I would like to share my ideas on tracking contributor journey.
A few weeks ago, with a help of marketing team and my own efforts to reach new Docs contributors, invitation to writing workshop was promoted to Fosstodon (Fedora Official account), Floss.social (my personal account, and reddit technical writing groups.
I wonder if there are ways to track contributors by traffic source using the link pointing to Fedocal (which I share agenda and details of workshop) or discussion.fedoraproject.org (also repost the details here).
If our attempt to reach new contributors is measurable, we could justify more efforts to use more social channels with well crafted marketing campaign.
An example of contributor journey is like this.
Clicked the link pointing to Fedora sites (Fedocal or discussion)
Visited Fedora account creation (if new comers)
Wrote a post in Discussion
Logged into Pagure or GitLab
Created issue ticket or made pull request
Is it feasible in terms of data privacy and practicalities? Or “it’s a long shot, but well worth trying”?
There are a couple of Badges that should be given by doing some of those actions, but most of them seem to be broken right now. If the Badges team manages to fix them they could be used to gather most of this data, I believe, as we’d only need to count how many Badges were given in a release cycle.
I love this idea, by the way, we really should have some more consistent metrics to work with.
I don’t want to bring a negative point so early, but I would like to raise awareness on the privacy aspect of gathering metrics. For some that can seem like crossing a line where we’re not just tracking the information to provide a service, but also to track contributors. Then depending on where we land we may want to have a simple plan for managing expectations on what we’re following and why. That’s the unfortunate allure of metrics. It’s great to have the data, but it comes at the cost of tracking.
Personally, the first pass at the idea gives me heebie jeebies, but I’ll step aside to see what you come up with.
Thanks @hankuoffroad for kickstarting this conversation. I am also taking a think at this, because in general, I do think we want to know more about our contributors in a measured, consistent way. We need to know the health of the community and understand how we can keep Fedora both a fun and productive place to contribute in the Open Source space.
For finding our footing, I am not sure the source of where people come from is that useful. Sure, it could tell us where people are finding our content, but the important part to me is what comes after people find our content. Are we onboarding people successfully as new contributors? After showing up in the community, do they continue and keep going, or do they engage with us and then drop off? The engagement in the community is more important to me right now than finding out where people are coming from.
So, instead, the question might be: how good of a job are we doing at keeping newcomers involved and active with Fedora?
I think Badges could be a part of this, and we might be able to do it without the broken badges. For example, say that we launched a campaign for driving new contributors to Fedora Docs. Every contributor (new and old) who participates with the campaign, we could give them a badge for it (e.g. attending a virtual workshop, solving a specially-marked good-first-issue, etc.). The badge would be awarded manually.
Then, we could simply collect the list of names from people who claimed the badge and consult our existing data to ask: what happened to these people after a month? Do they show up anywhere in the Fedora messaging bus? Did they continue contributing? Or did they drop off? If they started something and then dropped off, where are we seeing the most friction? The badge data would give us important clues on where to improve the onboarding process.
I think this is key and that it is also a conversation that we can manage. We have to be transparent, honest, and clear about why we would use this data. In many cases, we are already collecting it, but we don’t do much with it right now. My intent of using and studying this data is so that we contribute to Fedora’s vision and mission statements:
Vision: The Fedora Project envisions a world where everyone benefits from free and open source software built by inclusive, welcoming, and open-minded communities.
Mission: Fedora creates an innovative platform for hardware, clouds, and containers that enables software developers and community members to build tailored solutions for their users.
We are not trying to sell ads. We are not trying to sell anything, actually. We are trying to understand our vast and sprawling community better, in order to make our community more inclusive, welcoming, and open-minded.
Beyond just the Fedora community / perhaps as a side benefit related to the “First” goal of Fedora…establishing a really good way of measuring, analyzing and presenting the health of the project’s contribution model could be shared out to anyone else who wants to use it.
I would suspect that many upstream projects, but also even other distributions, would benefit from having a clear best practice laid out for understanding where their own projects’ contributor strengths and gaps are and sharing that back out to the community to help coordinate efforts.
(The things that keep popping in my head are the xkcd of the guy in Nebraska keeping the entire internet afloat, and the stat that something like 25% of all of Arch Linux is maintained by one person)
I’d say that this primarily lands in the side of feeling creepy depending on how it’s done. For example, if I come to a forum I understand that anything I post will be public. I also understand that private information I give to the forum is technically available to whoever manages the forum. By contrast, if I click a link to go somewhere in Fedora and by that link Fedora now knows where I came from, that kind of information is not something I expect for Fedora or anyone to be tracking.
That concern on its own is creepy, but that’s not the end of it. The next concern after the tracking is how that information is managed, what information is collected, and who has access to that information. For Fedora I trust that we would be responsible with how we do this, but for some people outside of the project they may not trust Fedora or want Fedora to have this information even if they trusted us. The telemetry proposal from this summer is a great example of a bunch of people who presumably use Fedora who do not want Fedora tracking anyone without clear consent.
And that’s the bigger picture that we’re playing in. For as much as I trust that Fedora will be responsible, we would have to manage the optics of how things look to outside observers who would see Fedora taking the same steps that look to be heading in a privacy invasive direction. Almost every company with a web presence has some form of digital tracking and says they’re handling it responsibly. Yet we’re plagued with weekly news about how that information, which did not need to be collected, was managed irresponsibly and led to something like a data breach. Those brands are the ones we would be getting compared to and every day it would be a matter of trust on the part of the community that we’re not doing the same. It can be argued that someone who uses Fedora already trusts Fedora in important ways, but the counter argument is that you don’t want to break that trust.
I know that privacy is not a core value of Fedora technically, but you do see the expectation of privacy permeate the entire project. It’s the reason why the Linux community caught fire at the idea of telemetry in Workstation. It’s the reason why we don’t have trackers on any Fedora websites currently (according to my uBlock Origin extension at least). It’s the reason why many in the last two years have come to Fedora - to use one of the most secure Linux distros that also comes with the privacy that is expected of an open source project. Like it or not, many consider open source to be synonymous with privacy.
If we already go this far to maintain user privacy, my personal opinion would be to not edge away from that unless there was a clear and digestible method, reason, and opt-in/out that aligns with the expectations people have of being private. It would be great to track every move that users or contributors make as they move through the conversion pipeline from user to contributor. However, that is the value that has lead businesses and marketing departments to justify being extremely privacy invasive. If you’re tracking techniques get more effective without the consent of the people you’re tracking, that’s just inherently invasive.
“Joseph, do you not have any solutions?” I’d have to think about it more, but here are the two main principles I would keep in mind if I had to get more data.
If I need more information on something:
Ask for it
Grab it from places where the information was given in a public context
Do ask people to take a survey at the end of a workshop
Do watch what folks have to say in comments
Do track public metrics like boosts and favorites that people gave knowing it would be public
Don’t learn information about users without them knowing that you’re collecting that information
Don’t collect personal information and tie it to identifiable information in ways users don’t expect
Don’t use redirect links that give unclear information about the person visiting a site
Avoid depending on systems that are effective due to invasive tracking
Avoid looking like you are doing invasive tracking
TL;DR - Spying on someone will always get more information on them than what we would get by just asking. That doesn’t mean we have a right to that information just because it would be more effective for us. Resigning ourselves to asking for information is the tradeoff we have to make if we want to respect digital privacy.
I could go on because there’s tons of nuance to unpack, but I’ll cut myself off there.
I suspect one of the ways to responsibly mitigate this ethical dilemma is to better use the data that we already have available from public sources instead of trying to collect more data. We can learn a lot just by watching GitLab, Pagure, Matrix, and Fedora Discussion. These are all public platforms with public activity that is reflected on the Fedora messaging bus.
Register Fedora account: this happens after understanding why I need it and agree to Fedora Project Contributor Agreement.
After becoming a regular participant for a group (or first contribution), interactions, engagement and relationship are more meaningful than git activities or badges.
Q: Are we reaching out to right audiences?
A: Check views and interactions in each social platform.
Q: Are messages clear enough like “Download now”?
A: Examine call to action message in ‘Contribution’ menu in the home page.
Q: Is onboarding effective in my team? When does onboarding start?
A: Onboarding can start before a person joins weekly meeting for the first time or stay for a few weeks in a group asking for onboading process.
Q: I don’t want to commit much. Can I contribute like ‘gigs’ for s specific project? How does Fedora acknowledge casual contributions?
A: Don’t know. Not sure contributor survey tells much for us to take action (timely and corrective).
I am thinking about launching a new survey that we could use for this effort as well as any other team. This survey would be a “contributor journey experience” survey. The survey would be filled by someone right when they start contributing to Fedora, while they are contributing, and at their option, when they stop contributing to Fedora.
This starts to build up a library of information that we can act on. When we run one of these contributor events or virtual workshops, we could ask people to complete the survey as part of the session. The idea is that this helps us determine a baseline when we do these outreach events. Are we effective at them? Where are our strengths and weaknesses? Where are gaps and how could they be filled? We could design this survey for the Fedora Docs use case specifically first, but I think the survey should be designed so that any team could use it too.
While the Badges 2.0 system is under development, we could also take a different approach. We can get this data in various forms with tools like Cauldron or platform-specific metrics. While we could still use a badge, we could also try recognizing contributors in other ways. The Fedora Quality team used to have a blog series called Heroes of Fedora (shout-out to @coremodule!) This was a report shared every release by the Quality team recognizing the top contributors from the release cycle.
We could do something like this with Fedora Docs. I could help out with even mailing a swag thank-you package to the person too.
I wholeheartedly agree with a survey tailored to contributor journey experience. If the survey is sent at the right time (moment of truth), it will be effective for us to follow up with a participant of the survey.
Moment of truth is something like;
within x weeks after a new joiner made an introduction at Fedora Join/Welcome to Fedora
within x weeks after first PR was made
Sure, we need to start with other ways to recognize people who made huge efforts to maintain/improve Fedora. Some working groups have good track record of rewarding contributors. We could do something like QA team on every release cycle. We just need to decide on criteria to acknowledge top contributors.
Are you and the Docs Team interested in trying this out? I think a good first step is figuring out the questions we want to answer. In the interest of getting more feedback collected in frequent intervals, the survey should be very brief. Maybe no more than five questions.
Things I’d like to know:
Did you interact with the Fedora Join SIG when starting to contribute to Fedora?
Have you faced an issue that is blocking you from contributing further?
What team(s) have you participated in so far?
Do you intend to continue contributing to Fedora?
What is something that would help you contribute further to Fedora?
This is just to get ideas going.
Starting with one specific area might be good. For example, “Quick Docs Heroes of F39” could be a good start. It limits the scope to one repo instead of the entire universe of Fedora Docs repos and projects. This lets us try out a new idea gradually and see how it goes.
Many thanks for the suggestion. I’ve been pondering about this. I’d say it would be worth sending a mini survey after the writing workshop. I guess I need to open a ticket to use Limesurvey.
Post-event survey for the workshop can be;
Did the Fedora Docs workshop meet your expectations?
What topics would you want to see more of at next Docs workshops?
Please share any additional comments, thoughts, suggestions for next workshops.
This is a project-wide questionnaire that requires consensus with other working groups and Fedora Join SIG with @ankursinha . I’m happy to engage with other working groups to agree on survey questions and practicalities.