Proposal: "Join" system

Project Discussion
,
1

I’m a new contributor in the Infra space, and I’ve historically put a lot of effort into creating great onboarding experiences at my previous employers. I’ve just started doing the same here, mainly focused on docs so far, and I joined the Join SIG about an hour ago.

The idea of improving onboarding is everywhere in Fedora. But what surprised me is that nobody really seems to have a finger on this particular pulse. I’d like to suggest the following new app, which I’m just calling “the Join system” for now.

High-level goals

  • Improve the success rate of onboarding.
  • Understand the success rate of onboarding / retention.
    • What percent achieved their goals? Why?
  • Understand the composition of our new members.
    • What drew them in, what skills they are bringing, how much time they have to contribute on average, etc.
  • Improve upon existing Join processes by automating them / providing automation opportunities.

Approach

  1. New users in FAS get plugged into a “Join” system. This sends them an automated email that A) offers to help them get oriented to the project, and B) requests that they participate in follow-up surveys so we can track how well we’re hepling people onboard. The email includes a link which opts them into the full “Join” experience.
  2. The user clicks the link to opt into the full experience.
  3. We give them a poll with a handful of questions like, “What convinced you to become a Fedora contributor?” “How excited are you about contributing right now? (Mildly curious, Highly motivated, Annoyed but determined, etc etc.)”
  4. All the existing Welcome Wagon stuff still happens. Perhaps this system can help automate some of it to make sure it happens / track how often it doesn’t.
  5. At various points in the new contributor’s journey (1 week, 2 weeks, 4 weeks, 8 weeks, 6 months, 1 year):
    • The system sends them an email with a very brief poll - no more than 5 minutes. This would include maybe 3 simple questions like, “On a scale of 1-10: I feel like I know what’s happening in Fedora and where I could contribute.” It would also include at the end, “Which of the following is the biggest obstacle to contributing more to Fedora?” We could also include a “Halp!” button for someone who still wants to succeed but feels like they’re failing, which would summon any available human outreach.
    • We collect metrics from various systems to measure “onboarding success” - number of Discussion comments, number of Pull Requests, etc.
  6. Once a quarter, the system emits a “State of Fedora Onboarding Report” which can become a Magazine post or whatever and probably drive future recruiting / retention efforts.

Next steps

  • What don’t I know? Does this already exist? Has it been tried and failed and retired? Objections?
  • Who needs to agree that this is a good idea? How many is enough to say, “We should try it”?
  • I have the skills and time (currently) to build it, but I don’t know anything about how new apps are born within Fedora. (Approval? Owners? etc.)
2

It sounds similar to this thing that I share a link to from time-to-time:

Maybe you can build on that?

3

That’s a great site for convincing people to click the Join button (recruiting), but this proposal is about everything that happens next (onboarding and retention).

Certainly this proposal could be integrated under that domain if it makes sense. But ideally I think it would avoid having any user-facing web UI at all and simply integrate with our existing tools. (For example, maybe we can use Matrix to issue the polls via bots.)

4

This is what we have in place now, if you haven’t had a chance to take a look. It includes quite a bit of what you’ve noted above:

We want to make sure newcomers interact with the people of Fedora, which is why we haven’t automated the ticket opening bit. A lot can be done to keep track of tickets and automatically ping them, gather metrics and so on, but we need to implement the scripts to automate this. Some bits already exist. We’re moving off Pagure to Forgejo soonish, so it’ll all be reimplemented, which is why we’re holding off on more automation.

1 Like
5

The process I’m proposing is essentially an augmentation to the existing process. See step #4 above. The differences are:

  • First contact scope:
    • Current system: only people who reach out to Join SIG
    • Proposed: every new FAS user
  • Contact requirements:
    • Current: A Join SIG staffer at each contact
    • Proposed: Nobody (though consistently offered and available!)
  • Data gathered:
    • Current: only what is manually shared in email or other comms
    • Proposed: automated, and sufficient to track onboarding and retention success by various cohorts and report on that. This is in addition to whatever we can gather with Join SIG contact.

I admit I’m totally new here, so there’s probably a lot I don’t know. Please do enlighten me!

What I can tell you is that I knew what I wanted to do, so I did not visit the “What can I do” site or go through the Join SIG – I went straight to my destination SIG. So the current process is not tracking me and whether or not I’m a “new contributor” success story. If some easily-addressed difficulties cause me to bail and Fedora loses out on another contributor, nobody will know that it happened, or why it happened, or how many others it happened to.

And I feel that this is likely to be the common case for most technical contributors. We tend to not seek help from groups like the Join SIG, but if Fedora isn’t tracking the success of those new contributors then we are likely missing out on opportunities to improve it.

3 Likes
6

I forgot to reply to this:

  1. So you’re saying this is the perfect time to consider some changes? :wink:
  2. We don’t have to wait for this to be complete to begin making progress.
1 Like
7

Like much of the internet, we are tracking you, but you don’t see it :wink:

Joking aside, Fedora contributor numbers are measured in various ways, such as active members, monthly commits, Matrix messages etc.
I’m not sure of the value of polls and surveys, above what we gather now in the ‘do-ocracy’ of Fedora. Output is everything.

We get a lot of new people coming in through Join - maybe 50 to 100 a year. I see two groups there - one that integrates quickly, and one that immediately drops off (and might come back, sometimes years later).

Will polls and tracking change this? I don’t think so. On Matrix, email and Discussion, people are available to help newcomers 7 days a week, 18+ hours per day. It is up to the user to engage - sometimes it is difficult - especially in infrastructure where one needs a long term reputation to gain access permissions.

8

I’m not for automation.

Community involvement is the key to long term Fedora success. People who want to contribute to Fedora know that they can find their way in. I think that establishing automated procedures will add a layer of dev and maint work, and will not add very much.

What I want is an engaged and active community to be available - and I want you to be a part of that as your motivation, ideas and conversation are excellent.

2 Likes
9

We have discussed sending out an automated e-mail to all new FAS account creators saying “not sure where to begin, speak to the Join SIG here”. This can certainly be done.

So, a join SIG staffer isn’t necessary at each time point. Lots of people do things themselves—they don’t reply to the ticket etc. It is better to have new people know that there is someone always available to help—the human connection is important.

This is also something we’ve discussed. We certainly need more data. The implementation here is very difficult because of the diversity of onboarding processes in Fedora. Each team has its own process. Some require formal steps, other don’t. Sometimes people just jump in and participate, other times people don’t communicate on the channels at all but are making contributions all the same. So, we have not been able to come up with a way that catches all onboarding activities. The tickets at least let us gather data on people that come through that channel.

This is not a new issue. Multiple teams have tried to figure out how to do this. If there are people that would like to look into this more, that’ll be great—any progress here will be welcome.

People like you (us) that know what to do don’t need a lot of onboarding help. However, people like us are also not necessarily the majority. What the onboarding process tries to do is cater to the uninitiated—the user who has gone from “this is a cool OS/community” to “hey, maybe I can join the community—but what do I do?” So, in the big picture, I don’t mind if more technical people with FOSS experience jump right in and don’t get tracked by us. What I’d really like to be able to do is track the people that do need a helping hand and see what we can do to increase our conversion rate with them. Does that make sense?

On certain things we do—scripts that will gather data from Pagure will not work with Forgejo. They will need to be re-written. So the question is whether we want to do it now and do it again soon, or if we wait for the migration and do it once later. There’s no right answer here, it depends on who has the cycles—if folks have the cycles now, they can go ahead and do it now.

I’ll soften that to say “I’m not for 100% automation”. I’m sure there are things we can automate. However, if this will replace an opportunity for human contact, I’d rather not do it.

The people that stick around in Fedora don’t stick around because they enjoy doing tasks. They stick around because they have made friends in the community, and even if they don’t have the time to do tasks, they can still hang around with their friends—and then when they do have a minute, do a task. The alternative—people stick around while they have time to do tasks—is not sustainable, because as volunteers, there will always be times when we cannot find free time to devote to Fedora.

What’s really great about the Join SIG now is that we’ve gone past the “critical mass” that is required to make it self sustaining. So, a few of us can drop off from time to time but others are always there to help newcomers. This is already better than what we’ve had in the past, so we’re improving already :slight_smile:

3 Likes
10

I’m responding here to both of the above comments. And thank you for engaging on this!

Tell me more! (In Matrix or email, if you don’t mind, to keep this thread on-topic.) I assumed there was some tracking but I haven’t stumbled across any of it / the analysis it enabled. Some of the things you mentioned are exactly what I’ve thought of building, so obviously let’s build on that.

Not directly, no. But I do think that knowing more about the people who try to join and which of them succeed would only give us more insight into how we can improve.

Certainly, more apps = more work. And that’s not to be taken lightly in a volunteer org.

Regarding predictions of success – is my proposal something that we could back out of easily if we tried it and your predictions proved correct? From the very little that I know, the answer there seems to be yes. So how do you feel about trying it if that is true?

:heart:

100%. It is only my intent to augment this. A few thoughts:

1: I didn’t do a great job of calling out that “improve human connection” is a core design goal of my proposal. (I assumed it was implied, which is a mistake I make too often.) But I agree that we should analyze every part of the proposal (and Join SIG processes in general! and almost all technology!) through the lense of, “will this improve or diminish human connection?”

2: I believe (possibly incorrectly) that periodically putting a link in front of people that says, “Click here to summon a human! We’re here to help!” will increase the amount of human connection. This turns the interaction from one that they have to initiate into one that they can simply respond to, thus lowering the “activation energy”.

And of course, this process must respect their preferences, which is why I said it should be opt-in. This includes an opt-out link too.

3: The idea behind “get more data” is “help us understand how to better facilitate those human connections”.

  • I want to give all humans who take that first step of creating a FAS account to feel that this matters greatly to us. I think that we can send this message both explicitly in automated email verbiage, and implicitly by the sort of questions we ask people in the polls. (Again, in addition to the human conversations we have!)
  • We also can demonstrate our values here by sharing publicly what we’ve gathered and how we’re thinking about it. I have not seen anything similar anywhere else in Fedora’s branding or publications. (This is really about Mindshare, btw. Should we ring them in?)

Yes, I totally agree. It will be very different to identify success in e.g. Infra onboarding vs. Join SIG. I do think that it is possible – especially if we have an eye on this from the beginning.

Tell me more! (In Matrix or email?) I of course want to learn from the past.

Hmm … I’m of two minds here. I want everyone to find the assistance they need.

So on the one hand, absolutely, those who seek help should find it. I think it’s correct to focus on that as a first priority.

On the other hand, I don’t think there’s a good reason to ignore the success or failure of people whose first impulse is to look for a git repo rather than a human. They too are core to our success, and in fact it is this group who might benefit the most from a broader scope / comms that we initiate. So while it might not be the first priority, I don’t think it’s far behind if we can achieve it.

Yep, understood. Though A) I think there is plenty we can do now that doesn’t need to touch Pagure or other work in flight. And B) nothing is permanent, so whatever we build should be modular. For example, we may never get fully off of Gitlab or Github, and who knows whether we’ll be on Matrix in 10 years. There may even be some relevant IRC rooms still in use.

:heart:

1 Like
11

Prior art

It turns out, this is not a new idea! It even is (or was?) a part of Project 2028! I’m going to (continually) collect in this thread the relevant resources I’ve found since starting this discussion.

Most of this is currently WIP or TODO for me to ingest.

Philosophy / design

Tooling

  • “Fedora Contributor Activity Statistics (FCAS)” aka “Fedora User Activity Stats Service (FUAS)”
  • find-inactive-packagers - script that queries for some of this data on a per-user basis
    • We probably wouldn’t want to run this against the entire userbase because it hammers a bunch of our services, but perhaps a restricted list of “users who joined within the past year” would be acceptable.
    • Sources:
  • “CHAOSS” - Community Health Analytics in Open Source Software
    • A relatively new Linux Foundation tool (and community) to measure the health of open source projects.
    • Includes “Newcomer experience” measurements.
1 Like
12

Sadly, it seems that we have no place in Fedora today (or anywhere on the horizon) where we can effectively share and collaborate on documents such as my above “Prior art” findings comment. I’m still making progress on this investigation, and I had hoped to quietly update the info above as that work continued, but as it turns out: that’s not how Discourse works.

So instead, I’ve created Michael Winters / fedora-metrics-notes · GitLab where I’ll be updating my thoughts and findings. My impression is that we’re generally “open to the idea”, so I expect to be quiet on this while I plug away, and I hope to return to the conversation here in the near future with good news.

I want to make this effort slightly more visible, if only to inform any interested parties / lurking collaborators that it’s WIP, so I’ll post to the mailing list and Matrix in a few places before going dark. If anyone already on this thread is one of those creepy lurkers, hit me up in Matrix (mwinters) or email.

1 Like
13

Maybe you are looking for Discourse’s wiki mode?

1 Like
14

Thanks, I considered / attempted it. However it’s not really a solution IMO for anything other than “FAQs” or similar:

  • You can’t convert a post into a Wiki after it’s turned to stone (24h I think?)
    • So make every new post a wiki, just in case you later want to collaborate?
  • According to the upstream docs you must be at least trust level 3.
  • I don’t think you can convert comments into wikis. So I would need to start a new thread for each new document, even if they’re all on the same topic, thus sprawling the convo across Discourse.
    • If my topic needs multiple docs, should I make a “topic” wiki that links to all the relevant wiki posts on a topic, just so we can have folders? Both messy and noisy for those who subscribe to notifications.

So no, I don’t think wiki mode is a good general solution for docs. But maybe I’m just holding it wrong?

1 Like
15

I’m sure its possible to get you some Fedora space at some time soon.

There is the Fedora wiki, which is mostly deprecated, but still used - or we can talk to the docs team and see what we can do there.

Many teams use Gitlab, HackMD or Codeberg for works in progress. And you can use the issue tracker on Fedora Join if you want.

17

Oh, please, not yet another place to store Fedora related information. Our current documentation / discussion / information space is already excessively fragmented enough.

To have really a discussion about a document, we would need something as HackMD.io in Fedora space, or at least EtherPad. Unfortunately, however, there is a lack of suitable OSS software and/or time slots in our infrastructure team.

For the time being, the best solution seems to be to create a document on HackMD and keep referring back and forth from here, Matrix, or the mailing list thread.

2 Likes
18

The Fedora wiki isn’t quite deprecated. (Software/infrastructure wise it’s very well maintained.) It’s just no longer being used for documentation, as it was before the docs infra was set up. This sort of thing is what it’s meant to be for—collaborative editing, especially for stuff that is not meant to be public facing documentation.

I would just use that.

@mwinters this is not documentation yet, right? It’s notes etc. at the moment. If/when it becomes documentation, it can moved to the right bits on the docs.

2 Likes
19

So, this is news to me. The general impression (apparently both mine and that of others) is that it’s on its way out and thus should be avoided. Some of the new docs smell faintly of old leather already (e.g. references to Fedora on IRC), so it’s not clear that mentions of the wiki are still valid. I think we should address / clarify that impression of deprecation in the docs. (I’ve added this to my already-long TODO list.)

I agree that the wiki (or similar, such as HackMD) might be a good place to put stuff like this, because I wanted to make my project notes available for anyone to contribute to and/or run with. On any of the git forges, this has to be an explicit permissions grant on a per-user basis, or living with the unneeded friction of making people do PRs.

I agree that a tool dedicated to collaborative writing would be ideal. Has this been proposed somewhere? Have we investigated which tools would be ideal and why? That legwork is the hard part, in my opinion. Adding this idea to my growing backlog of initiative proposals.

20

I think the definition of “documentation” changes depending on context, so:

  • It is not user-facing documentation, i.e. informs about something that moves slowly / is fairly “static”. I think this is the definition you meant?
  • It is contributor-facing documentation, i.e. informs about something that is currently in motion / dynamic. It is also WIP project output, where the project is to answer the question, “What are our options for pursuing these goals / this initiative?” So it documents what I’m trying to do, why I’m doing it, what I’ve done so far, status, and next steps.
21

So, this is news to me. The general impression (apparently both mine and that of others) is that it’s on its way out and thus should be avoided. Some of the new docs smell faintly of old leather already (e.g. references to Fedora on IRC), so it’s not clear that mentions of the wiki are still valid. I think we should address / clarify that impression of deprecation in the docs. (I’ve added this to my already-long TODO list.)

The wiki is not going away anytime soon.
Our QE folks use the wiki as their test case management system/results
reporting for release validation. Until that could be moved to something
else the wiki is going to be there.

docs on the wiki are actively being moved off to docs.fedoraproject.org
because wiki’s are not great for docs over time.

In any case the wiki might work for your needs, feel free to use it. :wink:

I agree that the wiki (or similar, such as HackMD) might be a good place to put stuff like this, because I wanted to make my project notes available for anyone to contribute to and/or run with. On any of the git forges, this has to be an explicit permissions grant on a per-user basis, or living with the unneeded friction of making people do PRs.

I agree that a tool dedicated to collaborative writing would be ideal. Has this been proposed somewhere? Have we investigated which tools would be ideal and why? That legwork is the hard part, in my opinion. Adding this idea to my growing backlog of initiative proposals.

We used to run a collabrative document service (who’s name no escapes
me), but it was difficult to maintain and only infra folks were using
it.

Since there’s so many free existing services ( hackmd.io, board.net,
etc), I don’t know that we have much desire to run a fedora local thing.

2 Likes