Infrastructure and Release Engineering onboading/mentoring workshop report/feedback

[kevin (Kevin Fenzi)]

Greetings infra folks (and any others)!

At the recent 2023 flock in Cork, James Richardson and I held a workshop on onboading, mentoring and documentation in the Fedora Infrastructure and Release engineering teams. We had a very lively discussion for the first part of the workshop and then mush less so for the second part after Lunch. (Many people had to leave as it was the last day and other excellent discussions were going on too).

So, this is going to be a long post, but we wanted to recap what we discussed there, present the current plans for documentation changes, and then ask some questions that we didn’t get to or didn’t finish talking about.

First I started in with some background and a drawing (all be it small to read for the people in the room). Fedora Infrastructure and Releng have always been teams that depended on community involvement. While there’s some folks paid by Red Hat to work on these things, they act more as a flywheel ( The flywheel theory of community engagement - Duck Alignment Academy ) to keep things moving along and onboard new folks.

A expanded version of what I had drawn:

marketing / outreach

onboarding - apprentice program - drive by fixers - experienced folks

progression and learning

sysadmin groups

sysadmin main

offboarding

Basically a flow or set of flows for people to follow when they wished to contribute, either by joining as an apprentice and learning from there, or also handling cases of drive by fixes or people who are already knowledgeable who want to contribute.

Some of the items discussed:

• what CPE is (the community platform engineering team in Red Hat where
  many of the people working in this space are). There was discussion about
  changing the weekly CPE reports to be infra and releng one with a CPE section
  in it. We need to document the relationship and what it is.

• Mentoring. How can we match people to mentors? Some kind of greeter/person
  to direct new people. Can 1x1 mentors even work? What if there’s no mentors
  available or in the same timezones?

• Badges. Doing series of badges so people know what to do/learn.
  This is going to take the new badges re-write being available.

• Different types of newcomers: Totally new to linux, just want to fix something,
  know a lot, want to learn another area.

• Talk about permissions. We have always operated from the idea that
  you shouldnt request permissions, you should be granted them if you need
  to do something, but this wasn’t explained in docs at all. Also, there’s a
  lot of subjective ‘are you ready for this permission’ that perhaps should
  be more concrete and in documentation.

• Some discussion of offboarding. Perhaps tie some things to Fedora release cycles
  so people are removed when they become inactive. Also, having a mentor might help
  to see when someone wants to step back.

So, based on all this, we came up with the following outline for docs
(subject to change):

getting help for users

• How to report issues / how our team works day to day
• How to contact us

Introduction

• pointer to “one off contributions”
• who we are and what we do
• development
• operations
• release engineering
• what is CPE? How does it relate?
• stakeholders and customers
  • fedora council
  • fesco
  • developers
  • users
• overview of the rest of the docs

Philosophy

• default to open
• polite, friendly, welcoming
• teach others who want to learn
• ask in the most public space you are comfortable with
• How our permissions model works
  trust and permissions gained over time as needed
  when you might be given more permissions
• sysadmin-* groups
  Usually specific apps and areas.
• sysadmin-main
  how and why people are added
  what kind of perms do they have
• Red Hat only resources
  explain hardware budgeting, etc

One off contributions

• what sort of account(s) do you need?
• explain how to submit PR’s/MR’s
• explain how to figure out where
• expectations

Joining the team

• Apprenticeship track

  • intro tasks
  • make an account
  • (other non philosophy getting started things)
  • get a mentor assigned
  • pool of tasks to learn from / qualifications list
    • attend a meeting
    • run a meeting
    • shadow an oncall person
    • be the oncall person
    • submit a PR/MR
    • solve a ticket
    • get X karma answering questions in matrix
    • login to batcave01
    • run an ansible playbook
    • enroll a 2fa on your account
    • … brainstorm and setup a bunch more…
  • be promoted to a sysadmin group

• Experienced track

  • bunch of questions about this one

• Developer track

  • intro tasks / setup
  • setup development env
  • how to find issues to work on
  • … bunch of other tasks …
  • bunch of questions about this one

Leaving the team

• When someone announces leaving
• Some kinds of cleanup processes for inactive members?
• Tied to fedora release cycles?

We need to fill in a good deal more things, but hopefully this docs setup makes sense?

There’s a bunch of outstanding questions we would love to get feedback on:

• Need to brainstorm a bunch more things for the qualification list.
  I think it would be great to have just a ton of these to get new people
  interested doing the ones they find interacting and learning more.
  We also need a way to check off things (badges might be great,
  but perhaps we need a tracking ticket or something?)

• should we fold developers into operations/apprentice?
  Or should they have a completely different path?
  What does onboarding/progression look like on the development side?

• How can we tell what to do with experienced people?
  Make them follow the apprentice track or part of it?

• What kinds of cleanup/inactive process do we want? after every cycle?
  We need to list all the places to clean up and how to do it.

• How can we match mentors/mentees if we try and do 1x1… and
  if we don’t have enough mentors what do we do?
  or not enough in some timezone?
  Can group mentoring still work?

So, if you made it to the end, thanks! Here’s a cat ( ) I hope we can work up a nicer onboarding workflow and get more people interested and involved. Feedback in particular from people who would like to use these docs would be very welcome.

Note that tomorrow I am off and then the weekend and then monday is a holiday,
so I may not reply too much here until next week, but I will definitely read
all the feedback.

[zlopez (Michal Konečný)]

@kevin

• Need to brainstorm a bunch more things for the qualification list.
  I think it would be great to have just a ton of these to get new people
  interested doing the ones they find interacting and learning more.
  We also need a way to check off things (badges might be great,
  but perhaps we need a tracking ticket or something?)

The idea with ticket sounds nice. The onboarded person could ask all their questions there and everybody from the current infra or releng folks could answer them if they know.

@kevin

• should we fold developers into operations/apprentice?
  Or should they have a completely different path?
  What does onboarding/progression look like on the development side?

It would be probably best to treat them as separate group as they have different workflow and in most cases don’t need access to infra.

@kevin

• How can we tell what to do with experienced people?
  Make them follow the apprentice track or part of it?

It’s hard to say who is experienced till they do at least something in infra. So making them part of apprentice track and when we see that they are experienced we can give them more permissions when they need them.

@kevin

• What kinds of cleanup/inactive process do we want? after every cycle?
  We need to list all the places to clean up and how to do it.

The Fedora cycle sounds right, if somebody wasn’t active for one whole Fedora cycle that is usually the sign that they don’t have much time and don’t be again soon. We can always add them back when they return.

@kevin

• How can we match mentors/mentees if we try and do 1x1… and
  if we don’t have enough mentors what do we do?
  or not enough in some timezone?
  Can group mentoring still work?

I don’t think we need 1:1 mentor/mentee. One mentor could have multiple mentees. We just need to make sure that there isn’t too much mentees on one mentor.

[seddik (saibug)]

Thanks @kevin for sharing this report…
So my questions are :

I’m very interested to know, what’s the best definition of, contributing and contributors?? .

What are the limits for fedora contributors (public) to achieve the tasks , access permissions?? Knowledges ?? Etc…

How to get involved to the infra community??

Thanks
saibug

[kevin (Kevin Fenzi)]

Thanks @kevin for sharing this report…
So my questions are :

I’m very interested to know, what’s the best definition of, contributing and contributors?? .

In the case of infra and releng, I’d say anyone who submits pr’s,
answer’s tickets or questions, fixes issues, etc.

What are the limits for fedora contributors (public) to achieve the tasks , access permissions?? Knowledges ?? Etc…

I’m not sure I understand this question… you mean, what permissions
map to what tasks? I think it might be very hard to document that, but
of course some of it could/should be.

How to get involved to the infra community??

Thats whats we are talking about documenting and working on here.

Basically you pick a track (apprentice, developer, etc) and then we have
docs that tell you things you can do to learn more and do more over
time.

[kevin (Kevin Fenzi)]

The idea with ticket sounds nice. The onboarded person could ask all their questions there and everybody from the current infra or releng folks could answer them if they know.

Yeah, perhaps we make a fedora-infra/apprentice tracker or something.
Or fedora-infra/onboarding.

We would have to make sure the entire team is watching/reading the
comments there tho.

It would be probably best to treat them as separate group as they have different workflow and in most cases don’t need access to infra.

Yeah, makes sense.

It’s hard to say who is experienced till they do at least something in infra. So making them part of apprentice track and when we see that they are experienced we can give them more permissions when they need them.

Yep.

The Fedora cycle sounds right, if somebody wasn’t active for one whole Fedora cycle that is usually the sign that they don’t have much time and don’t be again soon. We can always add them back when they return.

Yeah, it seems natural to tie our processes to the fedora release cycle.

I don’t think we need 1:1 mentor/mentee. One mentor could have multiple mentees. We just need to make sure that there isn’t too much mentees on one mentor.

Yeah, but how much is too much? And how do we map them?

[jakfrost (Stephen Douglas Snow)]

@kevin

Yeah, but how much is too much?

There is a limit of apprentice mentoring that has been established in the Trades for some time. I have never seen a Journeyman with more than three. This is based upon two primary concerns …

1. Ensuring adequate knowledge is being learned on the job, this is validate by periodic testing by the oversight body.
2. Ensuring safety of the workers, journeyman included. There should never arise a situation where safety of a mentor/mentee is compromised by a lack of effective supervision/guidance due to overloading the journeyman/mentor.

The safety aspect can be of concern, as in, did the mentee break something due to the mentor not having enough time to properly check the work?

[jflory7 (Justin W. Wheeler (JWW))]

Thanks @kevin for following up on the Flock discussion and carrying this forward with @jrichardson.

@kevin

Different types of newcomers: Totally new to linux, just want to fix something,
know a lot, want to learn another area.

You did not ask for feedback here, but I thought this analysis was interesting. They write as possible personas for the documentation. Each newcomer persona has different motivations that lead to different outcomes, and that influences how that newcomer would travel through Fedora Infrastructure as a contributor.

For example, someone who wants to fix something specific may only want to contribute to one task and only one task. They be good candidates to train and mentor on that one task if they are willing, but they may not be good mentors for other parts of Fedora Infra. Someone who wants to learn is more motivated to try some of everything. This person is flexible for what they help with. They may make good generalists around Fedora Infra, or they may only want to learn something and contribute over a short period before they go on to something else.

Perhaps the contribution pathways you outlined could be shaped into different personas, so they are more relatable to readers with different motivations.

@kevin

So, based on all this, we came up with the following outline for docs
(subject to change):

My first impression is that this outline is thorough. But is it too thorough? What I mean to say is that the scope is big and ambitious. It is a noble quest, but could this be scoped down to different smaller quests/phases? For example, what would be considered the “Docs MVP” as most important topics to document and publish first? What content could answer the most common questions that keep coming up? Then, once that is published, what would be Phase 2 of the onboarding/mentoring docs revamp?

Basically, I like all of the topics. They are thorough and thoughtful. I would like to see more about what a timeline could look like.

@kevin

We also need a way to check off things (badges might be great,
but perhaps we need a tracking ticket or something?)

I have thought about this for a while. But what if we offered a certificate of completion as a Fedora Infrastructure Apprentice, class of YYYY? What I need in order to do that is a basic outline of what Fedora Infra Apprenticeship is and what is satisfactory as evidence for completing an apprenticeship.

There are several reasons why to do this, but the best one I have is for career recognition and skill development. By offering a certificate, Fedora provides an apprentice with a useful tool that helps them build and develop their own career. This is something they could write on their LinkedIn profile, write into the resume/CV, and/or publish in a personal portfolio.

The benefit is that we incentivize a new generation of contributors to participate in Fedora and receive a tangible benefit of getting involved. In exchange for completing small tasks and learning about open source infrastructure best practices with mentors, they receive a career stepping stone that they can use to pursue careers in areas that are relevant to Fedora. This also creates a new level of trust that could be used for vetting individuals for gaining more access to Fedora systems^[1].

The downside is that we invest time and effort into training up a newcomer, and once they receive their certificate, they disappear from the Fedora community and we never see them again. Doing something like this, we should not expect 100% retention, but we could consider something like 75% as successful? More? Less?

What do you think of this idea? Does it widen the scope too much? I would volunteer myself for the paperwork and admin side of this idea, but what I need to succeed is a basic outline of what Fedora Infra Apprenticeship is and what is satisfactory as evidence for completing an apprenticeship.

@kevin

should we fold developers into operations/apprentice?
Or should they have a completely different path?

I like a single entrypoint where prospective apprentices/newcomers are “triaged” by an experienced team member before moving forward. This is similar to how the Join SIG does it and I think it works well for them. By having a single entrypoint, then an experienced member of Fedora Infra could find the right person to work with a prospective apprentice depending on whether they want to do development or operations work.

@kevin

What does onboarding/progression look like on the development side?

The Websites & Apps Team likely needs to be a stakeholder representing the development side.

@kevin

How can we tell what to do with experienced people?
Make them follow the apprentice track or part of it?

I defer to the above about a certificate of completion for being a Fedora Infra apprentice. The certificate could solve this problem as being a first entrypoint for any newcomer to achieve a baseline of understanding with Fedora Infrastructure, our tools, and our workflows.

@kevin

How can we match mentors/mentees if we try and do 1x1… and
if we don’t have enough mentors what do we do?
or not enough in some timezone?
Can group mentoring still work?

I would initially focus less on matching. Instead, I would focus on the personas you described earlier. What are the different outcomes that a newcomer might be wanting to achieve? Is it to learn, is it to do the one task they always wanted to do, or is it to be a generalist? The kind of mentoring needed (and who would be the right mentor) would be easier to solve after framing it this way, in my opinion.

---

1. I suggest also combining any certification to the year that the apprenticeship was completed. This is fairly common, and also allows us to distinguish what kind of experience or interaction an apprentice had when they were certified. For example, if someone showed up today with a hypothetical Fedora Infra Apprentice certificate from 2013, it has a different context than a certificate from 2022. ↩︎

[rwright (Robert Wright)]

@kevin

what CPE is (the community platform engineering team in Red Hat where
many of the people working in this space are). There was discussion about
changing the weekly CPE reports to be infra and releng one with a CPE section
in it. We need to document the relationship and what it is.

I just wanted to highlight this piece and thank you for including it in your summary. CPE is a “Red Hat Team” who also are folks who are deeply involved in Fedora Infra. My hope is that more apprentices join Fedora Infra Apprentices out of the onboarding ramps from Fedora Join / other avenues to this for the team and it’s clearer the relationship/difference between the two.

[kevin (Kevin Fenzi)]

Thanks @kevin for following up on the Flock discussion and carrying this forward with @jrichardson.

Thanks for the feedback! Sorry for the delay replying…

@kevin

Different types of newcomers: Totally new to linux, just want to fix something,
know a lot, want to learn another area.

You did not ask for feedback here, but I thought this analysis was interesting. They write as possible personas for the documentation. Each newcomer persona has different motivations that lead to different outcomes, and that influences how that newcomer would travel through Fedora Infrastructure as a contributor.

For example, someone who wants to fix something specific may only want to contribute to one task and only one task. They be good candidates to train and mentor on that one task if they are willing, but they may not be good mentors for other parts of Fedora Infra. Someone who wants to learn is more motivated to try some of everything. This person is flexible for what they help with. They may make good generalists around Fedora Infra, or they may only want to learn something and contribute over a short period before they go on to something else.

Perhaps the contribution pathways you outlined could be shaped into different personas, so they are more relatable to readers with different motivations.

Yes. Thats quite what I was getting at.

I can see operations and developer as different personas sometimes at
least, but sometimes someone will be interested in both. Or in something
like working on documentation or marketing/communication.

I guess fedora infra is sort of a microcosim of the entire project.

So, I think some personas could be helpfull, but we shouldn’t try to be
too rigid about it.

My first impression is that this outline is thorough. But is it too thorough? What I mean to say is that the scope is big and ambitious. It is a noble quest, but could this be scoped down to different smaller quests/phases? For example, what would be considered the “Docs MVP” as most important topics to document and publish first? What content could answer the most common questions that keep coming up? Then, once that is published, what would be Phase 2 of the onboarding/mentoring docs revamp?

Yeah, it is.

I was pondering on how to tackle things, and perhaps a phased approach
would be good. Also, since it’s moving a lot and adding a lot, I am not
sure if we should:

1. Just do pages as we do them and push them out

or

2. Make a staging branch, push a bunch of things to that and update the
   stg site until it looks like it’s in a reasonable state, then merge that
   all out to prod.

Any thoughts from anyone?

as fas as phases, we could look at:

1. All the top level pages

then

2. adding to apprentice / developer tasks, etc.

Basically, I like all of the topics. They are thorough and thoughtful. I would like to see more about what a timeline could look like.

Thanks.

I am not sure on timeline. It’s really hard to say since so much of our
jobs is firefighting, it’s hard to get planned work time, but I am
hoping we can get time later this year when f39 is out to work on this.
It’s a good task for more quiet days, IMHO.

I have thought about this for a while. But what if we offered a certificate of completion as a Fedora Infrastructure Apprentice, class of YYYY? What I need in order to do that is a basic outline of what Fedora Infra Apprenticeship is and what is satisfactory as evidence for completing an apprenticeship.

Right. I was hoping many / most / all of the tasks would be something
that could have a thing that could be checked by the mentor/anyone.
ie, ‘ran an infra standup’ would be something anyone at that standup
could tell, and have a fedora-message related to it too.

There are several reasons why to do this, but the best one I have is for career recognition and skill development. By offering a certificate, Fedora provides an apprentice with a useful tool that helps them build and develop their own career. This is something they could write on their LinkedIn profile, write into the resume/CV, and/or publish in a personal portfolio.

Good points.

The benefit is that we incentivize a new generation of contributors to participate in Fedora and receive a tangible benefit of getting involved. In exchange for completing small tasks and learning about open source infrastructure best practices with mentors, they receive a career stepping stone that they can use to pursue careers in areas that are relevant to Fedora. This also creates a new level of trust that could be used for vetting individuals for gaining more access to Fedora systems^[1].

I think this might be something we could do for a ‘phase 3’ after the
initial docs are in place.

The downside is that we invest time and effort into training up a newcomer, and once they receive their certificate, they disappear from the Fedora community and we never see them again. Doing something like this, we should not expect 100% retention, but we could consider something like 75% as successful? More? Less?

Yeah, thats always the case. Most people have time and energy to learn
but then things change or they move to another part of the project they
enjoy more or get busy with life, etc.

I personally don’t think we should mind at all that we are teaching
people who move on. Thats just spreading the knowledge and joy around.

What do you think of this idea? Does it widen the scope too much? I would volunteer myself for the paperwork and admin side of this idea, but what I need to succeed is a basic outline of what Fedora Infra Apprenticeship is and what is satisfactory as evidence for completing an apprenticeship.

I think it’s an awesome idea. I think it might make sense to wait until
we have docs so we can answer those questions better.

I like a single entrypoint where prospective apprentices/newcomers are “triaged” by an experienced team member before moving forward. This is similar to how the Join SIG does it and I think it works well for them. By having a single entrypoint, then an experienced member of Fedora Infra could find the right person to work with a prospective apprentice depending on whether they want to do development or operations work.

Yeah. I think there’s overlap along with people who want to do things in
both areas. If you join and do a bunch of development tasks thats just
fine, you shouldn’t be required to do anything you don’t like to.

The Websites & Apps Team likely needs to be a stakeholder representing the development side.

Indeed. Or… we point to them for that side of things more clearly.

How active is websites and apps these days?
They did a ton of great stuff, but I’ve not heard too much recently…

I defer to the above about a certificate of completion for being a Fedora Infra apprentice. The certificate could solve this problem as being a first entrypoint for any newcomer to achieve a baseline of understanding with Fedora Infrastructure, our tools, and our workflows.

Yeah, if we had that we could also have some way for someone to get it
more quickly if they know a lot already. Do note however that some
things are good for everyone to learn, no matter how much they know
about sysadmin or development. Ie, how to interact with our community,
how things work, etc. I wouldn’t want to bog them down, but I think
having them learn those project specific things would be good.

I would initially focus less on matching. Instead, I would focus on the personas you described earlier. What are the different outcomes that a newcomer might be wanting to achieve? Is it to learn, is it to do the one task they always wanted to do, or is it to be a generalist? The kind of mentoring needed (and who would be the right mentor) would be easier to solve after framing it this way, in my opinion.

Yeah. I am kind of thinking perhaps now we should have a tracker that we
open a issue for new folks and everyone can help them there and track
their progress, etc. We would also need to make sure we have a way to
close/timeout people who are no longer around/interested so we don’t
have a ton of tickets hanging around.

Thanks for all the great feedback!

---

1. I suggest also combining any certification to the year that the apprenticeship was completed. This is fairly common, and also allows us to distinguish what kind of experience or interaction an apprentice had when they were certified. For example, if someone showed up today with a hypothetical Fedora Infra Apprentice certificate from 2013, it has a different context than a certificate from 2022. ↩︎

[kevin (Kevin Fenzi)]

Absolutely. Right now we just kind of assume people will ask someone
what CPE means to figure it out. We really need to spell things out in a
page that explains in detail.