Introduction: Earlier this year, the community was invited to share their thoughts on a potential new initiative called “Fedora Verified“. The goal of this survey was not to make final decisions, but to listen – to understand what contributors value, where opinions differ, and what questions still need answering. …
Read More
The magazine article says there were 90 fully qualified responses.
But the linked analysis has the lowest answer rate for a question at 81 answers. The analysis says 241 people participated in the survey.
What explanation can be given for two thirds of survey participants not answering some questions?
2 Likes
Fedora Verified: Help Shape a New Way to Recognize Fedora Contributors - #16 by decathorpe
Fedora Verified: Help Shape a New Way to Recognize Fedora Contributors - #17 by py0xc3
1 Like
The results of Q8 were:
| Approach | Count | % of 81 |
|---|---|---|
| Moderately structured, with some flexibility in progression | 30 | 37% |
| Very clearly structured, with defined milestones and criteria | 23 | 28% |
| Lightly structured, allowing organic / self-driven progression | 20 | 25% |
| No strong preference | 8 | 10% |
Now, in the Key Themes & Recommended Analysis, you say:
Keep the structure flexible, not prescriptive
62% prefer moderate or light structure. The model should avoid overly rigid criteria and allow contributors to demonstrate value through multiple, flexible pathways.
But, when you look at the underlying data, it would have been equally true to say:
65% prefer moderate or very clearly defined structure
…and to use that number to recommend more rigid structure. It’s not clear that an unbiased analysis would have interpreted the data to mean “push in the direction of lighter structure”.
So questions on this:
- Why was the “62% prefer moderate or light structure” conclusion selected rather than the “65% prefer moderate or very clearly defined structure”?
- To what extent was AI used in putting together this analysis and writing the Magazine article? If any, can an “Assisted-By” tag be added, in compliance with the Fedora AI-Assisted Contributions Policy?
To put my thoughts in some place for future discussions:
I do not like the term “Fedora Verified” in general. It has a quite a lot of bad connotations for me, and also I don’t believe that it correctly explains the desired system.
I think that what we are not aiming to verify, as in to judge, whether or not you are worthy to be a Fedora contributor.
I think the concept we discussed originally was closer to “the Fedora Project found a way to acknowledge your contribution”.
So if we decide to go further with this whole initiative, I would propose to rename it to something like Fedora Ack.
–
I do agree with the main conclusion of the article though: we need more ways to acknowledge more types of contributions. We need to get creative, but figure out the ways to support the creativity with some data points.
The data does help a lot, even when it can not cover everything and every flavor of contribution.
Do more dashboards 
2 Likes
It does: it says the majority of people who saw the survey have chosen to not participate, and the majority of participants have rejected to respond to some questions. We have strong indication this data is not expressive for this community, nor are the presumptions that are integrated into the survey. The survey as such has been rejected, along with its premises/presumptions. I still think Fabio summed up the points best that many have made in the past in many discussions, until they gave up and became silent, in his post I linked above…
The low number of answers and participants are the sole means people still have to add their opinions if surveys create premises/presumptions that are taken for granted (but to which people disagree) and that can be only confirmed with any response (…when surveys become tautologies…). This number is expressive… but it seems to not matter and, at least so far, was just ignored in derivations. Again, all that was discussed already …
1 Like
My point is - I don’t think we are going to do any decisions on the original Verified/Membership/.. proposal soon. Not based on this survey results.
But I would like to encourage the work on highlighting and measuring various type of contributions in general.
Systems like https://www.stackalytics.io/ provide interesting information about the project community. I would love to see something like that in Fedora.
2 Likes
This is attributable to the quirks of LimeSurvey with how it counts complete versus incomplete responses. Over the course of using our LimeSurvey instance, I have found this data to be misleading, and because we do not track IP addresses or set browser cookies, we have no way of knowing whether the same person opened the survey five different times, or whether five incomplete responses are actually five different people.
We have not published raw data from surveys before due to anonymization concerns, but the survey responses here allowed us to be more open with the results. In hindsight, I think we should have filtered out the incomplete responses in the data dump because they are misleading. I would not read into the incomplete response data too much, because LimeSurvey does not actually make this meaningful and useful unless we collect additional data like timings (i.e., how long someone takes on each page). This feels mildly invasive, so we typically do not use it.
Absolutely. Whenever this conversation gets picked up, I feel like Fedora Verified is unlikely to be the final name. For now, it is all conceptual, as a way to get a sense of what the community thinks about this idea so far.
I do not agree. Is this data absolute, binding, and fully representative? No. I don’t think this claim was ever made. Was the goal to get every single contributor ever to engage and fill this out? No. Some people will have more to share and give an opinion on than others. A survey provides us some sort of structure to start collecting insight and feedback. It would be different if we were driving something forward with clear next steps and plans, but that is not the case here.
The discussion originates all the back to the February summaries from the Council Strategy Summit. This survey was an evolution of that original discussion.
Would you be willing to join the Community Ops Team as someone who could better help steward our surveys? It came up in discussion that we used to have the Mindshare Committee gate-keep surveys, but this didn’t make sense with the committee’s focus on events instead of infrastructure. We were discussing ways to build more community input on how we design and execute surveys, and perhaps bringing this into the Community Ops umbrella.
Is this something I could ask you to join us and help out with?
This is a very strong assumption. There are posts (such as those I linked above) of community members that suggest otherwise, and they got several “100%” flags from several other members. We do not know if that is representative either, neither is it a widely read category, nor is the number high enough to suggest to be representative. But such comments had came up in many discussions over time, and we can say with a high likelihood that they represent a part of this community (which part, and how big it is, remains unclear). So the question is, if you chose to not use this indication but use some other indication to derive this assumption, what indication is that? I try to help: derivations, assumptions, limitations, etc → creating them is not a trivial task, as elaborated earlier. But it still feels as if it is handled that way.
Again, I see a strong presumption (I think actually, several): is there something to support them?
How do you know that people did not reject these questions due to the contained presumptions? We have data that suggests this is the case in some cases, we don’t know for sure how many and if they are representative, but its actually the only data I have seen so far that can answer this question for “some” people. Do you have data that suggests another group? To me, this feels a little like selective use of data, but without a reproducible/clear foundation.
Not if you do not know the group it represents. This makes the data arbitrary and essentially useless. If people have to accept what Fabio named a “premise”, to which they actually don’t agree, they have to choose to not take the survey too seriously and answer some questions willingly wrong, or choose to cease participation. Both leads to the tautology, and/or the groups being no longer known, I described. Now, if users can choose to not answer single questions, giving them a less invasive means than ceasing participation at all, it is decided this is can be ignored? Again, this leads inevitably to a tautology… and the related reasons why many have already chosen to no longer engage in surveys.
The problems begin with this question: it should start with “What qualification do you have?”. Again, this is not a trivial task. People are doing a lot of training to create expressive data, and much more training to interpret it.
At this time, my answer is no: this answer is partially the outcome of the question I have been asked (would you join rather than how are you qualified). I once answered this question with yes (many years ago), and then I was presented with presumptions that had to be taken for granted, no one cared of defining limitations or actual research questions (“takes too much time, for what do we need that, we just need to know!”-like): just get the survey done and published. Essentially, the evaluation of questions already went along with the preferred derivation that could be only confirmed by most of the survey, in order to later ignore the data that did not confirm what was determined in the beginning (“that part is not relevant or expressive anyway”-like). I really mean this as a constructive incentive, but before I engage in something like that again, I would expect indication that there is interest to do this in a scientific tangible way: this is not trivial, not easy, not quick. It takes its time and a lot of discipline.
That said, I am happy to provide review and give feedback: help to identify (not-yet-identified) limitations, presumptions, selective use, scope, and relations between questions and goals, up to identify if it goes towards a tautology.
But that means that any proposal that is provided to me must already include from the beginning:
- who is defining this and in what organization (research subject)
- superordinated research question that is to be answered
- group/audience (research object)
- how to approach+get this specific audience (publishing a survey to the everybody, distributing it through all channels and hoping for the best, is creating useless data unless the participants are (a) limited to the community [everybody or just FAS group contributors?] AND (b) the majority of those [active?] participates → the issues here are related to the issue of those people that have been already chosen to no longer participate in surveys → how to mitigate?)
- questions (+ elaboration of what each question aims to achieve)
- order of question (+ elaboration of the reason for the order, if there is one)
- limitations (maybe the most important question of all)
- plan how to evaluate the results (towards the research question)
This is the rough thing I have in my mind off the cuff (before my first morning coffee 
): if something like this is created, I might consider to start to believe 
… and to read carefully through it and give feedback about presumptions, (actual) groups, limitations, selections, and risks to end up in confirming a tautology etc: so feedback of what the team has already defined but also what I can identify additionally to that (if applicable). And if so, I am happy to also make this an iterative process.
Also, keep in mind that we might have more qualified people than me in this community. And to be honest, I would prefer to contribute if any person who has a qualification (I am aware that I leave myself a lot of space for interpretaton here) for this has a type of veto. My own contribution will be always limited as long as qualification is not the first question to be asked, as I expect it will determine the outcome, may it be at the level of creating a survey, or at the level of interpreting data.
3 Likes
It comes from the six years or so I have been helping out with surveys. But now I’m the only one left helping run them.
I do not, but I also do not have the resources to make meaningful changes and do the follow-up engagement even if I did know the answer.
One could ask at what level we are trying to run surveys. If our goal is to run surveys as if someone is professionally employed to manage surveys, think about them all the time, and manage them end-to-end, then we should shut down LimeSurvey and stop running surveys ever again.
If our goal is to get a pulse check, then I think it works. We could also use Discourse polls for the same effect and get a similar level of engagement.
You seem to have a lot of insights. I am not mathematically qualified to be running a survey program, but I am here because nobody else is stepping up to do the work.
So, unless this changes, either I will burn out or the situation will continue as it is, or we should decide that surveys are bad ideas and we should cancel our LimeSurvey contract and go back to using Google Forms when we want to run surveys.
Yes, but you are the one here giving the feedback! I am certainly not qualified, but here I am. Burning myself out, trying to help maintain the services the community uses, and not getting the support from above me or beside me to do it.
Maybe we need to decide that LimeSurvey is not worth the investment, if nobody wants to do the work of running surveys more seriously.
I will still need to run surveys for Flock, but I am content to use Google Forms. The issue there is that many people objected to use of Google for surveys, which is why we pay for LimeSurvey. But if people are not happy and going to object to every survey we build and design, then something needs to change.
I just really don’t have the energy for this anymore. 
@py0xc3 Let’s move this discussion somewhere else, that is not going to also be repeated on the Fedora Magazine article too:
1 Like
I think the survey had some good insights so I wouldn’t say it was a waste of time, but I would agree that it might not be worth the time, stress, resources to do third party surveys. I would much rather see informal surveys here on the fedora forums at regular intervals (every 2-3 months) to see where the community is and how we all feel about certain things. It should be open to everyone with a fedora account in good standing that meets trust level 1 and has had an acount for at least 90 days.
I must also agree with bookwar…the name ‘fedora verified’ will unfortunately automatically drive people away from engaging with the survey 
…maybe something more like this:
“Creation of the Contributor Badge: Come decide what counts as a contribution! Vote Today~!”
Then on the actual forum page you can clarify that: this is a non-binding vote, meaning that all decision-making will go through the normal processes and procedures. Your vote helps to better inform the council on important decision-making.
1 Like
That was certainly the case for me… I opened it and looked at the first page of questions, decided I couldn’t really answer them and closed it again.
As I recall the problem seemed to be that it was asking questions that were seeking feed back on a proposal but it wasn’t really clear what the proposal was because as far as I knew there wasn’t a concrete proposal yet, and the discourse thread that I think it linked to didn’t seem to help much with understanding what the proposal was.