This is a very strong assumption. There are posts (such as those I linked above) of community members that suggest otherwise, and they got several “100%” flags from several other members. We do not know if that is representative either, neither is it a widely read category, nor is the number high enough to suggest to be representative. But such comments had came up in many discussions over time, and we can say with a high likelihood that they represent a part of this community (which part, and how big it is, remains unclear). So the question is, if you chose to not use this indication but use some other indication to derive this assumption, what indication is that? I try to help: derivations, assumptions, limitations, etc → creating them is not a trivial task, as elaborated earlier. But it still feels as if it is handled that way.
I would not read into the incomplete response data too much, because LimeSurvey does not actually make this meaningful and useful unless we collect additional data like timings
Again, I see a strong presumption (I think actually, several): is there something to support them?
How do you know that people did not reject these questions due to the contained presumptions? We have data that suggests this is the case in some cases, we don’t know for sure how many and if they are representative, but its actually the only data I have seen so far that can answer this question for “some” people. Do you have data that suggests another group? To me, this feels a little like selective use of data, but without a reproducible/clear foundation.
A survey provides us some sort of structure to start collecting insight and feedback
Not if you do not know the group it represents. This makes the data arbitrary and essentially useless. If people have to accept what Fabio named a “premise”, to which they actually don’t agree, they have to choose to not take the survey too seriously and answer some questions willingly wrong, or choose to cease participation. Both leads to the tautology, and/or the groups being no longer known, I described. Now, if users can choose to not answer single questions, giving them a less invasive means than ceasing participation at all, it is decided this is can be ignored? Again, this leads inevitably to a tautology… and the related reasons why many have already chosen to no longer engage in surveys.
Would you be willing to join the Community Ops Team as someone who could better help steward our surveys?
The problems begin with this question: it should start with “What qualification do you have?”. Again, this is not a trivial task. People are doing a lot of training to create expressive data, and much more training to interpret it.
At this time, my answer is no: this answer is partially the outcome of the question I have been asked (would you join rather than how are you qualified). I once answered this question with yes (many years ago), and then I was presented with presumptions that had to be taken for granted, no one cared of defining limitations or actual research questions (“takes too much time, for what do we need that, we just need to know!”-like): just get the survey done and published. Essentially, the evaluation of questions already went along with the preferred derivation that could be only confirmed by most of the survey, in order to later ignore the data that did not confirm what was determined in the beginning (“that part is not relevant or expressive anyway”-like). I really mean this as a constructive incentive, but before I engage in something like that again, I would expect indication that there is interest to do this in a scientific tangible way: this is not trivial, not easy, not quick. It takes its time and a lot of discipline.
That said, I am happy to provide review and give feedback: help to identify (not-yet-identified) limitations, presumptions, selective use, scope, and relations between questions and goals, up to identify if it goes towards a tautology.
But that means that any proposal that is provided to me must already include from the beginning:
- who is defining this and in what organization (research subject)
- superordinated research question that is to be answered
- group/audience (research object)
- how to approach+get this specific audience (publishing a survey to the everybody, distributing it through all channels and hoping for the best, is creating useless data unless the participants are (a) limited to the community [everybody or just FAS group contributors?] AND (b) the majority of those [active?] participates → the issues here are related to the issue of those people that have been already chosen to no longer participate in surveys → how to mitigate?)
- questions (+ elaboration of what each question aims to achieve)
- order of question (+ elaboration of the reason for the order, if there is one)
- limitations (maybe the most important question of all)
- plan how to evaluate the results (towards the research question)
This is the rough thing I have in my mind off the cuff (before my first morning coffee
): if something like this is created, I might consider to start to believe
… and to read carefully through it and give feedback about presumptions, (actual) groups, limitations, selections, and risks to end up in confirming a tautology etc: so feedback of what the team has already defined but also what I can identify additionally to that (if applicable). And if so, I am happy to also make this an iterative process.
Also, keep in mind that we might have more qualified people than me in this community. And to be honest, I would prefer to contribute if any person who has a qualification (I am aware that I leave myself a lot of space for interpretaton here) for this has a type of veto. My own contribution will be always limited as long as qualification is not the first question to be asked, as I expect it will determine the outcome, may it be at the level of creating a survey, or at the level of interpreting data.