Is it time to shut down LimeSurvey and stop running surveys?

Continuing the discussion from Fedora Verified: What Does the Community Think?:

Hey CommOps folks, I wanted to open a discussion related to feedback @py0xc3 and @decathorpe shared in the above-linked topic or other topics referred from the one above. You can read the full context in that topic, but it is very clear that the community is not satisfied with the level of professionalism that we are running surveys. It seems to be damaging our reputation to run surveys.

I have been the only person tasked to manage surveys for a few years now. Others have helped as they had their own surveys to run, but due to billing challenges with how LimeSurvey has changed to a per-user billing model this year, we had to lock out several accounts just to keep our main “root” admin account active. Also, once the people who want to run their own surveys finish, they move on and do other things. Managing LimeSurvey is not part of their responsibilities. And I get it, it is fine. Not everyone who wants to run a survey will want to help manage all Fedora surveys too.

I know that data and surveys can be useful, but if the community will reject the surveys that we run and throws them out as invalid, we are missing the mark. I have been on a burnout track for a while, and LimeSurvey is just one of those things that I have been doing because it seemed like the right thing to do. But I don’t get any support from others to help run it, and the LimeSurvey billing model is putting a strain on us that makes it hard to onboard others now.

I am feeling like it is time to throw in the towel and say that it was good while it lasted. This does likely mean that when people want to run surveys in Fedora, either they will give up or they will likely use Google Forms. Maybe this is not the worst outcome. But the truth is, I am already overwhelmed and struggling to keep up with responsibilities, I am not getting support to help maintain and run surveys, and the community is upset about the level of professionalism with our existing process for surveys.

As much as I would like to spend time on this, I simply cannot. My work in Fedora is having impact on my personal life, and I have to find some things to drop.

So, I want to see if anyone wants to step up to own this, or if I should cancel our subscription at the end of this year.

4 Likes

This argument is not about survey tools.

It is about who has the right to be a citizen of Fedora. Questions about citizenship are very important questions.

The discussion now has become about staffing and responsibility and capacity.

And you are right that you, Justin, do too many things. The nature of the corporation is that they will work you as hard as they can. But you also have a lot of freedom in your role.

I think we should drop the Fedora Verified project. I think you should reanalyse your duties, and work more on top level community architecture than also attempting to be the ‘boots on the ground’.

2 Likes

Nothing is happening with this right now. The article was also super clear about this.

But we need this conversation to focus on surveys. Because this problem is going to keep happening, and keep happening, and keep happening.

It is not about the tool. Cancel the subscription. We have other tools we can talk about when you have capacity.

1: People

I want to commend you on being open about your burnout :heart: . I’ve burned out hard a few times, and induced some serious lifelong personal losses. The natural instinct is to hide it, which (speaking from personal experience) ultimately only intensifies it.

It’s important to normalize these conversations, so thank you. I also strongly suggest / request that you 1) shamelessly put out calls for help and 2) try to delegate or cut out as much work as humanly possible until your health improves. Do it until you feel guilty for working so little, then do it some more. You never know how much of your burden could be lifted unless you ask.

Burnout recovery is not just “reduce workload from >100% to 100%” – it requires reducing it to well below 100% for an extended period of time. Let me know if you need me to cite any sources, because it’s been studied extensively across several professions.

2: Process

I don’t think “having surveys” is the problem. There were several factors which combined to form a “perfect storm” around the last survey, but I think the root of the problem was that the process depends entirely on you to champion it, and you were absent for personal reasons. (That is not an admonishment! The wheels should not fall off the bus just because one person needed to take 5.)

Myself and others provided some feedback that it was not ready to go out, but in short that feedback wasn’t acted upon because there was nobody around to ensure / enforce that it got acted upon.

We see this all across the project – people are heroically supporting things that will fall apart quickly in their sudden absence. (Myself included!) It’s one of the reasons I started FDWG: to figure out how to shore up our contributor base to improve the resiliency of the project.

I think that surveys in general can be an extremely useful tool, and I even have ideas for some I’d like to run myself, though obviously we’re seeing that surveys can cut both ways. We could put some basic structure around this, like (spitballing):

  • Every official survey must have 3 “authors” committed to producing it, ideally representing at least 3 different areas of the project.
  • All authors must sign off on the final draft before it’s published.

Just an idea, but you get the gist: you must have at least 3 wheels on the bus to put it into Drive. And maybe some of the loudest critics of the last survey would like to be some of our new wheels?

3: Tools

As for the tool itself, LimeSurvey’s website frustratingly doesn’t seem to call out what’s provided by the paid version vs. self-hosted. I want to support open source, but if the financial cost is effectively costing us “community trust” due to poor process, then I think it’s clear that we must part ways.

If the community edition provides for our needs, then I think the answer is straightforward: just get this deployed somewhere. (Communishift?) I could probably even host this on Hatlas, though that might be awkward.

If the community edition is insufficient, then I would ask: what do we need in a survey platform? I think you’re likely the guy best equipped to answer that question, and perhaps this is the thread for that convo?

2 Likes

To be honest, I am cannot remember the last time that I had extra capacity to throw around. I don’t want to narrow in on this part too much right now, but I don’t think it is wise to wait for Justin to have capacity to pick something back up.

Thanks. :heart: To be honest, it was not the point of me making this topic, but I realized something has to change here.

It is a long holiday weekend, so after this reply, I will be offline until Tuesday. I hope to come back a little more recharged, a little more refreshed!

I confess that I am not always excellent at this, but hey, if anyone does want to help out more, here is a good opportunity. It is not urgent or anything for someone to step up on this right now, but I guess this is the window for someone with more passion and dedication to community surveys than me to help out.

I apologize for that.

I think you might have something workable here.

To be 100% clear, this is not a budget issue. We can totally afford to keep paying for it. It does not cost so much, although the per-seat pricing is… really annoying, and it is a pricing model which is always harmful for public, open communities. If we had a plan to manage it better, the cost is justified.

The issue here is that the management of the service is not good, and it falls into the pile of things that, “I guess the FCA will do that.”

I have wondered about what self-hosting might do for us, especially on this silly cap on the number of seats they have put into place now. I haven’t dug into the premium model or if there are any feature differences, but I am also so certain that we are not using advanced features. I could be wrong there, but we are not so sophisticated in how we use the tool. LimeSurvey is powerful, which is nice, but it is also complex.

Honestly, I think LimeSurvey does work well for us, but because it is a complex, challenging tool, it requires more cognitive load to configure and set up than something like a Google Form. I’m always remembering which knobs I have to turn and which settings I need to tweak when setting up a new survey. This is also the kind of thing that prevents me from looking as closely at the questions and the way they are designed, because I get stuck in the weeds on how do I even publish this survey correctly in the first place? It is not that hard, but it is complex enough that it doesn’t take me a minute to do.

I’m afraid you’ve missed my point, friend :slight_smile: . This is like apologizing for needing oxygen.

This, right here …

I only see maybe 2% of what you do, but I see you picking up a lot of pieces all across the project. In fact, I would challenge you to start a bullet list of all the different things you do. Like, not just “Run meetings” but “Run meeting for team X”, “Run meeting for team Y”, “Create all TODOs for team Y”, “Chase down people from team Y when they’re not doing their TODOs”, etc etc.

It would take some time to build, but I bet this list is truly enormous, and just seeing it all in one place might help you realize that your burnout is justified. It might also help you communicate to others the fact that perhaps they could pick up some of their own pieces, i.e. “What happens if Justin goes on permanent vacation?”

To be blunt, you’ve picked up a few of my pieces along the way, which I’ve genuinely appreciated, but also – if those pieces hadn’t gotten picked up, the consequences would have been mine. Instead they became yours, which isn’t really fair to you.

(I’m sure that having a large portion of this thread shine on you is uncomfortable – sorry about that. But I do see “the survey didn’t go well” as being closely linked to, “Justin is just too dang nice of a guy and is burning himself out trying to take care of everyone and everything.”)

Okay, let’s spec this out. Do you need anything more than this?

  • Question type:
    • Checkboxes
      • Include “Something else” textbox option?
    • Radio buttons
      • Include “Something else” textbox option?
    • Textarea

That’s all I remember seeing.

I answer by email to the head post of the topic, so I hope I do not repeat too much of what has been already said by others. I respond to both this and indirectly to the final post of the preceding topic.

I am very sorry that I have contributed to a development that seems to affect your personal life.

It is two partially-related topics that come together here. First, let me focus on the more important of the two: your health.

If you “destroy” yourself, neither you, nor the community have a benefit from it. It is in the interest of the community that you maintain a “healthy egoism” about yourself: first, focus on your obligations. I assume that effectively is your contract and your exact role in the Council. This is what you need to get done and that is your obligation to get what we all need, our salaries :smiley: Second, your personal life: you and your family. THEN, third, comes the community. This is in the interest of the community to achieve predictable long term contributions: you do this for yourself but also for the community. Only contribute beyond what you can and what feels balanced and good.

If you have a task that you cannot get rid of, just leave it be, and trust the community: if it is healthy, it will find its way to mitigate. May it be by someone taking over, or by the community to mitigate as a team by other means, adjusting itself etc. Also, see the positive side: while your “healthy egoism” can help the community to occasionally re-enforce its dynamics and learn to train its flexibility and to adjust, you can sometimes cause an intuitive streamlining process with this, in which the community gets rid of parts of a process, or a process as a whole, that is currently not the most efficient use of its resources, making the community more efficient and thus more strong on the long term. If the community is healthy, it can mitigate. Trust it :slight_smile:

The highest obligation that can be expected about these tasks in the “third” category is that you allow a handover: IF compatible with the healthy egoism, put it to the applicable channels, let them know, and allow the community to decide itself if and what resources it deploys for this. If you don’t have the time for that without risking yourself, trust the community will survive and be able to mitigate if it is critical. And trust that the “dynamics of people” are very competitive and resilient, and will find their way. That’s all you can do in such a situation.

If it feels too hard, you might reconsider cases like XZUtils: it shows how resilient this community is and how quick it can respond and adjust if it really is necessary. Maybe that reasoning can help a little to make it easier to make the hard decisions to protect yourself.

Concerning the survey, I think there will be always negative responses if the survey contains effectively the message “you/your-group doesn’t exist and we go ahead in decision-making with this assumption, please confirm” (I exaggerate to illustrate the point of course) for some users or user groups. The more important the underlying decision is, the more likely is that such user groups will respond negatively. Also, this situation can be very exhaustive, and that is why some maybe not respond in the beginning, especially if they have experienced before that it doesn’t help them to respond, but the more the results of such surveys is used, communicated and implemented, the more people might be affected and/or confronted by the data/results, and therefore, the more likely they might respond at some time. But the more the processes are progressed, the more hostile this response might be.

My response was intended to be direct but constructive. In the context, I feared and still fear that later responses of those currently silent can come later, being also direct but maybe also DEstructive (not necessarily only responses by comments, especially if outcomes are strategically impactful).

With this, I want to say, I agree that we should get rid of surveys at all as long as we have not the capability to do them according to scientific standards, as everything else is arbitrary in its outcome: let surveys be the task of the Descriptive Statistics SIG. And as long as we cannot maintain this SIG, let’s stick with the well-understood data at hands.

That said, you seem to look for opportunities to further focus, and in my opinion, you can add the Google Forms surveys too. The situation is the same. I agree with you that the impact of these surveys are far below the current one and unlikely to cause noteworthy issues, and the “annual ranting software developers at Flock” after the survey can be seen also with a little of humor :slight_smile: But what I read already at first glance of what this is going to be, we again remove a group from the beginning, even before we formulate questions, which is a group the survey therefore will “pretend” to not exist: the privacy-oriented users who reject to use Google services due to its privacy implications, a group mostly existing in Europe AFAIK. In the result, the data will have limited use, many might have already ceased to participate in surveys, plus this privacy group, which means we effectively will not know to “what group this data will apply to” if it is conducted as before. So I don’t
see that a survey on Google Forms rather then this LimeSurvey has advantages or does mitigate any of the issues: the impact is unlikely to be critical, it likely will not hurt, but it still can alienate user groups, and the more you use this data, the more likely it gets that users not considered by the survey (who could not submit their preference for whatever reason) will become active with negative feedback, by any means at any time. Not sure if that is the most efficient use of your time.

I hope you know me good enough to know that I write this at first with YOU in mind, rather than just to get some result I prefer :wink:

Just some thoughts. I hope they read as something constructive. You might want to make a coffee break after reading the monster post :slight_smile:

Best,
Chris, the essay writer.

I forgot. Obviously, I cannot help you much with many current tasks (flock, hotel, etc.), and see no chance that I could “save the surveys”, as elaborated in earlier posts/discussions. But if there is something I can help with, feel free to ping me. My capacity is limited, but if it helps, I can make some → maybe it can help to share / re-allocate some tasks, as far as they can be done by the non-RH-parts of the community. Just in case you think that could help.

Cancel it. To be frank, we have always been very bad at developing surveys (as most people are, so we’re not uniquely bad here). Questions are often leading, lack room for disagreement, incomplete, or otherwise hard to draw reliable conclusions from. Let’s save Fedora’s money and — more importantly — your energy for things that will help the community.

8 Likes

I’m not going to make any personal statement here, if you want to talk, you know where to reach me. About the topic, just drop it, it’s an awful tool.

I want to strengthen this by echoing it. Making surveyed is one of those simple looking things that is ridiculously hard. This one didn’t hit the mark. I’m not saying that to indict the authors as much to point out that missing the mark on a high profile survey will bring out a response.

As was said above, this is NOT a tool problem and making it a tooling or a tooling usage policy discussion is ignoring the core issue. in this case it is likely we have not collected actionable data to drive the conversation forward. That should be where the Council drives to next. If the Council can’t find interested parties to contribute to this without pushing people to burn out that is also a data point and one that may indicate the wrong problem or solution is being considered.

3 Likes

I think I’ll echo Ben and Bex here…

As a sysadmin, my initial instinct was with Michael - we could indeed self-host Lime (I have run it at home, it’s nothing unusual). But after some reflection, my worry is that we open ourselves up to a lot of bad surveys, which will then be used as “evidence” for a particular position.

Bex is right, writing good surveys is crazy-hard. After I did some training on this during my time in Ansible, I looked back on my Foreman/Satellite surveys with absolute horror. Those things did not do what I intended them to do - and I didn’t even know it, I had the best of intentions.

Data is valuable though, and I do think there’s an argument for “making it easier to collaborate on designing a survey” - but if we do that, we also need some people (more than one, not Justin!) who can approve the quality of that survey before it goes out. Those people could also deal with the longer-term survey plan (it’s often useful to ask the same questions in multiple surveys to look for drift, or to do population weighting, for example).

That’s the real blocker here - the tool can be fixed if we solve the design part.

1 Like

Disclaimer: im relatively unfamiliar with the specific situations being referenced here, so most of these thoughts are just general purpose

I think I’m also in the “Just drop limesurvey” camp and largely in agreement with the current consensus. Surveys are hard, having one person be responsible for that is almost never going to be fair (to the person or for the survey).

Wanted to leave a comment to potentially suggest that, given this thread is in the data WG, I’m curious how much the existing work of the data WG can serve as a better, more representative source of data we already have that can be used for similar kinds of analysis as a survey.

Maybe we can look at some recent surveys that have been done and see if there are ways to gather that data in a more repeatable/less one-off way. As an example: I recently got a github notification for Add support for Pretix service ¡ Issue #195 ¡ fedora-infra/webhook-to-fedora-messaging ¡ GitHub getting fixed - maybe being able to pipe pretix data into fedora message bus could be a better way to gather event attendande/check in data, automatically award badges, and generate reports about who was attending for the first time (vs repeat attendees) using this data, rather than asking questions like that on the flock survey.

I like the idea of shifting data survey design to a Descriptive Stats SIG (didn’t know we had one TBH). Maybe doing this would help clearly separate the purpose of surveys between “are we gathering numerical data” compared to a more simple, less statistical use of a survey/google form etc as largely just a unified suggestion box for freeform content/feedback (i.e. for Flock feedback).

None of these suggestions should be followed at the expense of anyone’s personal health or family time though.