Fedora-Council/tickets ticket #487: Budget request: Price increase for hosted Matrix (chat.fp.o) by Element Matrix Services (EMS)

@jflory7 filed Fedora-Council/tickets ticket #487. Discuss here and record votes and decisions in the ticket.

Ticket text:

1 Like

I think it makes sense for us to use hosted open source services where we can. However, when companies take this “per seat” pricing model, it’s really hard on us — and obviously contrary to our goals for community health and growth. I don’t feel like it’s sustainable.

I don’t think that we should stop offering fedora.im accounts. I think it’s very valuable to have consistent Fedora identities across all of our platforms (and this all clearly connects to Objective Review: Modernize our communications tooling).

It may be that, because of the pricing model, this is something we need to look at self-hosting. But, I don’t think we can do that quickly.

So that we’re not excluding people, I’m in favor of authorizing the increased price, but let’s look around for options.

Anyone want to work on Matrix protocol for chat - #47 by sam - feature - Discourse Meta ? :classic_smiley:

@mattdm and I had an extensive chat about this in #council:fedoraproject.org. I will try and summarize my own opinion. First though, I want to frame the perspective on which I am looking at this topic:

  1. I value Fedora having independent, autonomous communication platforms, and doing that powered by Free Software.
  2. I value spending money to support the development of Free Software and communities/projects that sustain the development of Free Software.
  3. In a world of so many competing priorities, I value protecting Fedora’s budget to be spent in ways that advance the Project, our community, and now the forthcoming Strategy.

How deep do we go?

So, we are hitting a point where we have to ask how far do we want to go down this pathway of per-user seating. Today it is $3k. Next year, or the year after? I don’t know.

We are always going to be pinned to a per-user cost and that cost may vary and also change with other economic factors, like inflation. The cost of our Matrix hosting is growing by the nature of the business model, which I see as broken for Fedora. I do not see this as a budget cut for Fedora, but rather, we are investing more and more into a business model which penalizes success for Fedora and does not actually unlock more value for us as a project.

On self-hosting

I believe we can host Matrix in some kind of Fedora-managed infrastructure without incurring greater human cost. And honestly, I feel like the equation of cost is different for Fedora and even Red Hat than it might be for a small-or-medium enterprise (SME) non-engineering company.

In Fedora and at Red Hat, we work with people who are exceptionally smart. We work with people who use best-in-class automation tools like Ansible and focus on reproducible infrastructure (e.g. containerized apps and k8s/OpenShift deployments). The human cost in maintaining today is not the same as it was when Fedora first began, or even from 10 years ago.

My concern is that I don’t want to rule out self-hosting. We believe strongly in it for our git forges. We can also believe in it strongly for our other tools. I feel like our communication platforms are possibly even more important to own and manage ourselves, so we don’t have situations like this where pricing terms lock out our community.

On hosting our own accounts

Lately, I am thinking more about the value of us hosting accounts for people though. Other than the vanity aspect of a :fedora.im username, what other motive/value/incentive is there for a Fedora-hosted Matrix account?

SSO integration is nice but I don’t think it is actually adding much value either… especially with the impossibly snail’s pace it is taking for sliding sync to be enabled on Matrix homeservers with SSO enabled. Sometimes it feels like we might never get that.

I am open to hearing counter-points or arguments in why we should host our own accounts. It may take some convincing for me though. I don’t think the activity metrics consideration carries much weight though (i.e. all users having the same username as FAS). Since FAS lets a user specify their own Matrix ID, we can instead encourage contributors to add their Matrix ID in their FAS profile, and still analyze any meaningful activity data that way.

For me, that’s one part, but the bigger one is coherence — although of course there’s no guarantee that the account isn’t compromised[1], I can reasonably assume that @jflory7:fedora.im is you, without having to do an extra verification.

Additionally, when someone logs into chat.fedoraproject.org, they are added to the Fedora Space, the various subspaces under it, and (if I remember right) some default rooms. I’d love if Matrix offered a Discord-like “server” approach, or even copied Slack here, but they don’t, so this is as close as we’re going to get to some kind of approximation of a connected hierarchy of channels.

Also, when someone logs in there, we have the opportunity to remind them of our code of conduct and expectations. We also have the ability to, if need be, remove someone’s privileges of acting under a fedora.im identity.


  1. related — see the 2FA discussion in FESCo! ↩︎

1 Like

I would like to understand this better. How would you describe the benefit of having the same username across all of our platforms? From this, I might assume you see value in this making collaboration easier. How does it make collaboration easier? How does it save time? Could you give an example of this happening?

I agree that any self-hosting is unlikely to come quickly.

Before signing on the dotted line, I want to be more assured in what happens come August. As a reminder, this $3k only covers us until August. In the original ticket, I wrote that the renewal cost would be $24,000. This may not be correct actually, and I have asked our sales rep for a clarification on the new final amount. I think it could be even more than this amount. I need to dig in on exactly how we would be billed in 2025.

If we are looking around, then I want to make a commitment to a future path before I sign on for a bigger bill. If I have to renew in August with a bigger amount, then I have to borrow money from somewhere else in Fedora to pay for the increased spend on infrastructure.

This cost has also been notoriously difficult for me to predict accurately since I joined Red Hat :anguished:

I guess I probably also should say that, although it does not have Matrix support, Discourse does have chat (a feature that we are effectively paying for but just haven’t enabled).

This approach would have the significant advantage of making one central place the only one you need to go to and check to keep up with Fedora communications, both synchronous and async.

I’m not advocating for this option[1], but if neither paying for ever-increasing Matrix-hosted use nor self-hosting Matrix is acceptable … there’s another option right here.


  1. at least not with how things stand now :classic_smiley: ↩︎

I’m okay with that. We can use the time to decide what we want to do next.

I definitely don’t want to commit to anything, but I do think it might be possible for us to use the ansible deployment role and deploy/run our own server if we need to.
There would definitely be some maint moving forward, as parts of that deployment change pretty often (so, probibly weekly updates or the like). But if thats just updating a ref and re-running the playbook thats not too hard.

I’m with mattdm that we should for right now just pay, but definitely we should consider our options. I don’t think we really have until Aug tho, if we want to move we should do it well before then so we don’t have to renew. But if we can get a bit of time here to get f40 out the door and the rhel7 migration work much of CPE is working on done, we should have more cycles to look at this.

Just my 2cents.

2 Likes

Second money problem I’m reading today. Is Discourse the next one?

Following recent chat discussions and community feedback, this is my proposal to the Council:

  1. Move to a self-hosted Matrix homeserver by August 2024.
  2. Stop offering hosted :fedora.im Matrix accounts, effective immediately.
  3. Create better documentation for using other Matrix homeservers for new accounts.
  4. Cancel the EMS contract after August 2024, freeing up $24k of the Fedora community budget for more impactful things like DevConf CZ, Flock 2024, GNOME Asia 2024, sponsoring our upstream communities, and possibly sending Fedora contributors to new events for the first time since pre-pandemic.
  5. Determine a reasonable amount for an annual donation to the Matrix Foundation, to continue supporting the development of Free Software that Fedora relies on.

This is the pathway that I am most comfortable moving forward with, although this creates a nontrivial amount of work to figure out migration.

Could this be mitigated by someone putting their Matrix ID into their FAS profile?

Could this be mitigated by creating documentation about how to join the Fedora Space?

I think it is a fair assumption to assume the Fedora Code of Conduct and our community rules apply in any Matrix room tied to :fedora.im or :fedoraproject.org. Also, we could simply avoid the moderation question of :fedora.im accounts by not offering them.

In a hypothetical situation, someone uses a :fedora.im account to join other non-Fedora rooms and be a troll or post obscene things, thus abusing our community reputation while also dodging our Code of Conduct by not being obscene in official Fedora channels. One could suggest that using a :fedora.im account implies following our community standards, but then that also implies someone is there to moderate and be a judge on behavior that happens outside of our community. We have not yet had to get into that sort of business, but I’d rather not.

I know you have run a homeserver for quite some time. Do you see a significantly different resource burden by offering hosted accounts with SSO integration, versus only hosting the existing ~500 accounts we already have?

I think it is reasonable to bump up until August… but if we will not continue offering hosted accounts, should we keep promoting them as an option?

If we intend to move to self-hosted, we also need a migration plan in place sometime in May/June to reasonably execute in July. I am not sure how much work is involved in migrating homeservers though. I don’t know how realistic my timeline is, or if I am significantly over/under estimating the work required.

This is fine. Nothing significant happens until August anyways. Come August, then I have to lock into one more year.

I’m not sure I understand this comment or what you intended to ask. Could you ask your question another way?

Following recent chat discussions and community feedback, this is my proposal to the Council:

  1. Move to a self-hosted Matrix homeserver by August 2024.
  2. Stop offering hosted :fedora.im Matrix accounts, effective immediately.

That means, no new accounts? Or ‘server is going away today, bye!’ ?

I would think we would need to let people have some time to find
alternatives, update their account info, etc.

  1. Create better documentation for using other Matrix homeservers for new accounts.
  2. Cancel the EMS contract after August 2024, freeing up $24k of the Fedora community budget for more impactful things like DevConf CZ, Flock 2024, GNOME Asia 2024, sponsoring our upstream communities, and possibly sending Fedora contributors to new events for the first time since pre-pandemic.
  3. Determine a reasonable amount for an annual donation to the Matrix Foundation, to continue supporting the development of Free Software that Fedora relies on.

I think we could join the matrix foundation at a lower cost than we are
paying ems right now.

This is the pathway that I am most comfortable moving forward with, although this creates a nontrivial amount of work to figure out migration.

Could this be mitigated by someone putting their Matrix ID into their FAS profile?

Yeah, I don’t think the verification argument is good compelling…

Could this be mitigated by creating documentation about how to join the Fedora Space?

It could, but it wouldn’t be as easy.

Also, onboarding in general would be a lot more complex.

I think it is a fair assumption to assume the Fedora Code of Conduct and our community rules apply in any Matrix room tied to :fedora.im or :fedoraproject.org. Also, we could simply avoid the moderation question of :fedora.im accounts by not offering them.

In a hypothetical situation, someone uses a :fedora.im account to join other non-Fedora rooms and be a troll or post obscene things, thus abusing our community reputation while also dodging our Code of Conduct by not being obscene in official Fedora channels. One could suggest that using a :fedora.im account implies following our community standards, but then that also implies someone is there to moderate and be a judge on behavior that happens outside of our community. We have not yet had to get into that sort of business, but I’d rather not.

Yeah, that is a point in favor of not offering them.

I know you have run a homeserver for quite some time. Do you see a significantly different resource burden by offering hosted accounts with SSO integration, versus only hosting the existing ~500 accounts we already have?

I… only have one account. :wink:

I don’t think the resources would be too bad for a bunch of people, many
of whom only join a few fedora rooms or never use their account. There
would of course be some heavy users, but mostly thats database disk
space, which isn’t too difficult.

I think it is reasonable to bump up until August… but if we will not continue offering hosted accounts, should we keep promoting them as an option?

If we are going to drop them, then yes, we should communicate that.
We should be sure we are dropping them first tho. :wink:

If we intend to move to self-hosted, we also need a migration plan in place sometime in May/June to reasonably execute in July. I am not sure how much work is involved in migrating homeservers though. I don’t know how realistic my timeline is, or if I am significantly over/under estimating the work required.

yeah, will need some investigation, but I think the ansible role I
mentioned can take exports from ems instances. ie, it just imports the
db and such, so then you just need to take down the old server and bring
up the new and users login again.

But I could be missing some issues for sure, and we would want to
openshift-ize it and make sure we can do easy upgrades, etc.

Also, us self hosting opens a bunch of possible options we could add
on… like, we could look at running our own irc/matrix bridge?
Moderation bots, etc.

This is fine. Nothing significant happens until August anyways. Come August, then I have to lock into one more year.

yeah, but I think we would need to get pretty far on this in may/june as
you note in order to hit a ‘before aug’ deadline. It’s tight, but might
be doable.

1 Like

The problem we’re facing is per-user cost for :fedora.im accounts. Either one of these seems sufficient to solve that problem alone. If there are other reasons you don’t want to offer those, let’s treat that separately.

A third option might be to restrict :fedora.im names to contributors with a certain level of activity in the project, so they’re not just used as random “vanity plates”. That could keep the costs from potential insane spiraling, but still goes against the goal of growing our contributor base.

If we can do self-hosted reasonably, I’m for that — but keeping :fedora.im names for contributors.

Could this be mitigated by someone putting their Matrix ID into their FAS profile?

I don’t think sufficiently, unless we made it mandatory in some way.

And even then, I’d still have to do an extra lookup, not possible from a normal Matrix client.

And it’s not just that — consider integrations, like with Discourse or Wordpress. Shared usernames make that a lot nicer in so many ways[1]. Particularly if we’re connecting software that we’re not maintaining as a separate codebase, we might not even have the opportunity to insert some kind of FAS lookup as the cookie bot does.

Further, I think a shared namespace is important for conceptual coherence. If we want to demote Matrix to a non-official channel, that’s be different — but I don’t think we do.


  1. or would be if Filling in discourse_username when not using DiscourseConnect - wordpress - Discourse Meta were resolved ↩︎

1 Like

At least one market segment alternative offering to Element reduces the fees if a user is not active (for some period of time), reducing the costs for an organization should there be many inactive users. Does Element offer anything like that (or have plans to offer something like that) for organizations?

@jflory7 can correct me if I’m wrong on this, but the pricing is for active users per month, so they’re already doing that.

For what it’s worth, the scenario I’m more concerned with is something like: a whole ton of 1930s fashion enthusiasts show up and decide that this is the perfect place for their interests.

1 Like

IMO there is value in providing clear Fedora-affiliated identities via :fedora.im, in the same way that we provide email ones via the @fedoraproject.org aliases. It’s not uncommon for Fedora contributors to engage with other projects (be it the upstream for a package, another distro to coordinate some ecosystem work, etc.). Engaging from a clearly Fedora-affiliated identity makes it clear what hat the person is wearing at the time. As someone that wears many different hats, I’ve found this valuable in the past to reduce confusion and help start conversations off the right foot.

1 Like

Catching up here and from #matrix-meta-chat:fedoraproject.org.

No new accounts is what I meant. I would not want to close out peoples’ accounts. Doing away with only new hosted accounts is already going to create work for people and Fedora teams, and I don’t want to create more work for people who already rely on a Fedora-hosted Matrix account for basic project communication.

For sure.

This has come up a few times already, but my mind is beginning to turn a bit. Someone shared in chat earlier today about the amount of time they spend to onboard newcomers to Matrix and that a hosted Fedora account makes that onboarding for new contributors easier. That is a fair point. You (@kevin) also shared this in chat too, which I thought was a good explanation:

One item I thought of… it makes onboarding a bit harder. ie, now: join us on matrix! go to chat.fedoraproject.org and login with your fedora account vs join us on matrix! Go to List of Matrix servers and pick a server and install a client or go to a web page if the server you selected has one, create an account there, login there

I guess one group who has a strong case to benefit from would be newcomers who are not used to federated chat tools.

I forgot I had access to this data, but I pulled data from our top 8 users with the most data on :fedora.im today. This includes all 2,386 users currently registered since we first started offering Matrix accounts in 2021.

The user with the most data on their Matrix account is approximately 4.5 GB, and the eight user with the most data is 848 MB. Overall… this isn’t too bad, given our homeserver has been online since August 2021.

This is a good point, but as I understand it, this is about self-hosting the server in general and not providing hosted accounts. Is that right?

Admittedly, if we continue offering hosted Matrix accounts, I am gradually being convinced that one of the users who have the most to benefit from this is newcomers who have not used Matrix or federated chat tools before.

But let’s play this out… would FPCA+1 be the limit that we want before someone could register an account? Or is there some other proxy metric we could use?

This is accurate.

This is a helpful use case. Could you either a specific or generalized example of how this was helpful to you before? It would be useful to hear more about this.

1 Like

A couple of examples that come to mind (and apply just the same to email and Matrix):

  • if I’m making a case for an upstream change that comes from / directly impacts Fedora (e.g. some PR to improve a package build), using a Fedora identity makes it clear where I’m coming from (and, for a subset of projects, could help get noticed and/or grease the wheels a bit)
  • sometimes I contribute to things that could be work related/work adjacent, but are actually not (and I just happen to personally care); using a Fedora identity here helps avoid confusion (both on the project side and on the work side)

For new contributors (and not only for new contributors), identities can also often be a point of pride. In my experience, some people like being able to publicly wear a hat that signals them as being part of their community. They’d likely be sad if we took that away.

Doesn’t Zodbot depend on the Matrix account matching the FAS username (cookies et al.)?