Getting Fedora Badges Back in Shape

I’m glad to announce that after much discussion the first steps have been taken to get Fedora Badges back in shape again.

Based on all the feedback received so far, Fedora Badges is still much liked throughout the community. Therefore, getting it back in shape should be a reward in itself.

Next Steps

There are many ideas floating around on how to go forward with Fedora Badges (see discussion). None have materialized further than discussion in various channels. I would therefore like to seize the opportunity and outline the next steps and where we think the ideas brought forward would fit in.

Short Term

The short term goal is to get on top of the current state of affairs. Along the way roadblocks will undoubtedly be encountered. So, items may move as appropriate.

  • Sort out tickets in Pagure (currently 145 open issues)
  • Update documentation (outdated references, etc.)
  • Attempt to get the backend in a reliable working state
    • Document issues and possible solutions
    • Apply hotfixes as needed

I’m, personally, in favour of using Pagure as the issue and task tracker and distribute work from there by way of GitHub issues in the appropriate component (tahrir, tahrir-api, fedbadges, datanommer).

:bulb: Automate this, so that assigning a label in Pagure will automagically create an issue on GitHub for the correspondig project.

Medium Term

This is were fundamental decisions need to be made regarding frontend, backend refactoring or rewrite and such. Depending on the amount of work involved there may be intermediate solutions to persue and larger projects moved to the long term.

  • Decide on the frontend
    • Tahrir is written in Python2, so needs an overhaul, if we decide to keep it
    • Use Discourse as the frontend tied to this forum as suggested and followed up by @mattdm
    • Use badgr-server as a replacement for tahrir (upstream project no longer exists, though)
  • Shape up the backend

There’s a helpful document in the fedora-infra repo.

Long Term

What goes here pretty much depends on what we encounter and come up with in the long and medium term. So, we leave it open for now.

Contributing

As I have experienced myself, it currently takes quite some effort to onboard the fedora-badges. I sincerely hope that will change once documentation has been updated and word about the current effort is spreading.

Nonetheless, help is welcome in all areas mentioned above and maybe others as well. Once the fedora-badges project in Pagure has been brought into shape, issues and tasks will appear there and everyone is welcome to contribute.

Since Fedora Badges is made up out of different components, some of which have already been deemed to be replaced/refactored. Work in different areas will happen in parallel (e.g. short term fixing of the environment and mid/long term refactoring of fedbadges).

I will be focusing on the short term items in the coming weeks, while @erolkeskin has already started work on the backend.

Communication

Updates regarding progress will be posted regularly in this thread. Make sure to watch this topic and any updates will be delivered to your inbox.

Or, if you really want to stay on top of everything regarding Fedora Badges, watch the all new #badges-team tag.

So long and thanks for all the fish badges! :badger:

6 Likes

Hey, I’m interested to contribute. I’ve contributed to Fedora Badges in the past but had been busy for a while due to my university schedule. I’d love to get involved again and help ramp up the much-loved Fedora Badges project! :sparkling_heart:

Let me know how I can be of assistance.

3 Likes

Welcome back @BeeBombshell!

What would you be interested in? We, very roughly, have three areas of activity: design, development and sysadmin. See Join Fedora Badges community :: Fedora Docs for a little more information on that.

Please keep in mind that the documentation itself is also subject to review and still outdated / work in progress in some places. Let me know if anything is unclear.

1 Like

Hi @gui1ty, I would love to contribute to Fedora Design and Documentation. Do let me know if there are any active topics to contribute to! :blush:

Now that Fedora Docs are on GitLab, did anybody evaluate if design features there are more convenient for badges?

Thanks for the pointer. It would certainly help us with PNGs and SVGs:

$ du -hsx fedora-badges/{pngs,svgs}
25M    fedora-badges/pngs
170M   fedora-badges/svgs

Let me ask our head designer, what she has to say about it. @riecatnor :microphone:

Talking about repo size, the biggest chunk in our repo is currently taken up by STL files as pointed out by @misc:

$ du -hsx fedora-badges/stls
1.1G	fedora-badges/stls

So, we definitely want to move these out of the repo short term.

@BeeBombshell hi there :smile: happy to have you contribute to badges design! Take a look on the pagure repo for “AW: needed” tagged tickets, and pick out something you’d like to work on (some of those also have “AW: approved”, avoid those ones). Make sure to read through the whole comment history to get any background info or see if other designers have been working as you might have a nice place to start. If you have any questions please send them my way on Element/Matrix, I am on there as @riecatnor. Once you have some art created, please add it to the ticket and then tag me in the badges chat channel with the ticket link asking for a review. Thank you!

@gui1ty I think a move to Gitlab is a good idea- my only concern is being able to move open tickets over to a Gitlab repo in an efficient way. I wonder if there is an easy way to do this? Doing it by hand seems time consuming, but if we have to do it that way, maybe we could organize a sprint where a couple of us work on it together?

There are a ton of closed tickets in the Pagure repo with some important context so I would want to label it as deprecated but preserve it so we can look back on closed tickets for background as needed.

Or use the artwork board, Board Artwork - fedora-badges - Pagure.io, makes it more accessible, I hope.

I’ll look into it. Regarding open tickets, that’s a valid concern. More so, because the ticket leading to a new badge, is referenced in the badge itself, so users can check the rules of the badge. Regarding currently open tickets, I wouldn’t mind starting fresh and only carry over what’s not stale.

But moving to GitLab almost implies, that the projects now on GitHub (tharir, fedbadges, etc.) also move to GitLab for easier (cross)reference. That would be something to consult infrastructure about.

I see one advantage of having everything on GitLab: one place for issue tracking and ticket distribution. Something to think about and discuss further.

Thank you so much for taking the initiative, @gui1ty and @erolkeskin!

Please feel free to reach out to the Websites and Apps Team at https://chat.fedoraproject.org/#/room/#websites:fedoraproject.org, whenever there is any work that needs to be done on the application’s frontend and backend. We would be very glad to be of assistance.

1 Like

Hi @riecatnor,

Assuming that access is available in the said repositories, I think it is possible to import the entire repository (along with the pull requests and issues) from GitHub to GitLab. If it is on Pagure (or any other Git forge for that matter), we would have to resort to the use of automation scripts.

While the former is the best possible solution, the latter works with some catches. It is quite likely that some information (like comments under the issues) would be lost and pull requests cannot be imported that way. Not a very elegant way of doing things in my opinion but can be a last resort.

It’s been two weeks since the topic started. Time for an update…

Bad news

Badges is still in bad shape. I spent some time, probably way too much, trying to improve the current state of affairs. Unfortunately, I wasn’t able to. Processing gets stuck at least once a day with the message queue overflowing and occasional crashes thrown in for good measure. This usually results in messages having to be dropped, meaning users miss out on badges they qualify for. :frowning_face:

There was almost no response to my mail sent out on the mailing list and to sysadmin-badges-members. I assume most of them have moved on or are busy with other things.

Good news

I made a start with getting the open issues categorized, setup a board for artwork and badge requests and added tickets to them according to current status. I hope this makes it easier for the groups involved in badges to organize tickets and manage their workflow. Feedback is welcome.

A large part of my available time for Badges in the past two weeks was spent on getting acquainted and getting access to the backend of the frontend. Yes, the frontend has its own backend! :face_with_raised_eyebrow:

Plans

In the coming weeks I intend to go through open tickets, update, categorize and close as applicable. There are two main categories it seems:

  1. Badges not awarded
  2. Badge ideas/requests

I will also contact current members of fedora-badges again and clean up the group in due time to get a better picture of who’s still active and who’s not.

@erolkeskin is working on a PoC for a new/rewritten badges backend using fedora-messaging. I’m looking forward to the proposal and hope help will arrive implementing it. Looking at the current state of Badges, the urgency for replacing fedmsg-hub is quite high. It might well be the only piece of software in Fedora Apps that still relies on Python 2. :snake:

Thanks for doing this! I hope it is not too discouraging. Is the queue overflow a hard limit or a soft limit? Can we mitigate the problem by throwing more resources at it, like moving it to a system with a ridiculous amount of RAM?

Since the active discussion is here, I suggest we shut down that mailing list[1]. Our comms are fragmented, but at least we can make it clear where to go.

Microservices!


  1. send out one final message and then request infra to icebox it ↩︎

There’s no real limit. I used it as a figure of speech. Nagios monitoring croaks alarm (problem) when the queue reaches 25.000 and again (critical) when it reaches 35.000.

Throwing more resources at it, won’t solve it, I’m afraid. The VM it’s running on hardly breaks a sweat. It’s something in the code (fedmsg vs. fedora-messaging) or simply the fact that there are too many messages entailing a lot of message traversal in order to determine if a badge should be awarded.

Quite often when I looked at what was going on[1] while the messages were stockpiling, I saw lots of entries for Building the Outer Ring (Copr Build I - VII)[2] on messages which appeared to be coming from build automation. Two examples:

distrobuildsync-eln/jenkins-continuous-infra.apps.ci.centos.org
https://api.github.com/users/openshift (rhcontainerbot)

These are running on a schedule and will build a large collection of packages for multiple architectures and chroots. My guess is, they won’t ever lead to a badge, unless the package owner is considered in the check. But even if, for every message it retrieves the entire history in order to assert if any of the stages (I - VII) have been reached.

Sisyphus

Since fedmsg-hub keeps track of which message it was processing last, upon crash and restart by systemd, the entire queue will be reloaded. The reload process appears to become slower the larger the queue[3] and can take upwards of 10 minutes to complete on a queue of 40.000 messages.

Yeah. Someone else already suggested that and I think it makes sense. I will send out a notification to the list[4] and file a ticket with infra.


  1. The logging on the machine is rather limited and doesn’t provide a complete picture. ↩︎

  2. I’m pondering turning of the Outer Ring badges for a day to see if it makes a difference. ↩︎

  3. Just from observation. Nothing to substantiate my claim. ↩︎

  4. Let’s see if I get a reaction when threatening to take their toy away. :face_with_raised_eyebrow: ↩︎

1 Like

I bit the bullet and disabled it over the weekend with underwhelming results. The queue still runs up while fedmsg-hub logs plenty of messages regarding Copr Build. No idea where this is coming from. There’s no badge rule present on the system anymore mentioning Copr Build. :face_with_raised_eyebrow:

2 Likes

Well, that sounds frustrating! :frowning: Something cached, or hard-coded (!) somewhere?

Hi All :smiley:

I initially set up a separate ticket but it makes more sense to join in on the conversation here. Thank you @gui1ty for guiding me to this ticket.

For anyone who doesn’t know me, I’m Ellen. I will be working as the Product Owner for the Badges project. I see that there has been some great discussion about initial requirements of Badges.
I think it would be a great idea for us all to sit down together on a call, get to know each other, and go through the initial queries.
It could be a great time to get either Pagure or Git hub boards and issues up to date also.

Please let me know what platform and time of week suits you all best and we can get things started.

Looking forward to chatting with you all!!

1 Like

I have no preference for a particular platform other than it must not require any separate sign up or such. And it should be working with/on Fedora. But that goes without saying. :wink:

:timer_clock: DST ends on Sunday in Europe times below are for the standard TZ offset, back in effect from Sunday.

On weekdays, I’m generally available after 20:00 UTC up until 23:00 UTC. I can add an hour or two at the end if need be. During office hours (08:00 - 16:00 UTC) I usually can chip in a meeting. Mornings are less favorable. After 10:00 UTC chances for me being available increase.

I’m on vacation for a week starting 7 November.

Hi @eocarrol thanks for jumping into the discussion. I would happy to get on a call and provide what input I can on requirements for Badges.

My best availability is generally 12PM-9PM UTC, give or take an hour on either end. Tuesdays are a bit packed for me already, Friday I try to keep meeting free, so I’d say Mon, Weds, & Thurs are best days of the week for me. Side note: In my locale, the time zones will be changing this upcoming weekend, on November 6th. That would push my availability to 1PM-10PM UTC if I am looking at it right :smile:

As for meeting platform, happy to use whatever is most convenient and everyone is on board with. I would suggest a public hackmd for meeting notes.

It sure is. Turns out fedmsg-hub/fedbadges[1] are very admin unfriendly beasts.

That was true from my POV, but a complete lie from the backend’s POV. I followed best practices and renamed the rules files by appending .disabled to them. It turned out they were read anyway. All the rule files carry the extension .yml. Apparently, that’s only for the admin to enjoy knowing what kind of file it is. The backend will consume whatever is in the rules directory, no matter the name of the file. Try putting a README in there. The backend would consume it and be polite enough telling you it can’t digest it for lack of being in yaml syntax and then refuse to pick up any work. Adding insult to injury. :face_with_thermometer:


  1. Let’s just call 'em backend. They are separate beasts, but one can’t live without the other. ↩︎

1 Like

Glad you were able to track it down!