RHEL7 hosts left

Project Discussion
1

Greetings infrastructure folks. :slight_smile:

I thought I would go over what RHEL7 hosts we have left and some information about them. There’s some we need to discuss a bit or decide what to do with.

RHEL7 is going to move to extended lifecycle support on June 30, 2024. At this point EPEL7 ends and we do not want to be running any RHEL7 hosts anymore.

This is long, but I wanted to go over the entire list.

Here’s the list and discussion:

badges-backend01.iad2.fedoraproject.org
badges-backend01.stg.iad2.fedoraproject.org
badges-web01.iad2.fedoraproject.org
badges-web01.stg.iad2.fedoraproject.org

Badges is being re-written, but this means there’s a hard deadline to have the new version rolled out before RHEL7 goes EOL.

busgateway01.iad2.fedoraproject.org
busgateway01.stg.iad2.fedoraproject.org

We need fedmsg on rhel9 to move these to rhel9. We could just move them to rhel8 anytime I think. These are needed for fedmsg to be able to work externally. We could also just say that we are going to retire them before then. Perhaps we could figure out a way to see how many consumers/producers are still using this?

db-fas01.iad2.fedoraproject.org

Can do this anytime with an outage. Right now the only thing still using it is ipsilon.
We could just move it to rhel9 or… we could just move this database to db01 and retire this instance.

dedicatedsolutions01.fedoraproject.org

Can do this most anytime. There’s no console though, so it will be fun. :wink:

fedimg01.iad2.fedoraproject.org
github2fedmsg01.iad2.fedoraproject.org
github2fedmsg01.stg.iad2.fedoraproject.org

These are planned to be replaced.

ibiblio01.fedoraproject.org
ibiblio05.fedoraproject.org

Can do these with an outage anytime.

kerneltest01.iad2.fedoraproject.org

This is planned to move to openshift, but we need to do it before RHEL7 eol.

mailman01.iad2.fedoraproject.org

Planned to move to rhel9 once mailman3 stack lands.

mbs-backend01.iad2.fedoraproject.org
mbs-backend01.stg.iad2.fedoraproject.org
mbs-frontend01.iad2.fedoraproject.org
mbs-frontend01.stg.iad2.fedoraproject.org

I guess we need to move these to rhel8. Everything should be place, thanks to @dherrera
We can retire them after f38 goes eol.
https://pagure.io/fedora-infrastructure/issue/11042

mm-backend01.iad2.fedoraproject.org
mm-backend01.stg.iad2.fedoraproject.org
mm-crawler01.iad2.fedoraproject.org
mm-crawler01.stg.iad2.fedoraproject.org
mm-crawler02.iad2.fedoraproject.org
mm-frontend01.iad2.fedoraproject.org
mm-frontend01.stg.iad2.fedoraproject.org
mm-frontend-checkin01.iad2.fedoraproject.org

@abompard has been working on a python3 re-write here. Ideally we would move them to rhel9.

noc02.fedoraproject.org

I can do this one anytime outside freeze.

nuancier01.iad2.fedoraproject.org
nuancier02.iad2.fedoraproject.org

These should be gone soon. @ryanlerch was going to finish retiring them.
https://pagure.io/fedora-infrastructure/issue/11371

osbs-control01.iad2.fedoraproject.org
osbs-control01.stg.iad2.fedoraproject.org
osbs-master01.iad2.fedoraproject.org
osbs-master01.stg.iad2.fedoraproject.org
osbs-node01.iad2.fedoraproject.org
osbs-node01.stg.iad2.fedoraproject.org
osbs-node02.iad2.fedoraproject.org
osbs-node02.stg.iad2.fedoraproject.org

I am really hoping we can retire these after f39 release.
flatpaks are no longer using them. Our base containers are done with ImageBuilder. toolbox container is also now done with ImageBuilder (in f39+). The only thing I am not sure of is if we need to do able to build toolbox for f37/f38. I’ve asked maintainers. We still may be able to move them to ImageBuilder nightly anyhow. It will be great to retire these (and also the aarch64 osbs clusters).

osuosl01.fedoraproject.org

This one has a bad drive and we are going to replace it next year. I am hoping we can limp along until then and then retire the hardware completely.

pdc-web01.iad2.fedoraproject.org
pdc-web01.stg.iad2.fedoraproject.org
pdc-web02.iad2.fedoraproject.org

Hopefully we will retire these this fall. finally.

people02.fedoraproject.org

I can plan on doing this one sometime soon. Will require a small outage.

sundries01.iad2.fedoraproject.org
sundries01.stg.iad2.fedoraproject.org
sundries02.iad2.fedoraproject.org

There are still some things these do. Ideally we would remove any need for them and just retire them, but if not, it should be reasonably possible to move them to rhel9.
Will require some work and investigation tho.

torrent02.fedoraproject.org

I think this one should be possible to move to rhel9. Will require some work probibly to get opentracker built for epel9, but that should be doable.

value01.iad2.fedoraproject.org
value01.stg.iad2.fedoraproject.org

These have just one thing left on them: mote.
So, we need a plan for mote. Moving it to rhel9 and openshift?
@t0xic0der I think did the current version?

virthost-cc-rdu01.fedoraproject.org
virthost-cc-rdu02.fedoraproject.org
vmhost-x86-08.iad2.fedoraproject.org
vmhost-x86-09.iad2.fedoraproject.org

All these I can do outside of freeze.

Thanks to anyone who read all the way to the end. Have a cute little :penguin: as a reward. :slight_smile:

1 Like
2

Do we have plan to replace Ipsilon with something else? I think we were talking about keycloak in the past.

3

Well, we talked about that, but never decided to do it.
keycloak is actively maintained which is good. It could run in our
openshift clusters, which is good for deploying it.

On the downside it does not support openid, so we would need to make
sure and have pagure.io moved all the way to oidc before we could say
even our apps were not using openid anymore. External applications could
well be still using openid, but perhaps we don’t care so much about
that.

ipsilon is overall great I think, but it’s not under very active
development/maint and it’s mostly on our team to do anything with it.

So, I would say, we can have the conversation about moving to keycloak
after pagure.io is on oidc, but until then we are stuck with ipsilon I
think.

4

Is all this work split in tickets? I think some of them are a good fit to the easy_fix tag, and since I want to start working on infra, I would love to work on some of these

5

I suspect it is the https://meetbot-raw.fedoraproject.org/ service that is running off them as the revamped Mote is now an OpenShift application.

Correct me if I am mistaken @darknao we still pull our meeting logs and summaries from there, a shared storage or a service that synchronizes these assets periodically?

Depending on that, we can come up with a contingency plan for the move (that is, of course, if we need to).

6

Right, meetbot is now on Openshift and pulls logs directly from the NetApp filer.

Value01/02 still hosts meetbot-raw and the associated supybot instance.
I think the bot is only active on 02 and the meetbot-raw part on value01. That can be moved to Openshift easily (this is basically just an apache instance).

We have a couple of cron scripts running on 01 tied to supybot/meetbot. I don’t know if they are still useful, but they can be moved to value02 or Openshift If needed.

7

Well, they all fall under the ‘migrate to rhel9’ ticket (
Issue #10632: rhel9 adoption - fedora-infrastructure - Pagure.io )

But I don’t think any of these are particularly easy_fixes.

In order to install a new instance and migrate all it’s data you need to
be root on the virthost that runs it, which basically means you have
full access to everything on that host and all it’s guests.
This isn’t something that many people have access for, so I think it’s
not really a good easy fix thing.

That said, there is probibly some adjusting of playbooks to make sure
everything works on the new version which most anyone could do, but of
course it would need to be closely coordinated with the actual migration
of the instances.

1 Like
8

Right, meetbot is now on Openshift and pulls logs directly from the NetApp filer.

Ah right, I forgot we moved it.

Value01/02 still hosts meetbot-raw and the associated supybot instance.

I think the bot is only active on 02 and the meetbot-raw part on value01. That can be moved to Openshift easily (this is basically just an apache instance).

yep, zodbot is only on 02 (the rhel8/newer one).

meetbot-raw is still going to value01, but I think we could switch it
anytime to 02 and it would work fine there.

We could move zodbot to and meetbot-raw to openshift, but I’m not sure
zodbot is going to be around longer term if we move to a matrix bot.

We have a couple of cron scripts running on 01 tied to supybot/meetbot. I don’t know if they are still useful, but they can be moved to value02 or Openshift If needed.

Yeah.

There’s one more thing running on value01: fedmsg-irc.
This is the thing that sends fedmsgs to #fedora-fedmsg irc channel and
also sends notice of some events to #fedora-releng (composes, releng
tickets, etc). I think we decided that it’s ok to kill it, but it would
be nice if we had a matrix bot to replace the notification part, but I’m
sure we could live without it. :wink:

So, I guess after freeze:

  • point meetbot-raw to value02
  • make sure value02 is running any cron/scripts needed
  • take down value01/value01.stg
9

Aah, deadlines! [1]

I think the end of June 2024 deadline should be feasible. I make sure to
put this on the agenda of the next round table meeting.

Edit: Added quote for context. Not sure why it was stripped from my e-mail reply.

  1. Can you hear the swooshing noise already?
    :stuck_out_tongue_winking_eye: ↩︎