A future for tiny-stage

Project Discussion
1

Hey folks!

To develop on most of our infrastructure applications, we use a tool called tiny-stage which is based on Vagrant, to run a few virtual machines with the authentication stack and a few common tools. The apps themselves all have a Vagrant VM for development that is setup to call the Tinystage VMs for authentication and the rest. This setup has worked well for us but it has shortcomings:

  1. the VMs take up quite a bit of RAM. We try to colocalize the apps on some VMs but we usulally need at least two, IPA and the auth stack.
  2. Vagrant is not Free software anymore, and it doesn’t look like any of the forks have gathered enough traction
  3. This setup has been said to make it harder to contribute to our applications (please say something about that if it’s your case)
  4. Almost all our applications are deployed as containers in Openshift, so there’s an opportunity here to make the dev environment closer to the prod environment

The Vagrant setup does have a few qualities that I would like to keep:

  • the apps have a fixed DNS name that can be reached by the host and by other apps in VMs outside of tinystage
  • it’s easy to modify the code of applications in tinystage and restart the service without having to rebuild anything (this seems harder to do in containers, although I’ve heard of containers that start services using systemd, which can then be restarted, etc)
  • files are synced between the host and the virtualized app. Vagrant uses sshfs for that but I suppose a container volume mount would do the same?

I’m looking into alternatives to this setup. What are people using these days? Do you have suggestions? At this point I’ve heard of or had a quick look at the following options:

  • bcd, @adamwill 's solution for Bodhi, based on Ansible playbooks starting containers with Podman
  • k3s, minikube, minishift and microshift, which are all kubernetes distributions targeted at lower-resources devices
  • CRC, an OpenShift/Microshift deployment optimized for development and testing purposes (looks interesting!)
  • podman-compose, an implementation of docker-compose for podman, to describe and run multiple containers.
  • quadlet, a systemd-based solution to run containers
  • Dev Containers, a spec to make it easier to do development on containers, with a reference CLI implementation and IDE extensions

Am I missing important options?

I don’t think we need to have actual k8s distributions for this use case, with scaling, redundency, etc, so k3s, and {mini,micro}{kube,shift} seem a bit too much.

I would really like to keep having a setup with shared resources such as authentication, and a separated development setup (possibly run by the same tool?) that hooks into it, because I would like to avoid having to stand up multiple instances of IPA/Ipsilon/FASJSON when I’m hacking on mutiple applications at the same time.

Any ideas? Are you using or did you use some of those tools? Feedbacks?
Or is there a completely different approach that I haven’t considered?
My main goal is to make it easier for newcomers to contribute to our apps without making it too hard for existing developers that regularly contribute to many of those apps.

Thanks!

2 Likes
2

I’m personally having trouble getting Dev environments started with Vagrant. I’v got tiny stage set up and running but I can’t seem to get fedbadges ortahrir going also.

I feel like part of it is that I have never used it before, and I’m figuring it all out. The part is that the Dev setup guide is a bit outdated, and I am finding myself having to spend a bunch of time looking up how things are done now since some of the software used has been through many updates.

I understand most contributors that are poking around in this space probably have more experience than me, and I expect that my understanding stuff is just growing pains and that I’ll figure it out. Don’t take this as me complaining. I’m incredibly grateful that someone took the time to write a guide to help someone like me. <3

Seeing as Vagrant isn’t free anymore, I just don’t want to use it or bother to spend much time learning more than the basics.

I know nothing about it, so I’m down to look into it.

Haven’t yet used any Kubernetes stuff, but I’ve read some stuff about minishift and microshift.

Yeah, it does look interesting! The other day, I was working on setting up a local OpenShift install and messing around with it. So far, so good.

Ayy podman-compose is pretty cool, and I’ve used it before. It would also be good for people who have used docker-compose. I believe it’s 1:1 with it, so it’s easy onboarding.

I used this a bunch with VsCode, and it was super easy to set up when I still ran Windows and Docker. Now that I’ve switched to Fedora and Podman, it’s been a struggle. VsCode seems to hate Podman, and getting it working is a pain. (I’ve had it working before, and kind of working now, but dang… if you have any tips or can show me how you set it up, DM me)

I’m happy that it’s being considered to switch from VMs to Containers.(from what I understand, Vagrant makes VMs). Containers, for me, are so much easier to control, configure, and get up-to-date info about if I need it.

1 Like
3

The simple solution could be to have podman(docker)-compose file and Makefile to control it.

We have something similar in the-new-hotness. It’s easy to use, but the only issue I have with this setup is the need to redeploy it after each change.

But maybe having containers that allow you to restart the service inside the container could be a better solution. Which is the main reason I’m using vagrant for development.

4

I’ve been reading the Developer Guide docs the other day, I think this would be topic would be helpful in my case as I’m starting to familiarize myself how infra works.

Regarding vagrant as usage, I think it’s the easiest setup as its already documented and all but yeah, it’s not using common OSS license. I’ve used vagrant before and it’s quite heavy when you’re trying to provision multiple VMs in a resource constraint workstation.

I agree with this making dev as similar to prod environment, to easily replicate hard to track bugs. There are pros and cons for this like we don’t need scaling and there are things like setting up CNI, SA’s, metallb or custom LB’s to access it outside host (aside from ingress gateways) which makes this too complicated for app dev work. The pro’s is using something Dev Containers vscode plugin to make changes directly on running containers I’ve used this setup before in development.

Dev Containers the answer can’t find anything close to this in openvsx plugins. I think there’s one in Emacs.

Possible for this is put the mutable app source in a mounted volume and restarting the service would be like podman restart name. This would restart one service (e.g. fasjson) without affecting the other auth services.

Other important options to run:

  • podman-kube-play - It uses pods similar to k8s pods, so to separate services you need to create multiple pod definitions.

As of now I’m still reading docs and trying to setup the dev environment. Maybe I’ll try a different method, and also why don’t we use CoreOS image using layering

5

Can you file an issue if the Docs are outdated ?

https://pagure.io/infra-docs-fpo/new_issue

1 Like
6

bcd solves “the VMs take up quite a bit of RAM”, “Vagrant is not free software”, “opportunity to make dev closer to prod”, “easy to modify and restart”, and “files synced between host and virtualized app”. It does not solve “apps have a fixed DNS name” because I don’t like the editing of /etc/hosts that vagrant does, so I did not want to reproduce that, and I didn’t want to get deeper into unprivileged podman networking than necessary (I just got far enough for it to work using localhost, which is all I wanted). It could probably be solved if somebody cared, though.

I can’t comment on whether it solves “make it harder to contribute to our applications”, but I hope so. :stuck_out_tongue: There is room for improvement in making it easier to rebuild the whole environment when needed/desired, I never got around to hooking that up but it would be easy.

I found the mini-kubernetes things still way too complicated for the purpose, and Dan Walsh tells me another drawback of them is they will run as daemons and sit around eating a ton of resources waiting to see if you need to scale (to which the answer is always “no”).

I did not find CRC in my research.

The second version of bcd used podman-compose, but I wound up liking the ansible design more; it just felt nicer, and seems a bit more maintained. (There was an outright problem I had with podman-compose, too, but I can’t remember what it was; probably networking related).

I did not try the quadlet stuff, but I agree it looks promising, though I’m not sure it’s nicer to ship a bunch of systemd services than some ansible plays, which is what it boils down to.

I found dev containers after I’d built bcd, so haven’t looked into it in a lot of detail.

As a general note I don’t think there’s a ton of difference between the bcd approach (which uses Containers.Podman — Ansible Community Documentation - bcd is really just a very small script which wraps a bunch of ansible plays), podman-compose, and quadlets. Ultimately they’re just slightly different ways of defining some podman containers. Once I’d built bcd twice (once with vagrant’s container backend, once on podman-compose) it was pretty easy to build it a third time on the ansible collection - it becomes more apparent that you’re more or less doing the same thing with minor variations.

I made bcd have its own ipsilon and greenwave because I didn’t want it to depend on the VM version of tinystage, but if we get a nice modernized containerized tinystage I can tweak it back to relying on that again (and/or rebuild bcd on the new tinystage platform, if it looks good).

Thanks for looking at this!

1 Like
7

Oh, forgot to mention: Dan suggesting asking @walters for ideas about this topic, so tagging him.

8

When working on my matrix bot I did have to solve some issues with tinystage first (git got updated and was freaking out about the security implications of having different user ownership of files due to vagrants mounting of folders into the VMs, among some others). I was able to get past them without being fazed too much, but ive also seen some new contributors trying to work on tahrir having issues with tinystage in the badges chat on matrix.

not sure why this made me think of bootc for a minute until i realized having a suite of containers each with their own kernel is overkill - heh

I feel a bit underqualified to take on an infra migration like this, but i like the idea!

9

Thanks @adamwill for your headsup here[1])!

I would like to echo my comment[2] from the official tracker[3] here.

As of 01 Aug 2024, the namespace on which viagrunts had been previously kept has been changed from viagrunts to Be-Secure and the updated URL to the project is here. I am not sure if this is a simple renaming of the namespace or a change of hands but the official documentation does not spark confidence. The last commit to the repository was made on 09 Feb 2024 (which is approximately six months back) and the most recent tag is from 13 Jan 2024 (which is approximately seven months back). The maintainers do not seem to have bothered to change the now-obsolete URL in the about section and the issue tracker seems to be disabled limiting our reach to those who might tell us something.

I have not yet explored the containers-bound implementation but might I suggest cloud-init? It is dual licensed under Apache License 2.0 and GNU General Public License 3.0 and later as mentioned in their repository. We generate base cloud images anyway for Fedora Linux and all we need to do is write some YAML instead of Vagrantfile. Sure, this approach would be heavy in terms of storage capacity usage as compared to Vagrant which used to provide us with minimal images. I do think that we should be able to retain most of the libvirt based implementation that we have now. The project is well maintained (with the last commit being made six hours before writing this comment) and the documentation is also extensive.

Alternatively, might I suggest systemd-nspawn? With a host-only network - this can provide us with lightweight Systemd containers that still make use of the host kernel while providing facilities that are very close to those of VMs. This is following up the suggestion I made during the Infra & Releng call where I was finding it hard to recall the name and @arrfab reminded me of the same.

  1. ↩︎

  2. ↩︎

  3. ↩︎

10

My preferred solutions are podman-compose/docker-compose or something utilizing ansible playbooks like bcd.

Podman containers on a shared network can reach each other internally by their names, but this doesn’t apply from the host. Dealing with /etc/hosts to add custom entries is a bit wonky, but I of course possible with the ansible lineinfile module.

You can always mount the sources into the container via a volume mount and only keep the dependencies in the pre-built container to avoid having to rebuild them every time a line is changed in the code.

Yeah, container volume mounts are simple and more performant than sshfs.

I am a fan of Ansible playbooks. We could have a shared Ansible Collection in its own repository that sets up the authentication and other shared resources and then each project can import the role alongside its own tasks to run the project-specific containers.

Any type of kubernetes seems a bit overkill to me. Also, I don’t think any of these are packaged in Fedora.

Using the compose format is nice, as it’s familiar to many people and can also be used with docker-compose and other tools. Compose is probably simpler than a custom, ansible-based approach.

As far as I know, quadlet is meant for managing long-running containers with systemd, not ephemeral ones for development. Also, relying on systemd excludes people who use other operating systems to develop for Fedora.

I’m not super familiar with these. Aren’t they pretty tied to the proprietary Microsoft version of VSCode?

Yup.

That would be more difficult with the podman-compose setup, as the idea is that each repo has its own separate compose file with its own containers, but perhaps more doable with a more custom, ansible-based approach.

11

bcd does implement the “use a volume to have the working tree in the container” thing and the “container with systemd in it” thing, btw. Neither is very difficult. To use systemd in a container all you really have to do in the Containerfile is install it, enable the services you want, and have the container’s CMD line be CMD [ "/usr/sbin/init" ].

1 Like
12

I have enjoyed docker compose , but I am a bit disappointed by podman-compose. I set up a trivial example with rabbitmq, and I realized that depends_on doesn’t seem to be supported. I feel like this would make larger environments hard to manage. (After a quick glance, it seems most services/containers in tiny-stage would probably depend on fedora-messaging. Please correct me if I’m wrong.)

Missing depends_on service_healthy Issue