A future for tiny-stage

Project Discussion
1

Hey folks!

To develop on most of our infrastructure applications, we use a tool called tiny-stage which is based on Vagrant, to run a few virtual machines with the authentication stack and a few common tools. The apps themselves all have a Vagrant VM for development that is setup to call the Tinystage VMs for authentication and the rest. This setup has worked well for us but it has shortcomings:

  1. the VMs take up quite a bit of RAM. We try to colocalize the apps on some VMs but we usulally need at least two, IPA and the auth stack.
  2. Vagrant is not Free software anymore, and it doesn’t look like any of the forks have gathered enough traction
  3. This setup has been said to make it harder to contribute to our applications (please say something about that if it’s your case)
  4. Almost all our applications are deployed as containers in Openshift, so there’s an opportunity here to make the dev environment closer to the prod environment

The Vagrant setup does have a few qualities that I would like to keep:

  • the apps have a fixed DNS name that can be reached by the host and by other apps in VMs outside of tinystage
  • it’s easy to modify the code of applications in tinystage and restart the service without having to rebuild anything (this seems harder to do in containers, although I’ve heard of containers that start services using systemd, which can then be restarted, etc)
  • files are synced between the host and the virtualized app. Vagrant uses sshfs for that but I suppose a container volume mount would do the same?

I’m looking into alternatives to this setup. What are people using these days? Do you have suggestions? At this point I’ve heard of or had a quick look at the following options:

  • bcd, @adamwill 's solution for Bodhi, based on Ansible playbooks starting containers with Podman
  • k3s, minikube, minishift and microshift, which are all kubernetes distributions targeted at lower-resources devices
  • CRC, an OpenShift/Microshift deployment optimized for development and testing purposes (looks interesting!)
  • podman-compose, an implementation of docker-compose for podman, to describe and run multiple containers.
  • quadlet, a systemd-based solution to run containers
  • Dev Containers, a spec to make it easier to do development on containers, with a reference CLI implementation and IDE extensions

Am I missing important options?

I don’t think we need to have actual k8s distributions for this use case, with scaling, redundency, etc, so k3s, and {mini,micro}{kube,shift} seem a bit too much.

I would really like to keep having a setup with shared resources such as authentication, and a separated development setup (possibly run by the same tool?) that hooks into it, because I would like to avoid having to stand up multiple instances of IPA/Ipsilon/FASJSON when I’m hacking on mutiple applications at the same time.

Any ideas? Are you using or did you use some of those tools? Feedbacks?
Or is there a completely different approach that I haven’t considered?
My main goal is to make it easier for newcomers to contribute to our apps without making it too hard for existing developers that regularly contribute to many of those apps.

Thanks!

3 Likes
2

I’m personally having trouble getting Dev environments started with Vagrant. I’v got tiny stage set up and running but I can’t seem to get fedbadges ortahrir going also.

I feel like part of it is that I have never used it before, and I’m figuring it all out. The part is that the Dev setup guide is a bit outdated, and I am finding myself having to spend a bunch of time looking up how things are done now since some of the software used has been through many updates.

I understand most contributors that are poking around in this space probably have more experience than me, and I expect that my understanding stuff is just growing pains and that I’ll figure it out. Don’t take this as me complaining. I’m incredibly grateful that someone took the time to write a guide to help someone like me. <3

Seeing as Vagrant isn’t free anymore, I just don’t want to use it or bother to spend much time learning more than the basics.

I know nothing about it, so I’m down to look into it.

Haven’t yet used any Kubernetes stuff, but I’ve read some stuff about minishift and microshift.

Yeah, it does look interesting! The other day, I was working on setting up a local OpenShift install and messing around with it. So far, so good.

Ayy podman-compose is pretty cool, and I’ve used it before. It would also be good for people who have used docker-compose. I believe it’s 1:1 with it, so it’s easy onboarding.

I used this a bunch with VsCode, and it was super easy to set up when I still ran Windows and Docker. Now that I’ve switched to Fedora and Podman, it’s been a struggle. VsCode seems to hate Podman, and getting it working is a pain. (I’ve had it working before, and kind of working now, but dang… if you have any tips or can show me how you set it up, DM me)

I’m happy that it’s being considered to switch from VMs to Containers.(from what I understand, Vagrant makes VMs). Containers, for me, are so much easier to control, configure, and get up-to-date info about if I need it.

1 Like
3

The simple solution could be to have podman(docker)-compose file and Makefile to control it.

We have something similar in the-new-hotness. It’s easy to use, but the only issue I have with this setup is the need to redeploy it after each change.

But maybe having containers that allow you to restart the service inside the container could be a better solution. Which is the main reason I’m using vagrant for development.

4

I’ve been reading the Developer Guide docs the other day, I think this would be topic would be helpful in my case as I’m starting to familiarize myself how infra works.

Regarding vagrant as usage, I think it’s the easiest setup as its already documented and all but yeah, it’s not using common OSS license. I’ve used vagrant before and it’s quite heavy when you’re trying to provision multiple VMs in a resource constraint workstation.

I agree with this making dev as similar to prod environment, to easily replicate hard to track bugs. There are pros and cons for this like we don’t need scaling and there are things like setting up CNI, SA’s, metallb or custom LB’s to access it outside host (aside from ingress gateways) which makes this too complicated for app dev work. The pro’s is using something Dev Containers vscode plugin to make changes directly on running containers I’ve used this setup before in development.

Dev Containers the answer can’t find anything close to this in openvsx plugins. I think there’s one in Emacs.

Possible for this is put the mutable app source in a mounted volume and restarting the service would be like podman restart name. This would restart one service (e.g. fasjson) without affecting the other auth services.

Other important options to run:

  • podman-kube-play - It uses pods similar to k8s pods, so to separate services you need to create multiple pod definitions.

As of now I’m still reading docs and trying to setup the dev environment. Maybe I’ll try a different method, and also why don’t we use CoreOS image using layering

5

Can you file an issue if the Docs are outdated ?

https://pagure.io/infra-docs-fpo/new_issue

1 Like
6

bcd solves “the VMs take up quite a bit of RAM”, “Vagrant is not free software”, “opportunity to make dev closer to prod”, “easy to modify and restart”, and “files synced between host and virtualized app”. It does not solve “apps have a fixed DNS name” because I don’t like the editing of /etc/hosts that vagrant does, so I did not want to reproduce that, and I didn’t want to get deeper into unprivileged podman networking than necessary (I just got far enough for it to work using localhost, which is all I wanted). It could probably be solved if somebody cared, though.

I can’t comment on whether it solves “make it harder to contribute to our applications”, but I hope so. :stuck_out_tongue: There is room for improvement in making it easier to rebuild the whole environment when needed/desired, I never got around to hooking that up but it would be easy.

I found the mini-kubernetes things still way too complicated for the purpose, and Dan Walsh tells me another drawback of them is they will run as daemons and sit around eating a ton of resources waiting to see if you need to scale (to which the answer is always “no”).

I did not find CRC in my research.

The second version of bcd used podman-compose, but I wound up liking the ansible design more; it just felt nicer, and seems a bit more maintained. (There was an outright problem I had with podman-compose, too, but I can’t remember what it was; probably networking related).

I did not try the quadlet stuff, but I agree it looks promising, though I’m not sure it’s nicer to ship a bunch of systemd services than some ansible plays, which is what it boils down to.

I found dev containers after I’d built bcd, so haven’t looked into it in a lot of detail.

As a general note I don’t think there’s a ton of difference between the bcd approach (which uses Containers.Podman — Ansible Community Documentation - bcd is really just a very small script which wraps a bunch of ansible plays), podman-compose, and quadlets. Ultimately they’re just slightly different ways of defining some podman containers. Once I’d built bcd twice (once with vagrant’s container backend, once on podman-compose) it was pretty easy to build it a third time on the ansible collection - it becomes more apparent that you’re more or less doing the same thing with minor variations.

I made bcd have its own ipsilon and greenwave because I didn’t want it to depend on the VM version of tinystage, but if we get a nice modernized containerized tinystage I can tweak it back to relying on that again (and/or rebuild bcd on the new tinystage platform, if it looks good).

Thanks for looking at this!

1 Like
7

Oh, forgot to mention: Dan suggesting asking @walters for ideas about this topic, so tagging him.

8

When working on my matrix bot I did have to solve some issues with tinystage first (git got updated and was freaking out about the security implications of having different user ownership of files due to vagrants mounting of folders into the VMs, among some others). I was able to get past them without being fazed too much, but ive also seen some new contributors trying to work on tahrir having issues with tinystage in the badges chat on matrix.

not sure why this made me think of bootc for a minute until i realized having a suite of containers each with their own kernel is overkill - heh

I feel a bit underqualified to take on an infra migration like this, but i like the idea!

9

Thanks @adamwill for your headsup here[1])!

I would like to echo my comment[2] from the official tracker[3] here.

As of 01 Aug 2024, the namespace on which viagrunts had been previously kept has been changed from viagrunts to Be-Secure and the updated URL to the project is here. I am not sure if this is a simple renaming of the namespace or a change of hands but the official documentation does not spark confidence. The last commit to the repository was made on 09 Feb 2024 (which is approximately six months back) and the most recent tag is from 13 Jan 2024 (which is approximately seven months back). The maintainers do not seem to have bothered to change the now-obsolete URL in the about section and the issue tracker seems to be disabled limiting our reach to those who might tell us something.

I have not yet explored the containers-bound implementation but might I suggest cloud-init? It is dual licensed under Apache License 2.0 and GNU General Public License 3.0 and later as mentioned in their repository. We generate base cloud images anyway for Fedora Linux and all we need to do is write some YAML instead of Vagrantfile. Sure, this approach would be heavy in terms of storage capacity usage as compared to Vagrant which used to provide us with minimal images. I do think that we should be able to retain most of the libvirt based implementation that we have now. The project is well maintained (with the last commit being made six hours before writing this comment) and the documentation is also extensive.

Alternatively, might I suggest systemd-nspawn? With a host-only network - this can provide us with lightweight Systemd containers that still make use of the host kernel while providing facilities that are very close to those of VMs. This is following up the suggestion I made during the Infra & Releng call where I was finding it hard to recall the name and @arrfab reminded me of the same.

  1. ↩︎

  2. ↩︎

  3. ↩︎

10

My preferred solutions are podman-compose/docker-compose or something utilizing ansible playbooks like bcd.

Podman containers on a shared network can reach each other internally by their names, but this doesn’t apply from the host. Dealing with /etc/hosts to add custom entries is a bit wonky, but I of course possible with the ansible lineinfile module.

You can always mount the sources into the container via a volume mount and only keep the dependencies in the pre-built container to avoid having to rebuild them every time a line is changed in the code.

Yeah, container volume mounts are simple and more performant than sshfs.

I am a fan of Ansible playbooks. We could have a shared Ansible Collection in its own repository that sets up the authentication and other shared resources and then each project can import the role alongside its own tasks to run the project-specific containers.

Any type of kubernetes seems a bit overkill to me. Also, I don’t think any of these are packaged in Fedora.

Using the compose format is nice, as it’s familiar to many people and can also be used with docker-compose and other tools. Compose is probably simpler than a custom, ansible-based approach.

As far as I know, quadlet is meant for managing long-running containers with systemd, not ephemeral ones for development. Also, relying on systemd excludes people who use other operating systems to develop for Fedora.

I’m not super familiar with these. Aren’t they pretty tied to the proprietary Microsoft version of VSCode?

Yup.

That would be more difficult with the podman-compose setup, as the idea is that each repo has its own separate compose file with its own containers, but perhaps more doable with a more custom, ansible-based approach.

11

bcd does implement the “use a volume to have the working tree in the container” thing and the “container with systemd in it” thing, btw. Neither is very difficult. To use systemd in a container all you really have to do in the Containerfile is install it, enable the services you want, and have the container’s CMD line be CMD [ "/usr/sbin/init" ].

1 Like
12

I have enjoyed docker compose , but I am a bit disappointed by podman-compose. I set up a trivial example with rabbitmq, and I realized that depends_on doesn’t seem to be supported. I feel like this would make larger environments hard to manage. (After a quick glance, it seems most services/containers in tiny-stage would probably depend on fedora-messaging. Please correct me if I’m wrong.)

Missing depends_on service_healthy Issue

13

Hi folks, I’ve started working on that and I’m going the ansible route, like bcd. I’m however having a networking problem, and I thought you may point me in the right direction (especially @humaton who, I think, spoke to me about that already a few months ago).

Tinystage will run multiple containers in parallel that must:

  1. talk to each other (it’s fine if I have to list the opened ports but ideally I wouldn’t have to)
  2. be accessible from the host, ideally on priviledged ports such as 443
  3. be able to have their own 443 port opened and accessible from the host at the same time
  4. optionally be accessible from the host’s network, for those that prefer developing in the context of a home lab, and not on their local machine.

So ideally all the containers would have different IPs, and there would be a way to resolve the names from the host and the containers themselves. In vagrant there was the hostmanager plugin that would edit the host’s /etc/hosts file, it’s not very elegant and I would prefer some sort of DNS, but if I have to edit /etc/hosts it’s acceptable.

Do you know how I should setup Podman to get this working? I have created a tinystage-specific network, and added the containers to them. In there, containers can resolve other containers and access their ports, so point 1 is OK.

However, I’m unable to access the containers from the host.
I have no name resolution:

$ curl ipa.tinystage.test
curl: (6) Could not resolve host: ipa.tinystage.test

And I can’t access the ports with direct IP either:

$ podman container inspect tinystage-ipa | jq '.[0].NetworkSettings.Networks.tinystage.IPAddress'
"10.89.1.4"
$ curl --connect-timeout 10 10.89.1.4
curl: (28) Failed to connect to 10.89.1.4 port 80 after 10002 ms: Timeout was reached

I could expose the ports, but then I wouldn’t be able to have multiple containers serve on port 443.

I have read podman’s networking docs , but I’m not finding what I should do to get the name resolution and port access. I feel like there should be tons of tutorials to do what I want, but I can’t seem to find anything.

Any ideas?

14

I would explore podman-network-create — Podman documentation You can use --ipam-driver dhcp for the containers to get addresses via dhcp. Now there are a few approaches can take. Get the dhcp running on the host or create a container to run it. I only poked around dhcpd running on the host and netvark proxy.

1 Like
15

I did some messing around just now and i was able to access a container in a bridge network via ip (sort of)

podman run --replace -p 8081:80 --network=bridge --name docker-nginx docker.io/nginx

with my host connected to the podman0 bridge network, it was assigned the IP 10.88.0.1, allowing me to access this nginx container at http://10.88.0.1:8081/

This still has some problems with the port numbers (i.e. two things cannot be on 443 at the same time, and in order to use ports < 1024, rootful podman is needed) though.

Ive seen suggestions for things like using macvlan (or probably ipvlan) networks but i wasnt able to confirm this works on my machine

It sounds like the goal is to replicate the “each box has the ability to use its full range of ports” in containers - so i wonder if having each service (or group of services without conflicting ports) in its own podman-machine would work, even if slightly overkill

16

So, after a couple weeks, I’m not finding a way to make it work. Somebody with better container networking knowledge would probably find a solution quickly, so I’m asking for help here. Here are my requirements for the container network:

  1. Containers must be able to reach each other
  2. Containers must be able to access the internet
  3. Containers must be accessible via their hostname from the host and from each other (if they are accessible via IP, I can probably find a solution to get the name resolution working)
  4. Containers must not have to expose their services on the host on non-standard ports (the OIDC setup requires hardcoding urls and ports in the Ipsilon config)
  5. The setup must not break the host’s default route while being set up
  6. on tinystage shutdown, the host’s networking should be back to normal
  7. The setup must not assume interface names
  8. The network block for containers should be configurable, in case the default one conflicts with the user’s existing setup
  9. ideally rootless, but it’s fine if it requires some root commands
  10. ideally, the setup would keep working when the interface holding the default route goes up/down, or when the default route changes interface (a laptop goes from wifi to ethernet and back).
  11. ideally, the container network would be different from the host’s local network for isolation
  12. ideally, we would not have to manually assign static IPs to individual containers, to make it easier to add more containers.

For example, something looking like the networking that libvirtd maintains when running would be nice (dnsmasq on a bridged interface).

This is actually unrelated to tinystage’s applications, you can test those requirements with a default nginx container.

The references I’ve used so far:

Thanks for any help!