How to have a working unless-stopped behavior for containers?

Ask Fedora
,
1

Hi,

I am new to CoreOS.
I am using podman for all my services.
I can not figure out how to get unless-stopped working as expected (after a reboot, start only the containers that were running before the reboot).

podman-restart.service only starts containers on always restart policy. Adding unless-stopped to it only makes it the same behavior as always.

I am trying for the past 2 weeks to find a working way to have the correct behavior with the help of Claude AI. Here is what I have so far:

variant: fcos
version: 1.5.0
storage:
  files:
    - path: /usr/local/bin/podman-save-unless-stopped.sh
      mode: 0755
      contents:
        inline: |
          #!/bin/bash
          set -euo pipefail
          podman ps \
            --filter restart-policy=unless-stopped \
            --format '{{.Names}}' \
            > /var/lib/podman-unless-stopped-state

    - path: /usr/local/bin/podman-start-unless-stopped.sh
      mode: 0755
      contents:
        inline: |
          #!/bin/bash
          set -euo pipefail
          state=/var/lib/podman-unless-stopped-state
          if [[ ! -f "$state" ]]; then
            podman start --all --filter restart-policy=unless-stopped || true
            exit 0
          fi
          while IFS= read -r name; do
            [[ -n "$name" ]] && podman start "$name" || true
          done < "$state"

systemd:
  units:
    - name: podman-restart.service
      enabled: true

    - name: podman-unless-stopped-boot.service
      enabled: true
      contents: |
        [Unit]
        Description=Restore unless-stopped containers at boot
        After=network-online.target podman-restart.service
        Wants=network-online.target

        [Service]
        Type=oneshot
        RemainAfterExit=yes
        ExecStart=/usr/local/bin/podman-start-unless-stopped.sh

        [Install]
        WantedBy=multi-user.target

    - name: podman-unless-stopped-shutdown.service
      enabled: true
      contents: |
        [Unit]
        Description=Save unless-stopped container state before shutdown
        DefaultDependencies=no
        Before=shutdown.target reboot.target halt.target
        Requires=network.target

        [Service]
        Type=oneshot
        ExecStart=/usr/local/bin/podman-save-unless-stopped.sh

        [Install]
        WantedBy=shutdown.target reboot.target halt.target

This does not work.

I am going in circles with Claude for 2 weeks now and can not figure out why this is not saving the containers that are currently running.

Do you have a working way of getting the expected unless-stopped behavior for containers?

Thanks in advance for any answer, have a nice day

2

You’d usually want each container individually managed as a systemd service, instead of manually managing them in podman and building custom logic for them to be restarted after reboot. Those services you then enable or disable through systemctl. The recommended way is writing Quadlet files for such podman services, instead of plain systemd services.

Use systemctl enable --now my-service to start a service and mark it to be started at boot too. Use systemctl disable --now my-service to stop and disable it.

If you want crashing services to restart automatically, use Restart=always in the [Service] section. Be sure to check out the RestartSec=, RestartSteps=, and RestartMaxDelaySec= options though, to prevent a burst of failed start attempts from breaking restart behavior. See “StartLimitIntervalSec=” in man systemd.unit.

I’ve built an abstractions for generating Quadlet services running as unprivileged users from a YAML based format and an example of how to use it, that might be of interest to you.

3

Hello,

Thanks for your very complete answer.

My problem here is that I want to use sablier GitHub - sablierapp/sablier: Start your containers on demand, shut them down automatically when there's no activity. Docker, Docker Swarm Mode and Kubernetes compatible. · GitHub to start and stop containers on demand. And sablier can only start and stop containers from podman socket. From what I understood, it can not do this with systemd.

Do you know a way to have the unless-stopped behavior while still using podman start and stop?

4

I’m unfortunately not familiar with sablier. Does it not allow starting and managing a statically configured set of containers? Explicitly specifying stuff like that in a config file and adapting it if something needs to change, sounds like the cleaner approach to me.

5

I already have all my software stack configured as docker-compose files. Reverse proxy, sablier configuration is defined as compose labels. All the networking is done through podman networks. Deployment and modifications of containers are all done automatically through compose and portainer when the git containing the docker compose is updated. For these reasons, and from what I understand about quadlets, it will not fit my use case which is to use only podman features as abstraction from the OS.

I appreciate a lot not having to touch anything OS or systemd related when adding or editing containers. I am just editing my compose, commit, and it’s automatically deployed and updated.

I am migrating from debian to fedora coreos.

Sablier will be using the podman socket. From what I understand, it can not manage statically configured containers.

What Claude AI is suggesting me to do (and that I am failing to implement) is to have systemd scripts that save a list of all the running containers at shutdown time in a file, to start them again after the reboot. But there may be another way.

6

Okay. Consider saving state whenever you make a change though, to avoid losing it when the system doesn’t shut down cleanly. If your tools don’t have the ability to specify hooks, writing wrapper scripts might work.

Other than that I see debugging your scripts/services step by step until they do what they are supposed to as the best way forward. Perform the script’s steps manually, add debugging outputs, inspect systemd journals, etc. until you find what’s wrong :person_shrugging: