fcos /sysroot/ostree/deploy/fedora-coreos/var/tmp not getting cleaned when pulling images with podman

Project Discussion
1

I’ve created several Fedora CoreOS Virtual Machines for one service like Dokuwiki on each. In these machines an unprivileged user runs these services using podman and systemd units that run “lingered”.

As times go by, my node_exporter (prometheus, also running as systemd unit but with different user) made my grafana alerts trigger telling me that the folder /sysroot/ostree/deploy/fedora-coreos/var/ is running out of space (after short research it’s the tmp folder in there that grows without limit). When i open /sysroot/ostree/deploy/fedora-coreos/var/tmp i see 4 folders owned by root (with almost no content) and lots, lots of folders owned by the unpriviliged user dokuwiki that runs the dokuwiki container as systemd unit. Opening one of these folders show me that there are files just named by number 1, 2, 3 and so on, as the network traffic showed up te be the same like a docker pull i tried to convert these 1 2 3 files to something.tar.gz and opened it using 7zip showing me for an example that it’s content is a bitnami/dokuwiki container, what I expected. So each time the systemd unit starts it tries to pull the latest images, which is ok and works as intended. But at first, why these temporarily downloads remain at /sysroot/ostree/deploy/fedora-coreos/var/tmp and why these directories does not get cleaned up after reboot, onece a day (another systemd per default ought to run named 'systemd-tmpflies-clean.service / .timer) both are enabled and executed but did not clean up anything. So what do I miss here? I am pretty new to fcos but as far i know the /sysroot/ostree/deploy/fedora-coreos/var/ path is, from a ostree prerspective, ment to contain app data that changes and the tmp folder there ought to be deleted automatically. I’ve read that I might be able to set up the tmp file path for podman by using an environment variable but I just use the fcos image in a pretty default way, its version is ‘fedora-coreos-36.20220605.3.0-live.x86_64.iso’ and the problem remains even after zinactl service updated it to latest version.

I have no clue what the problem is and journalctl and /var/logs do not tell me any amazing story. So your help is very much appreciated :slight_smile:

With friendly greetings from Austria :wave:
Keksli

2

Do you have an example butane config of your setup or at least the systemd units that you are running (with linger) that are starting the containers and leaving behind files?

3 
variant: fcos
version: 1.4.0
passwd:
  users:
    - name: dokuwiki
      ssh_authorized_keys:
        - some key
    - name: sudouser
      ssh_authorized_keys:
        - some key
      groups:
        - sudo
storage:
  directories:
    - path: /home/dokuwiki/.config
      mode: 0755
      user:
        name: dokuwiki
      group:
        name: dokuwiki
    - path: /home/dokuwiki/.config/systemd
      mode: 0755
      user:
        name: dokuwiki
      group:
        name: dokuwiki
    - path: /home/dokuwiki/.config/systemd/user
      mode: 0755
      user:
        name: dokuwiki
      group:
        name: dokuwiki
    - path: /home/dokuwiki/.config/systemd/user/default.target.wants
      mode: 0755
      user:
        name: dokuwiki
      group:
        name: dokuwiki
    - path: /home/dokuwiki/dokuwiki
      mode: 0755
      user:
        name: dokuwiki
      group:
        name: dokuwiki
    - path: /opt/cert
      mode: 0755
      user:
        name: dokuwiki
      group:
        name: dokuwiki
    - path: /opt/node_exporter
      mode: 0755
      user:
        name: sudouser
      group:
        name: sudouser
  files:
    - path: /var/lib/systemd/linger/sudouser
      mode: 0644
      user:
        name: root
      group:
        name: root
    - path: /var/lib/systemd/linger/dokuwiki
      mode: 0644
      user:
        name: root
      group:
        name: root
    - path: /home/dokuwiki/create-export-and-upload-to-somewhere.sh
      mode: 0700
      contents:
        local: create-export-and-upload-to-somewhere.sh
      user:
        name: dokuwiki
      group:
        name: dokuwiki
    - path: /etc/hostname
      mode: 0644
      contents:
        inline: dokuwiki
      user:
        name: root
      group:
        name: root
      overwrite: true
    - path: /opt/cert/dokuwiki.pem
      mode: 0644
      contents:
        local: cert/dokuwiki.pem
      user:
        name: dokuwiki
      group:
        name: dokuwiki
    - path: /opt/cert/dokuwiki.key
      mode: 0600
      contents:
        local: cert/dokuwiki.key
      user:
        name: dokuwiki
      group:
        name: dokuwiki
    - path: /opt/nginx.conf
      mode: 0644
      contents:
        local: nginx/nginx.conf
      user:
        name: dokuwiki
      group:
        name: dokuwiki
    # node_exporter (will download the given exporter)
    - path: /opt/node_exporter/ensure_node_exporter_binary.sh
      mode: 0700
      contents:
        local: node_exporter/ensure_node_exporter_binary.sh
      user:
        name: sudouser
      group:
        name: sudouser
    - path: /opt/node_exporter/node_exporter.yml
      mode: 0644
      contents:
        local: node_exporter/node_exporter.yml
      user:
        name: sudouser
      group:
        name: sudouser
    - path: /opt/node_exporter/dokuwiki.pem
      mode: 0644
      contents:
        local: cert/dokuwiki.pem
      user:
        name: sudouser
      group:
        name: sudouser
    - path: /opt/node_exporter/dokuwiki.key
      mode: 0600
      contents:
        local: cert/dokuwiki.key
      user:
        name: sudouser
      group:
        name: sudouser
    - path: /home/dokuwiki/.config/systemd/user/dokuwiki-pod.service
      mode: 0644
      contents:
        inline: |
          [Unit]
          Description=Podman dokuwiki-pod.service
          Documentation=man:podman-generate-systemd(1)
          Wants=network-online.target
          After=network-online.target
          RequiresMountsFor=
          Requires=dokuwiki.service nginx.service
          Before=dokuwiki.service nginx.service

          [Service]
          Environment=PODMAN_SYSTEMD_UNIT=%n
          Restart=always
          RestartSec=60
          TimeoutStopSec=120
          ExecStartPre=/bin/rm -f %t/dokuwiki-pod.pid %t/dokuwiki-pod.pod-id
          ExecStartPre=/usr/bin/podman pod create --infra-conmon-pidfile %t/dokuwiki-pod.pid --pod-id-file %t/dokuwiki-pod.pod-id --infra=true --name dokuwiki-pod -p 8443:443 --replace
          ExecStart=/usr/bin/podman pod start --pod-id-file %t/dokuwiki-pod.pod-id
          ExecStop=/usr/bin/podman pod stop --ignore --pod-id-file %t/dokuwiki-pod.pod-id -t 60
          ExecStopPost=/usr/bin/podman pod rm --ignore -f --pod-id-file %t/dokuwiki-pod.pod-id
          PIDFile=%t/dokuwiki-pod.pid
          Type=forking
      user:
        name: dokuwiki
      group:
        name: dokuwiki
    - path: /home/dokuwiki/.config/systemd/user/nginx.service
      mode: 0644
      contents: 
        inline: |
          [Unit]
          Description=Podman nginx.service
          Documentation=man:podman-generate-systemd(1)
          Wants=network-online.target
          After=network-online.target
          RequiresMountsFor=%t/containers
          BindsTo=dokuwiki-pod.service
          After=dokuwiki-pod.service
          
          After=network.target
          
          [Service]
          Environment=PODMAN_SYSTEMD_UNIT=%n
          Restart=always
          TimeoutStartSec=60
          TimeoutStopSec=120
          ExecStartPre=/bin/rm -f %t/%n.ctr-id
          ExecStartPre=/usr/bin/podman pull docker.io/library/nginx:latest
          ExecStart=/usr/bin/podman run \
          --cidfile=%t/%n.ctr-id \
          --cgroups=no-conmon \
          --rm \
          --pod-id-file %t/dokuwiki-pod.pod-id \
          --sdnotify=conmon \
          -d \
          --replace \
          --name nginx \
          -v /opt/cert:/cert:Z \
          -v /opt/nginx.conf:/etc/nginx/conf.d/default.conf:Z docker.io/library/nginx:latest
          ExecStop=/usr/bin/podman stop --ignore --cidfile=%t/%n.ctr-id
          ExecStopPost=/usr/bin/podman rm -f --ignore --cidfile=%t/%n.ctr-id
          Type=notify
          NotifyAccess=all
      user:
        name: dokuwiki
      group:
        name: dokuwiki
    - path: /home/dokuwiki/.config/systemd/user/dokuwiki.service
      mode: 0644
      contents:
        inline: |
          [Unit]
          Description=Podman dokuwiki.service
          Documentation=man:podman-generate-systemd(1)
          Wants=network-online.target
          After=network-online.target
          RequiresMountsFor=%t/containers
          BindsTo=dokuwiki-pod.service
          After=dokuwiki-pod.service

          [Service]
          Environment=PODMAN_SYSTEMD_UNIT=%n
          Restart=always
          TimeoutStartSec=60
          TimeoutStopSec=120
          ExecStartPre=/bin/rm -f %t/%n.ctr-id

          ExecStartPre=-/usr/bin/podman unshare chown -R 1001:1001 /home/dokuwiki/dokuwiki
          
          ExecStartPre=/usr/bin/podman pull docker.io/bitnami/dokuwiki:latest
          ExecStart=/usr/bin/podman run \
          -u 1001 \
          --cidfile=%t/%n.ctr-id \
          --cgroups=no-conmon \
          --rm \
          --pod-id-file %t/dokuwiki-pod.pod-id \
          --sdnotify=conmon \
          -d \
          --replace \
          --name dokuwiki \
          --volume /home/dokuwiki/dokuwiki:/bitnami/dokuwiki:Z \
          -e DOKUWIKI_PASSWORD=unknownpassword \
          -e DOKUWIKI_USERNAME=admin \
          -e DOKUWIKI_WIKI_NAME=somewikiname \
          docker.io/bitnami/dokuwiki:latest
          
          ExecStop=/usr/bin/podman stop --ignore --cidfile=%t/%n.ctr-id
          ExecStopPost=/usr/bin/podman rm -f --ignore --cidfile=%t/%n.ctr-id
          Type=notify
          NotifyAccess=all
      user:
        name: dokuwiki
      group:
        name: dokuwiki
    - path: /home/dokuwiki/.config/systemd/user/dokuwiki-backup.service
      mode: 0644
      contents:
        inline: |
          [Unit]
          Description=Stops the pod, compresses the containers host file content and uploads the data to a Seafile library.

          [Service]
          Type=oneshot
          WorkingDirectory=/home/dokuwiki
          ExecStart=/usr/bin/bash -c '/home/dokuwiki/create-export-and-upload-to-somewhere.sh'
      user:
        name: dokuwiki
      group:
        name: dokuwiki
    - path: /home/dokuwiki/.config/systemd/user/dokuwiki-backup.timer
      mode: 0644
      contents:
        inline: |
          [Unit]
          Description=Run dokuwiki-backup.service everyday at 22:00.

          [Timer]
          OnCalendar=22:00
          Unit=dokuwiki-backup.service
      user:
        name: dokuwiki
      group:
        name: dokuwiki
  links:
    # configure timezone
    - path: /etc/localtime
      target: /usr/share/zoneinfo/Europe/Berlin
      user:
        name: root
      group:
        name: root
      overwrite: true
    # links to custom services
    - path: /home/dokuwiki/.config/systemd/user/default.target.wants/dokuwiki-pod.service
      user:
        name: dokuwiki
      group:
        name: dokuwiki
      target: /home/dokuwiki/.config/systemd/user/dokuwiki-pod.service
      hard: false
    - path: /home/dokuwiki/.config/systemd/user/default.target.wants/nginx.service
      user:
        name: dokuwiki
      group:
        name: dokuwiki
      target: /home/dokuwiki/.config/systemd/user/nginx.service
      hard: false
    - path: /home/dokuwiki/.config/systemd/user/default.target.wants/dokuwiki.service
      user:
        name: dokuwiki
      group:
        name: dokuwiki
      target: /home/dokuwiki/.config/systemd/user/dokuwiki.service
      hard: false
    - path: /home/dokuwiki/.config/systemd/user/default.target.wants/dokuwiki-backup.service
      user:
        name: dokuwiki
      group:
        name: dokuwiki
      target: /home/dokuwiki/.config/systemd/user/dokuwiki-backup.service
      hard: false
    - path: /home/dokuwiki/.config/systemd/user/default.target.wants/dokuwiki-backup.timer
      user:
        name: dokuwiki
      group:
        name: dokuwiki
      target: /home/dokuwiki/.config/systemd/user/dokuwiki-backup.timer
      hard: false
systemd:
  units:
    - name: node-exporter.service
      enabled: true
      contents: |
        [Unit]
        Description=node-exporter
        After=network-online.target user-runtime-dir@1002.service
        Wants=network-online.target user-runtime-dir@1002.service

        [Service]
        User=sudouser
        Group=sudouser
        Restart=always
        RestartSec=30

        WorkingDirectory=/opt/node_exporter
        # Ensure proper UNIX line endings.
        ExecStartPre=/bin/bash -c "sed -i.bak 's/\r$//' ensure_node_exporter_binary.sh"
        ExecStartPre=/bin/bash ensure_node_exporter_binary.sh
        ExecStart=/bin/bash -c "/opt/node_exporter/node_exporter --web.config=node_exporter.yml --collector.systemd '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc|usr|var)($$|/)'"

        [Install]
        WantedBy=multi-user.target
4

Sorry Dusty for the late response, I was unable to post due to illness.

Above is the ingnition.yml I use in combination with the windows version of buthane.
Here the content of the bat file I use to create the installer iso file:

butane.exe --files-dir=../config --strict ignition.yml -o ignition.json
docker run --rm -v %cd%:/app -w /app quay.io/coreos/coreos-installer:latest iso customize --dest-device=/dev/vda --dest-ignition ignition.json -o coreos-dokuwiki-custom.iso fedora-coreos-36.20220605.3.0-live.x86_64.iso
5

To temporarily clean up the disk space i used the folowing statement as root:
(Is it safe to use it? → it will delete all folders in there that are older than one day and containing storage in their name)

find /sysroot/ostree/deploy/fedora-coreos/var/tmp/ -iname '*storage*' -mtime +1 -exec rm -Rf {} \;
6

Any ideas how to go on?

7

Any additional information you need?

The disk consumption increases again, the problem still exists.

Have a nice day :slight_smile:

Keksli

8

/var/tmp is not cleaned-up automatically on reboot. So you’ll have to figure out which container or program is writing to it and use /tmp or something else instead.

9

Hello Timothee,

and thank you for the response!

Do you know why podman (and maebye docker aswell), which is preinstalled and preconfigured by FCOS, stores stuff in /sysroot/ostree/deploy/fedora-coreos/var/tmp instead of /tmp?

And shouldn’t the /sysroot/ostree/deploy/fedora-coreos/var/tmp path be cleaned automatically too, because it is named like a temporary content folder?

The program writing to /var/tmp ought to be podman because the folder is being filled up by downloaded container images, as written in the first entry of this discussion.

Have a nice weekend :slight_smile:
Christoph

10

/sysroot/ostree/deploy/fedora-coreos/var/tmp is /var/tmp.

I don’t have anything in /var/tmp on my FCOS server so you need to find in your setup what’s doing that. Maybe it’s podman storing the top layer changes from the container on the disk there?

11

Hello Tim, Dusty,

sorry for the very late response!

I was watching that Issue quite for a while now on my systems and as far I can tell, Core OS is working as intended.

As a newbie I mixed up things and I underestimated the complexity of my scenario and what podman needs to face to run proper in such a case.

Respect and thank you to everyone working on those tools and sorry for wasting your time rising an unnecessary ticket.

Best regards
Keksli

1 Like