What are people using to backup podman containers?
I am running a F40 server in a home lab environment and hosting my own instance of Joplin for study notes, that I installed using cockpit.
Ideally would like to have a local backup to usb or replicate to an s3 bucket. Any suggestions would be appreciated.
I guess in your case, I would just use an Image like fedora:latest in a podman compose/Dockerfile and install Joplin and any other tools in the image . You would only need to back up your notes.
I’m sure there are better solutions than what I recommend.
Welcome to 
!
Podman containers usually store their data in volumes. It is best practice to avoid storing state/data in the container itself because that makes image updates difficult. Usually creating backups of the volumes is enough, a container can easily be recreated using the same image and config. For recreating the container you might need backups of cockpit, but I never used cockpit so I am not sure.
With that in mind, here are some commands that can be used to create backups of the volumes:
Show all volumes
$ podman volume ls
DRIVER VOLUME NAME
local systemd-caddy
local systemd-git-postgres
local systemd-git-postgres-backup
Backup a volume
$ podman volume export systemd-caddy > caddy-backup.tar
$ ll -ah
-rw-r--r--. 1 core core 126K Jul 6 13:54 caddy-backup.tar
Restore a volume
$ cat caddy-backup.tar | podman volume import systemd-caddy -
Inspect a volume
$ podman volume inspect systemd-caddy
[
{
"Name": "systemd-caddy",
"Driver": "local",
"Mountpoint": "/var/home/core/.local/share/containers/storage/volumes/systemd-caddy/_data",
"CreatedAt": "redacted",
"Labels": {
"backup": "true"
},
"Scope": "local",
"Options": {},
"MountCount": 0,
"NeedsCopyUp": true,
"LockNumber": 0
}
]
As you can see the volume is just a normal directory stored in /var/home/[user]/.local/share/containers/storage/volumes/[volume name]/_data. Creating a backup of this directory is another option. Using a bind mount instead of a volume should also work, but then podman no longer handles the file permissions.
I am using restic to create backups of my container volumes. A script finds the volumes with a backup=true label and uploads them to blackblaze b2 (s3). The backup label is useful to avoid creating backups of cache / database volumes. Databases are backed-up by doing a pg_dump/mysqldump into another volume with a backup=true label (database backups can be inconsistent if backed-up directly). If you are interested in my setup I could share the script that I am using. I think it might be a bit overkill for your setup though, as you are just running a single container.
@brodomir would it be okay if I move this topic into ask fedora? This is a question and Ask gets a lot more visibility/answers than the water cooler.
Of course move it , It’s only here because I was unsure where to post.
I’ll have a look into restic and see if I can get it working. Thank you for the information.
Hello, thanks for your explanation. I’m currently trying to set up regular backups for my self-hosted S3 object storage, but I’m stuck with Restic. Would you be kind enough to share your setup?
Hey, here is what I am using for my servers.
Timers
For scheduling the backups and a weekly prune.
# ~/.config/containers/systemd/backup.timer
[Unit]
Description=Daily restic backup
[Timer]
OnCalendar=*-*-* 23:30
RandomizedDelaySec=900
Persistent=true
[Install]
WantedBy=timers.target
# ~/.config/containers/systemd/prune.timer
[Unit]
Description=Weekly restic prune
[Timer]
OnCalendar=Sun *-*-* 02:00
RandomizedDelaySec=900
Persistent=true
[Install]
WantedBy=timers.target
Backup tasks service
Runs before the restic backup. Used to pg_dump etc…
# ~/.config/containers/systemd/backup-tasks.service
[Service]
Type=oneshot
ExecStart=find %h/.config/containers/systemd/backup/config/tasks -type f -executable -exec '{}' ';'
Restart=on-failure
RestartSec=10min
IOSchedulingClass=idle
Nice=19
Here is one of the scripts that it triggers. My postgres containers have 2 volumes. The first one has live data. The other is mounted to /backup and has the backup=true label. Only the second volume is included in backups.
#!/bin/sh
# ================================================= #
# Create backups for the postgres service. Backups #
# are stored in a backup volume. #
# #
# To restore from the previous backup: #
# $ podman exec -it systemd-chat-postgres psql #
# -U synapse -f /backup/dump.sql #
# #
# For older backups first restore and older version #
# of the backup volume using the backup service. #
# ================================================= #
echo 'chat-postgres: creating backup'
podman exec systemd-chat-postgres pg_dumpall -U synapse -f /backup/dump.sql
echo 'chat-postgres: finished'
Backup container
Uploads the backup to restic. Environment file contains RESTIC_REPOSITORY, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and RESTIC_PASSWORD.
# ~/.config/containers/systemd/backup.container
[Container]
Image=docker.io/restic/restic:0.19.0
Exec=backup --files-from /backup/services/backup/config/backup-include-paths
Pull=newer
HostName=%H
# env
EnvironmentFile=%h/.config/containers/systemd/backup/environment
# storage
Volume=backup-cache.volume:/root/.cache
Volume=%h/.config/containers/systemd:/backup/services:ro
Volume=%h/.local/share/containers/storage/volumes:/backup/volumes:ro
# security
SecurityLabelDisable=true
ReadOnly=true
NoNewPrivileges=true
DropCapability=ALL
AddCapability=DAC_OVERRIDE
[Unit]
After=backup-tasks.service
Wants=backup-tasks.service
[Service]
Restart=on-failure
RestartSec=10min
IOSchedulingClass=idle
Nice=19
Prune container
# ~/.config/containers/systemd/prune.container
[Container]
Image=docker.io/restic/restic:0.19.0
Exec=forget --prune --keep-within 14d --keep-daily 14 --keep-weekly 8 --keep-monthly 6
Pull=newer
HostName=%H
# env
EnvironmentFile=%h/.config/containers/systemd/backup/environment
# storage
Volume=backup-cache.volume:/root/.cache
# security
ReadOnly=true
NoNewPrivileges=true
DropCapability=ALL
[Service]
Restart=on-failure
RestartSec=10min
IOSchedulingClass=idle
Nice=19
Generate script
Script that generates the include file used by the backup container and populates the backup tasks directory. All services have their own directory on my server so I can easily move them. This script detects the installed services and configures the backup service.
#!/bin/sh
# ============================================ #
# configure.sh v2.0 #
# Generate restic include file and backup #
# tasks directory based on installed services. #
# ============================================ #
set -eu
log() { printf '\e[%sm%s\e[0m %s\n' "${3:-36}" "${2:-○}" "$1"; }
ask() { printf '\e[33m?\e[0m %s ' "$1"; read a; [ "${a,,}" = 'y' ]; }
# generate_tasks_dir copies all *.backup scripts from installed services
# to the backup tasks directory.
generate_tasks_dir() {
log 'start generating task directory with *.backup scripts'
mkdir -p "$BACKUP_TASKS_DIR"
find "$BACKUP_TASKS_DIR" -name '*.backup' -exec rm {} \;
find "$SERVICES_DIR" -name '*.backup' -not -path "$BACKUP_TASKS_DIR/*" | while read -r script; do
log "found $script" '↖' 35
cp "$script" "$BACKUP_TASKS_DIR"
done
log 'finished generating task directory' '✓' 32
}
# generate_restic_include_file generates a file containing
# all locations that should be backed up by restic.
generate_restic_include_file() {
log 'start generating restic include file'
echo '/backup/services' > "$BACKUP_RESTIC_INCLUDE_FILE"
log 'search for volumes with backup=true label'
podman volume ls -f label=backup=true --format "{{.Name}}" | while read -r volume; do
log "found $volume" '↖' 35
echo "/backup/volumes/$volume" >> "$BACKUP_RESTIC_INCLUDE_FILE"
done
log 'finished generating include file' '✓' 32
}
## MAIN ##
SERVICES_DIR="$HOME/.config/containers/systemd"
BACKUP_RESTIC_INCLUDE_FILE="$SERVICES_DIR/backup/config/backup-include-paths"
BACKUP_TASKS_DIR="$SERVICES_DIR/backup/config/tasks"
generate_tasks_dir
generate_restic_include_file
ask "enable the daily backup timer by executing 'systemctl --user enable --now backup.timer' (y/N)?" && \
systemctl --user daemon-reload && \
systemctl --user enable --now backup.timer && \
log 'successfully enabled the backup timer' '✓' 32
ask "enable the weekly prune timer by executing 'systemctl --user enable --now prune.timer' (y/N)?" && \
systemctl --user daemon-reload && \
systemctl --user enable --now prune.timer && \
log 'successfully enabled the prune timer' '✓' 32
Restore helper
Script I use that helps with doing restores / interacting with the restic repositoy.
#!/bin/sh
podman run -it --rm \
-v systemd-backup-cache:/root/.cache \
-v /home:/host-home \
--security-opt label=disable \
--env-file ~/.config/containers/systemd/backup/environment \
--entrypoint ash \
docker.io/restic/restic