Using Borg Backup on Silverblue

[jakfrost (Stephen Snow)]

As some have noted throughout discussions here, there is no single solution to backing up your Silverblue system. I am trying to put together a short magazine article on using Borg Backup (which is available on F29 via fedora repos) as a layered package. What I am wanting to find out, and am asking for some suggestions and or links to documentation about the following

1. Real paths to directory locations, ie the difference between what the user and a layered application would “see” as the path on Silverblue.
2. Paths to, and are there metadata on the layered applications (ie version, etc…)
3. Paths to, and are there metadata on the flatpaks installed

Possibly, the flatpaks can be done by creating a script to make a human readable file which lists the apps installed in ~/.var/apps/ and just back that list up.
I know there is a web based GUI available for Borg Backup, and there is a script wrapper available and there is Vorta (3rd party, not open?).
It is a pretty interesting backup tool and I think it should be highlighted for the Silverblue community as a potential solution to backing up their system. I am well aware of Deja-Dup, and this is not a voting topic on which is better, I’ll leave each to determine that on their own. What I am asking is are you familiar with the directory structure of Silverblue and more pointedly the places information is stored that could be backed up to bring a system back to a known/desired state, layering, flatpaks, and all.

[rugk]

See also the general question:

How are we supposed to handle/do backups?

Use case: I want to have a backup of my “whole” system, i.e. nearly (ideally) all config and user’s data files, so I can easily restore it in case of a failure. What I see: I don’t need to backup the distro’s file/rpm-ostree obviously. As that can easy be restored and then likely also all packages installed. What I don’t know: However, how to backup the data/config files of all apps and your own personal data? The underlying issue may be that I can hardly install any command-line (backup) to…

IMHO your thread here is a duplicate of the one there, as your Silverblue-specific questions are hardly specific to Borgbackup. And paths etc. are needed by all backup tools.

So about the borgbackup specific things:

@jakfrost

Vorta (3rd party, not open?)

Kinda 3rd party, but with involvement of original borgbackup authors. Also, it is FLOSS:

borgbase/vorta

@jakfrost

What I am asking is are you familiar with the directory structure of Silverblue and more pointedly the places information is stored that could be backed up to bring a system back to a known/desired state, layering, flatpaks, and all.

And that is exactly my intend of How are we supposed to handle/do backups? as well. As it is obviously needed for all backup tools, not borgbackup.

@jakfrost

and there is a script wrapper available

There are many available…
But I am possibly a little biased here…

[jakfrost (Stephen Snow)]

I started this thread for the info I want for the mag article. I am specifically focusing on borg since that was the intent of the article I am writing. Whether such a question is limited to the one thread or not is superfluous to the question itself. As for the details, if my thread is going to cull responses that are beneficial to answering both threads question(s) then what difference does it make other than “my thread is superior to your thread” type of attitude? I knew about your thread, and commented on it too, but your question is not my question.

[refi64 (Ryan Gonzalez)]

Real paths to directory locations, ie the difference between what the user and a layered application would “see” as the path on Silverblue.

Do you mean the difference between what a user and a Flatpak would see? Layered applications would be the same.

Paths to, and are there metadata on the layered applications (ie version, etc…)

You can use the OSTree APIs to inspect the refs prefixed with rpm-ostree at /ostree/repo.

Paths to, and are there metadata on the flatpaks installed

More OSTree APIs: you can check the refs and the metadata files for /var/lib/flatpak/repo and ~/.local/share/flatpak/repo.

[jakfrost (Stephen Snow)]

Do you mean the difference between what a user and a Flatpak would see? Layered applications would be the same.

I guess I’m more referring to the fact most of the deduping backup programs, and borg is no exception, don’t use symlinks normally, even hard links, they want the real path.

You can use the OSTree APIs to inspect the refs prefixed with rpm-ostree at /ostree/repo.

Okay, that is beyond what I was trying for, but I will certainly look at that. I’m trying to write a pretty light overview of one backup tool available for Silverblue. Since Deja-Dup is now a layered package, it can benefit from whatever I can gather together regarding howto locate and what, but I am only focused on borg backup for this article mainly since it is an alternative backup tool that doesn’t seem to get a load of exposure. Plus, as noted above, this is very similar to the thread @rugk links to. Which also mentions Borg as well as Deja-Dup.

More OSTree APIs: you can check the refs and the metadata files for /var/lib/flatpak/repo and ~/.local/share/flatpak/repo.

Okay that is pretty much what I mean I think. I’ll browse through my directories.

[refi64 (Ryan Gonzalez)]

I guess I’m more referring to the fact most of the deduping backup programs, and borg is no exception, don’t use symlinks normally, even hard links, they want the real path.

I mean, if they go to copy a hardlink, they’ll get the original file anyway…

Okay, that is beyond what I was trying for, but I will certainly look at that.

AFAIK rpm-ostree status gets the list of packages in the currently deployment via D-Bus:

$ busctl get-property org.projectatomic.rpmostree1 /org/projectatomic/rpmostree1/fedora_workstation org.projectatomic.rpmostree1.OS DefaultDeployment
...
"requested-local-packages" as 5 "rpmfusion-nonfree-release-29-0.5.noarch" "bluecap-0.3-1.fc29.x86_64" "bat-0.9.0-1.fc30.x86_64" "rpmfusion-free-release-29-0.5.noarch" "refind-0.11.3-1.x86_64"
...
"rpmostree.packages" as 38 "ImageMagick" "WoeUSB" "adapta-gtk-theme" "ansible" "bsdtar" "evtest" "fd-find" "fedora-toolbox" "ffmpeg" "file-roller" "git" "git-lfs" "git-subrepo" "gnome-boxes" "gnome-tweaks" "google-roboto-fonts" "google-roboto-mono-fonts" "gparted" "htop" "kubectl" "libXScrnSaver" "neofetch" "powerline-fonts" "rclone" "ripgrep" "simple-scan" "sshfs" "systemd-container" "testdisk" "thai-scalable-purisa-fonts" "tilix" "tuned" "unrar" "virt-manager" "vulkan-tools" "xclip" "xdotool" "zsh"

However, as you can see, this doesn’t have version information.

ostree refs rpm-ostree/pkg (the repo defaults to /ostree/repo) will print all the layered packages and tehri versions, but it doesn’t distinguish between which versions are where, e.g.:

$ ostree refs rpmostree/pkg | rg kubectl
kubectl/1.13.3-0.x86__64
kubectl/1.13.4-0.x86__64

AFAIK this information isn’t stored anywhere; since once a deployment is created, it will never be modified, rpm-ostree never has needed this.

Okay that is pretty much what I mean I think. I’ll browse through my directories.

Some more info:

ostree refs --repo=/var/lib/flatpak/repo deploy (or --repo=~/.local/share/flatpak/repo for the user install) will list all the deployed refs:

...
runtime/org.gnome.Lollypop.Locale/x86_64/stable
runtime/org.audacityteam.Audacity.Codecs/x86_64/stable
runtime/org.gtk.Gtk3theme.AdwaitaRefresh/x86_64/3.22
app/org.gnome.Lollypop/x86_64/stable
runtime/org.gnome.Sdk/x86_64/3.30
runtime/org.freedesktop.Platform.Compat.i386/x86_64/18.08
app/org.gnome.BoxesDevel/x86_64/master
....

and then you can get the metadata for one via ostree --repo=/var/lib/flatpak/repo cat my-ref metadata, e.g.:

$ ostree --repo=/var/lib/flatpak/repo cat app/org.gnome.BoxesDevel/x86_64/master metadata
[Application]
name=org.gnome.BoxesDevel
runtime=org.gnome.Platform/x86_64/master
sdk=org.gnome.Sdk/x86_64/master
tags=devel;development;nightly;
command=gnome-boxes

[Context]
shared=network;ipc;
sockets=x11;wayland;pulseaudio;
devices=all;
filesystems=xdg-run/dconf;~/.config/dconf:ro;home;

[Session Bus Policy]
ca.desrt.dconf=talk
org.freedesktop.Tracker1=talk
org.freedesktop.timedate1=talk
org.freedesktop.Accounts=talk

[Environment]
DCONF_USER_CONFIG_DIR=.config/dconf

[Extension org.gnome.BoxesDevel.Locale]
directory=share/runtime/locale
autodelete=true
locale-subset=true

[Extension org.gnome.BoxesDevel.Debug]
directory=lib/debug
autodelete=true
no-autodownload=true

[Build]
built-extensions=org.gnome.BoxesDevel.Locale;org.gnome.BoxesDevel.Debug;

If you go the directory-walking route, you can check /var/lib/flatpak/app and /var/lib/flatpak/runtime for the different directories (or changed paths for your local install). For apps, you can then navigate to current/active to grab the deployment for the active commit, and then read the metadata file inside. For runtimes, you have go to whatever-architecture/whatever-version, e.g. x86_64/18.08 for the fd.o platform.

[otaylor (Owen Taylor)]

To me, the main thing that is different with Silverblue is that both podman and flatpak are going to be putting data in “unexpected” places within the home directory, so it may be hard to find everything to exclude. Some things that might be typically excluded:

podman images and containers: ~/.local/share/containers
Application caches: ~/.var/app/*/cache/
GNOME Boxes vms: /var/home/otaylor/.var/app/org.gnome.Boxes/data/

[jakfrost (Stephen Snow)]

AFAIK rpm-ostree status gets the list of packages in the currently deployment via D-Bus:

AFAIK this information isn’t stored anywhere ; since once a deployment is created, it will never be modified, rpm-ostree never has needed this.

So the info is there, just not maybe as apparent to extract. I understand where rpm-ostree would actually not have to care per se` about a commit since once it’s built, that’s pretty much that until the next update.

To me, the main thing that is different with Silverblue is that both podman and flatpak are going to be putting data in “unexpected” places within the home directory, so it may be hard to find everything to exclude.

Yeah, I was looking at my home directory tree, even at 3 levels deep it goes on forever. Flatpaks sure do carry a lot of baggage.
I was doing a bit of playing with Borg 1.1.8 which is in the Fedora 29 repo. It does lend itself to scripting, plus there is the option of borgmatic from torsion.org that is a python script wrapper for automating the backup/prune/verify tasks. Thank you both for your feedback, I certainly have enough ideas to write a basic article, and I may bug you two some more as I work through things.