Borg backups for a Docker host: pushing encrypted, deduplicated backups to a second machine

How I set up automated backups of a Debian 13 Docker host (GitLab, Gitea, Firecrawl and its postgres/rabbitmq/redis, a CI runner, a private registry) to a Proxmox machine on the same LAN, using Borg over SSH.

The finished setup:

Step 0: recon first — what is actually on this machine?

The plan started as "back up /etc and /home, the containers live there anyway." Recon said otherwise, in three findings:

1. Docker volumes don't live in /home. They live under /var/lib/docker/volumes. On this box that meant GitLab's repositories and internal database, the CI runner config, the private image registry, and Firecrawl's postgres and rabbitmq data — all invisible to the original plan. Only Gitea (bind-mounted from /home) would have been covered.
2. Several of those volumes are live databases. Copying a database file mid-write gives you a crash-consistent copy at best. Postgres in particular may refuse to start from it. A real backup needs consistent logical dumps.
3. One file was 29 GB: GitLab's docker json log. Docker does no log rotation out of the box. Left alone it would have poisoned the backup repo and eventually the disk. (Fix at the end of this post.)

So the data set became: /etc, /home, /var/lib/docker/volumes, /var/spool/cron, plus a staging directory of consistent dumps regenerated before every run:

• postgres: pg_dumpall — whole cluster including roles
• rabbitmq: rabbitmqctl export_definitions — users, vhosts, queues, bindings
• SQLite databases (gitea, hermes state, tv_schedule): sqlite3 .backup — online-safe consistent copies
• GitLab: the official gitlab-backup create tar (repos + db + uploads)
• docker inspect of every container, images/volumes/networks lists, dpkg --get-selections — enough to rebuild the machine

Step 1: choosing the transport — native Borg over SSH

Three options were on the table:

• sshfs / SFTP-mounted "local" repo — rejected. Borg needs real POSIX locking and atomic renames; FUSE filesystems break both. It usually works, until it corrupts a repo mid-transaction.
• Local repo, rsynced to the server afterwards — rejected. You copy a repo that may be mid-write unless you add locking yourself, and borg compact rewrites object files in ways rsync handles poorly. Reimplementing what Borg does natively, worse.
• Borg over SSH (borg serve) — chosen. Borg on the client does all the chunking, compression and encryption before sending; the server just writes opaque chunks. The server needs nothing but sshd and the borg binary. After the first full archive, nightly traffic is only changed chunks.

Version pairing: the client is 1.4.0, the server 1.2.4 (what Debian bookworm ships). Rather than argue about compatibility matrices, I tested it — a throwaway repo on the server, then init → create → list → extract → verify → delete from the client:

borg init --encryption=repokey-blake2 ssh://[email protected]/mnt/disk2/borgtest
echo "compat test" > /tmp/borgtest-src/file.txt   # (dir /tmp/borgtest-src)
borg create ssh://[email protected]/mnt/disk2/borgtest::t1 /tmp/borgtest-src
borg list --short ssh://[email protected]/mnt/disk2/borgtest
cd /tmp/borgtest-out && borg extract ssh://[email protected]/mnt/disk2/borgtest::t1
diff -r /tmp/borgtest-src /tmp/borgtest-out/tmp/borgtest-src   # clean
borg delete ssh://[email protected]/mnt/disk2/borgtest

Roundtrip verified, repo removed. Two minutes of testing beats any amount of documentation archaeology.

Step 2: encrypting, or not

The backup contains /home (API keys, ~/.ssh), /etc/shadow, host keys, and every GitLab/Gitea repository. So yes — repokey-blake2:

• The client encrypts before sending; the server stores ciphertext only. A compromised server leaks nothing usable.
• Performance cost is unmeasurable; the client does the work anyway.
• The cost is secret management: losing the passphrase loses the backup. Mitigation: passphrase lives in /etc/borg/backup.env (mode 600) on the client, one copy in the password manager, plus borg key export output stored off-machine.

If your threat model says "the backup server is exactly as trusted as the client and I refuse to manage a secret", Borg also offers authenticated (integrity without confidentiality) and none. For a whole-machine backup with this many credentials inside, I consider encryption the only sane default — but it is a choice, and Borg supports all three.

Step 3: server side — three things, once

The server already had borg (from Debian). If yours doesn't:

apt install borgbackup

3.1 — a borg-only wrapper. command= restrictions in authorized_keys don't block arguments, so don't restrict to borg serve directly — clients can pass options to it. A wrapper pins the restriction:

BORGBIN=$(command -v borg)
mkdir -p /etc/borgbackup
printf '#!/bin/sh\nexec %s serve --restrict-to-path=/mnt/disk2/borg "$@"\n' \
    "$BORGBIN" > /etc/borgbackup/serve-wrapper.sh
chmod 755 /etc/borgbackup/serve-wrapper.sh

The client can then append its own --restrict-* options (harmless), but the path restriction is enforced server-side no matter what the client asks for.

3.2 — a dedicated, born-restricted SSH key. Don't reuse your daily-driver key: a command= prefix would then apply everywhere that key authenticates. Generate a dedicated pair on the client (as root, since the backup runs as root):

ssh-keygen -t ed25519 -f /root/.ssh/id_ed25519_borg -N "" -C "borg@node8"

Add the public key on the server with the restriction prefix. On a Proxmox host, root's authorized_keys is a symlink into the cluster filesystem — edit the real file:

# /etc/pve/priv/authorized_keys
command="/etc/borgbackup/serve-wrapper.sh",no-pty,no-X11-forwarding,no-port-forwarding,no-agent-forwarding ssh-ed25519 AAAA... borg@node8

Now even a fully compromised client can do exactly one thing: talk borg, inside /mnt/disk2/borg.

Optional hardening for the paranoid (in a good way): add --append-only to the wrapper's borg serve call. The repo then can't be pruned or deleted from the client — ransomware on the client can't destroy history. The tradeoff: retention/pruning stops working over the network and needs an occasional server-side borg compact. I chose the simpler path because the server is mine and ZFS snapshots cover the "oops" cases; on shared or less trusted infra, append-only is the right call.

3.3 — the repository directory:

mkdir -p /mnt/disk2/borg/node8

That's the whole server config. No daemon, no extra port, no database.

Step 4: client side — packages, env, repo init

apt install borgbackup sqlite3
mkdir -p /etc/borg /var/backups/borg /var/backups/staging

The environment file — note the export keywords, so both the backup script and a wrapper pass these to borg properly:

# /etc/borg/backup.env
export BORG_REPO=ssh://[email protected]/mnt/disk2/borg/node8
export BORG_PASSPHRASE='<output of: openssl rand -base64 24>'
export BORG_RSH="ssh -i /root/.ssh/id_ed25519_borg -o BatchMode=yes"

chmod 600 /etc/borg/backup.env

Unattended runs need the server's host key pre-accepted for root:

ssh-keyscan -H node5.oiko6.com >> /root/.ssh/known_hosts

Excludes — the regenerable junk found during recon:

# /etc/borg/excludes
/home/dindi/.cache
/var/lib/docker/containers
/var/lib/docker/volumes/gitlab_gitlab_logs

(5.1 GB of package caches; 29 GB of docker json logs; GitLab's log volume, recreated on boot.)

Initialize:

source /etc/borg/backup.env
borg init --encryption=repokey-blake2 "$BORG_REPO"

Then immediately export the key as a disaster-recovery artifact and store it off-machine, separate from the passphrase:

borg key export "$BORG_REPO" /root/borg-repokey-backup.txt

Step 5: the backup script

/etc/borg/run-backup.sh. Phase 1 makes consistent dumps; phase 2 archives everything; phase 3 prunes. Failures degrade: a dead container logs a WARN and the rest still backs up; a borg failure skips prune (staging is kept for debugging):

#!/bin/bash
# Borg backup for node8: /etc, /home/dindi, docker volumes + consistent DB dumps.
# Config: /etc/borg/backup.env (repo + passphrase), /etc/borg/excludes
# Managed by Hermes Agent. Restore guide: /etc/borg/RESTORE.md
set -o pipefail

source /etc/borg/backup.env
STAGE=/var/backups/staging
GITLAB_BK=/var/lib/docker/volumes/gitlab_gitlab_data/_data/backups

exec > >(tee -a /var/log/borg-backup.log) 2>&1
exec 9>/var/backups/borg/.lock
flock -n 9 || { echo "another backup run is in progress, exiting"; exit 0; }

log(){ echo "[$(date '+%F %T')] $*"; }

log "=== dump phase ==="
rm -rf "$STAGE"
mkdir -p "$STAGE/docker" "$STAGE/sqlite"

# --- Firecrawl Postgres: consistent logical dump
if docker exec firecrawl-nuq-postgres-1 pg_dumpall -U postgres > "$STAGE/firecrawl-postgres.sql" 2>/dev/null; then
  log "postgres dump: ok ($(du -h "$STAGE/firecrawl-postgres.sql" | cut -f1))"
else
  log "WARN: firecrawl postgres pg_dumpall failed"
fi

# --- RabbitMQ definitions (users, vhosts, queues, bindings)
if docker exec firecrawl-rabbitmq-1 rabbitmqctl -q export_definitions /tmp/rmq-defs.json 2>/dev/null \
   && docker cp firecrawl-rabbitmq-1:/tmp/rmq-defs.json "$STAGE/rabbitmq-definitions.json" >/dev/null 2>&1 \
   && docker exec firecrawl-rabbitmq-1 rm -f /tmp/rmq-defs.json >/dev/null 2>&1; then
  log "rabbitmq definitions: ok"
else
  log "WARN: rabbitmq export failed"
fi

# --- SQLite databases: online-safe consistent copies
dump_sqlite(){
  if sqlite3 -cmd '.timeout 5000' "$1" ".backup '$2'" 2>/dev/null; then
    log "sqlite $(basename "$2"): ok"
  else
    log "WARN: sqlite backup failed: $1"
  fi
}
dump_sqlite /home/dindi/.hermes/state.db                "$STAGE/sqlite/hermes-state.db"
dump_sqlite /home/dindi/.hermes/profiles/coder/state.db  "$STAGE/sqlite/hermes-coder-state.db"
dump_sqlite /home/dindi/tv_schedule/data/sportstv.db    "$STAGE/sqlite/tv-sportstv.db"
dump_sqlite /home/dindi/gitea/data/gitea/gitea.db       "$STAGE/sqlite/gitea.db"

# --- Container/service metadata (enough to recreate containers via inspect output)
docker ps -a --no-trunc > "$STAGE/docker/ps-a.txt" 2>/dev/null
docker images --digests  > "$STAGE/docker/images.txt" 2>/dev/null
docker volume ls         > "$STAGE/docker/volumes.txt" 2>/dev/null
docker network ls        > "$STAGE/docker/networks.txt" 2>/dev/null
docker inspect $(docker ps -aq) > "$STAGE/docker/inspect-all.json" 2>/dev/null
docker system info       > "$STAGE/docker/info.txt" 2>/dev/null
dpkg --get-selections    > "$STAGE/dpkg-selections.txt" 2>/dev/null
log "docker metadata + dpkg list: ok"

# --- GitLab: official backup tool (db + repos + uploads -> tar in its volume)
if docker exec gitlab gitlab-backup create >/dev/null 2>&1; then
  log "gitlab-backup create: ok ($(ls -lh "$GITLAB_BK" 2>/dev/null | grep -c tar) tar(s))"
else
  log "WARN: gitlab-backup create failed (is gitlab running?)"
fi

log "=== borg phase ==="
borg create --verbose --stats --compression zstd,3 \
  --exclude-from /etc/borg/excludes \
  "${BORG_REPO}::{hostname}-{now:%Y-%m-%dT%H:%M:%S}" \
  /etc /home/dindi /var/lib/docker/volumes /var/backups/staging /var/spool/cron
rc=$?

[ $rc -eq 1 ] && log "WARN: borg create completed with warnings (rc=1) - continuing"
if [ $rc -ge 2 ]; then
  log "FATAL: borg create failed (rc=$rc) - keeping staging, skipping prune"
  exit 1
fi

borg prune --stats --keep-daily 7 --keep-weekly 4 --keep-monthly 6 "$BORG_REPO"
borg compact "$BORG_REPO"

# gitlab tars are now safely in the borg repo - remove from volume so it doesn't grow
rm -f "$GITLAB_BK"/*_gitlab_backup.tar "$GITLAB_BK"/backup_information.json

date '+%F %T SUCCESS' > /var/backups/borg/last-run
log "=== SUCCESS ==="

Notes on the design:

• borg's exit codes deserve their own warning. rc=0 is clean, rc=1 means "completed, with warnings" (e.g. a file changed while being read — the norm for live volumes), rc≥2 is real failure. My first version treated any non-zero as fatal and logged FATAL after a successful 10 GB archive. The correct pattern:

  borg create ...; rc=$? [ $rc -eq 1 ] && log "WARN: completed with warnings - continuing" if [ $rc -ge 2 ]; then log "FATAL (rc=$rc)"; exit 1; fi

• SQLite via .backup rather than copying the file: it's the documented online-safe copy method, and it takes the DB's own locking, so you get a consistent snapshot while the owning service keeps running.
• GitLab: gitlab-backup create writes its tar inside the data volume, which borg then archives like any other file; afterwards the script deletes the tar from the volume. Without that cleanup the volume would grow by a full backup every night. One gotcha bit me here: the volume's layout maps container /var/opt/gitlab directly onto the volume root, so the host-side path of the backups dir is /var/lib/docker/volumes/gitlab_gitlab_data/_data/backups — not _data/var/opt/gitlab/backups. Verify volume layouts with docker inspect before hardcoding paths in cleanup logic.
• The raw volumes are still archived alongside the dumps — a fallback if a logical dump turns out unusable, at the cost of some gigabytes.
• flock prevents overlapping runs (timer + manual).
• Nice=19 + best-effort I/O scheduling in the unit below keep the 02:30 run from disturbing anything.

Step 6: the daily-driver wrapper

#!/bin/bash
# Convenience wrapper: sources repo+passphrase env, then runs borg.
# Usage: sudo /etc/borg/borg-run.sh <any borg args>
source /etc/borg/backup.env
exec borg "$@"

Installed as /etc/borg/borg-run.sh, mode 700. Every interactive borg command goes through it — no memorizing repo URLs:

sudo /etc/borg/borg-run.sh list --short
sudo /etc/borg/borg-run.sh info
sudo mkdir -p /mnt/restore
sudo /etc/borg/borg-run.sh mount "$BORG_REPO" /mnt/restore   # browse, then umount

Step 7: systemd units

/etc/systemd/system/borg-backup.service:

[Unit]
Description=Borg backup (/etc, /home/dindi, docker volumes, db dumps)
After=docker.service network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/etc/borg/run-backup.sh
Nice=19
IOSchedulingClass=best-effort
IOSchedulingPriority=7

/etc/systemd/system/borg-backup.timer:

[Unit]
Description=Daily borg backup at 02:30

[Timer]
OnCalendar=*-*-* 02:30:00
RandomizedDelaySec=15m
Persistent=true

[Install]
WantedBy=timers.target

/etc/systemd/system/borg-check.service:

[Unit]
Description=Borg repository consistency check
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/etc/borg/borg-run.sh check --progress
Nice=19
IOSchedulingClass=idle

/etc/systemd/system/borg-check.timer:

[Unit]
Description=Monthly borg consistency check (1st of month, 04:00)

[Timer]
OnCalendar=*-*-01 04:00:00
RandomizedDelaySec=30m
Persistent=true

[Install]
WantedBy=timers.target

systemctl daemon-reload
systemctl enable --now borg-backup.timer borg-check.timer
systemctl list-timers | grep borg

Persistent=true means a missed run (machine off, network down) fires on next boot instead of being silently skipped.

Step 8: verify like you mean it

A backup you have never restored is a hope, not a backup. In order:

# 1. full manual run, watch both phases
sudo /etc/borg/run-backup.sh
cat /var/backups/borg/last-run        # should say SUCCESS

# 2. repo state
sudo /etc/borg/borg-run.sh list --short
sudo /etc/borg/borg-run.sh info       # size, dedup ratio, compression

# 3. extract the postgres dump from the repo and load it into a scratch
#    postgres container - proves dump + repo + restore path end to end
mkdir -p /tmp/verify && cd /tmp/verify
sudo /etc/borg/borg-run.sh extract \
  ssh://[email protected]/mnt/disk2/borg/node8::<archive> var/backups/staging
sudo chown -R $USER:$USER /tmp/verify     # extract runs as root

docker run -d --name pg-verify -e POSTGRES_PASSWORD=x postgres:17
# wait for readiness: docker exec pg-verify pg_isready -U postgres
docker exec -i pg-verify psql -U postgres -q \
  < var/backups/staging/firecrawl-postgres.sql

Two notes from doing this for real:

• pg_dumpall output starts by recreating bootstrap objects (role postgres, the postgres database) that any fresh cluster already has — with ON_ERROR_STOP=1 the restore dies at line 16 on "role postgres already exists". Load without it; the expected errors are exactly those already-exists lines. My restore also produced a second, content class of errors: the live cluster runs the pg_cron extension, which stock postgres:17 doesn't ship — everything cron.* failed. The lesson for real restores: match the target image to the source's extensions (the firecrawl nuq-postgres build would have restored it cleanly), or read the error list expecting extension-shaped holes.

• borg extract recreates paths as they were archived, root-owned; chown the tree before your next step.

  4. GitLab: extract the backup tar, then inside the gitlab container:

  gitlab-ctl stop puma && gitlab-ctl stop sidekiq

  gitlab-backup restore BACKUP=

  (full sequence lives in /etc/borg/RESTORE.md)

  5. prune/compact against the 1.2 server also gets exercised by the

  manual run in step 1 - watch for it in the log

The machine also keeps a restore guide at /etc/borg/RESTORE.md — writing it was part of the install, not an afterthought. Restore knowledge that lives only in the installer's head doesn't survive contact with a real incident.

Postscript: fix the 29 GB log, or the backup stays fat

Docker ships with no log rotation. One GitLab container, five weeks, 29 GB of json log. The backup excludes it, but the right fix is the daemon:

truncate -s 0 /var/lib/docker/containers/<id>/<id>-json.log
cat > /etc/docker/daemon.json <<'EOF'
{
  "log-driver": "json-file",
  "log-opts": { "max-size": "10m", "max-file": "3" }
}
EOF
systemctl restart docker

(The restart bounces all containers — maintenance window.)

What this setup deliberately is not

• Not fire-and-forget portable: the SSH key, repo path and container names are baked into config files. That's fine — this is a single-machine setup, and explicit beats clever.
• Not immutable/offline: the client can rewrite the repo (prune). For ransomware-proof history use --append-only on the server (Step 3.2) and lose automatic pruning.
• Not tested against 3-2-1: two machines, one LAN, one physical location. Off-site (rsync.net, a borg repo on a VPS, an USB disk in another building) is the natural next step — the same client config, a second repo.

Lessons worth keeping

1. Docker volumes live in /var/lib/docker/volumes, not in your compose directory. Recon before planning; the naive plan backed up almost none of the actual data.
2. Live databases need logical dumps; raw volume copies are a fallback, not a restore plan.
3. Version compatibility between borg client and server is a five-minute experiment, not a documentation research project. Throwaway repo, roundtrip, delete.
4. A dedicated SSH key born with command="borg serve --restrict-to-path=..." costs two minutes and removes the client-as-root-on-server risk entirely.
5. Docker's default log config will fill any disk. Check yours today.
6. The restore guide is part of the install. Untested restores are hopes.

Appendix: inventory

First archive (real numbers from this machine): 10.15 GB original, 137,914 files, 4.48 GB after zstd-3, 4.01 GB on disk after dedup — just under five minutes over the LAN. The second archive of the same data set, one hour later: 32 MB and 34 seconds. That ratio is the entire point of borg. Retention: 7 daily, 4 weekly, 6 monthly — a full month of history in roughly the size of two full archives, thanks to dedup.

Appendix: the passphrase

repokey mode stores the (passphrase-protected) key inside the repository. Lose the passphrase and the backup is unrecoverable — that is the deal you accept for at-rest encryption. So: passphrase in /etc/borg/backup.env (mode 600) on the client, a copy in the password manager, and borg key export output stored off-machine, separate from the passphrase. Three artifacts, two locations, zero excuses.