Skip to content

Backups ​

The operator's runbook

gammonchain runs as one hosted service at gammonchain.com; the server's source is not published and there is no self-hosted edition. This section documents how that deployment is built and run — useful if you operate it, or if you want to see exactly what the service does with your data.

What needs backing up, and why ​

datacan you rebuild it?
records/yes: immutable, write-once files. Copy them anywhere and sync them back
server-key.jsonno: if it changes, every existing record stops verifying against the published key
gammonchain.dbno

Losing the database locks out every registered player permanently

gammonchain.db holds registered players' keys, their email addresses, ratings, and live sign-in tokens. For an account, the stored key is the identity. Lose the file and every registered player is locked out for good. They cannot re-register, because the key is gone.

Three layers, in order of what they survive ​

layersurvivesneeds
snapshots beside the database (/data/backups, last 14)a bad deploy, a bad migration, a delete that should not have happenednothing — always on
a pulled copy on your own machine (deploy/pull-backup.sh)the server, the disk, the provideryou, running one command
offsite uploads (R2, B2, S3)everything, unattendeda bucket and four variables

Only the third is automatic and off the box. Until it is configured, run the second one regularly — before any risky deploy, and otherwise every few weeks:

bash
SSH_KEY=~/.ssh/hetzner ./deploy/pull-backup.sh gammon@<your server>

It asks SQLite for a consistent snapshot, brings back the database, the records and the signing key, and then opens the snapshot to prove it restores. A backup nobody has opened is a hope, not a backup. The archive contains the signing key, so it lands readable only by you: keep it off shared drives and out of the repository.

Restoring ​

bash
tar xzf gammonchain-<stamp>.tar.gz                 # pull-snapshot.db, records/, server-key.json
scp -i ~/.ssh/hetzner pull-snapshot.db gammon@<server>:/tmp/
ssh -i ~/.ssh/hetzner gammon@<server> '
  cd /opt/gammonchain/deploy
  docker compose -f docker-compose.prod.yml stop server
  docker compose -f docker-compose.prod.yml run --rm -T -v /tmp:/restore server \
    sh -c "cp /restore/pull-snapshot.db /data/gammonchain.db && rm -f /data/gammonchain.db-wal /data/gammonchain.db-shm"
  docker compose -f docker-compose.prod.yml start server'

Delete the -wal and -shm files with the database they belonged to: left behind, SQLite will try to replay a write-ahead log against a file that never saw those writes.

The server key and records are ordinary files on the same volume; restore them the same way if they are lost.

Offsite configuration ​

Uploads are off unless all four of endpoint, bucket, key and secret are set (the local snapshots above run regardless):

bash
BACKUP_S3_ENDPOINT=https://<account>.r2.cloudflarestorage.com
BACKUP_S3_BUCKET=gammonchain-backups
BACKUP_S3_KEY=...
BACKUP_S3_SECRET=...
BACKUP_S3_REGION=auto            # R2 wants "auto"; B2/AWS want a real region
BACKUP_INTERVAL_H=6              # hours between snapshots (minimum 1)
BACKUP_PREFIX=gammonchain        # object key prefix

DEPLOY.md suggests Cloudflare R2 (10 GB free, no egress fees) or Backblaze B2 (10 GB free). Both speak the S3 API. Changing provider means changing the endpoint and credentials.

With backups unset, nothing runs: no timer and no warning on every boot. /healthz reports backup.configured: false, so the state is visible instead of assumed.

With deploy/docker-compose.prod.yml, all of these (including BACKUP_INTERVAL_H and BACKUP_PREFIX) are read from deploy/.env.

How it works ​

A consistent snapshot: VACUUM INTO ​

Copying a .db file while the server is running is the classic way to get a backup that restores into a corrupt database. SQLite may be mid-transaction, and the -wal file holds pages the main file does not have yet. Instead, the backup runs

sql
VACUUM INTO '/tmp/gc-backup-<pid>-<run>.db'

This asks SQLite itself for a clean, complete copy at a single point in time. Under WAL mode it does not block writers. The temporary file is deleted afterwards whether the upload succeeds or not.

Upload: hand-written SigV4 ​

The snapshot is uploaded with a single PUT signed with AWS Signature Version 4. signV4() is about sixty lines built on node:crypto, which the code comment considers a much smaller liability than an SDK dependency tree. The details that are easy to get wrong are all explicit:

  • the payload hash is computed over the actual body (S3 verifies it);
  • headers are signed in lower-case, sorted order: content-type, host, x-amz-content-sha256, x-amz-date;
  • the credential scope date matches x-amz-date to the day;
  • the object path uses path-style addressing: <endpoint>/<bucket>/<key>.

Objects are named <prefix>/<ISO timestamp with : and . replaced by ->.db, so the bucket lists in date order and a lifecycle rule can expire old snapshots by prefix. The server never deletes old backups. Set up a lifecycle rule in the bucket if you want retention.

Schedule and failure reporting ​

  • One backup runs immediately at startup, so a misconfiguration shows up at deploy time, not six hours later. After that, one runs every BACKUP_INTERVAL_H hours.
  • A failed backup never throws. It records lastError, sends the error to the error reporter (kind: backup), and logs [backup] FAILED.
  • /healthz → backup shows { configured, intervalH, runs, lastOk, lastError }.

After deploying

Check that /healthz shows backup.configured: true, and a few minutes later that backup.lastOk is set. A backup that silently fails is exactly the kind of thing nobody notices until the day it matters.

Restoring ​

The repository has no restore script. A snapshot is an ordinary SQLite database file. To restore, stop the server, put the downloaded snapshot at <DB_DIR>/gammonchain.db (with no stale -wal/-shm files next to it), and start the server again.

Rules engine, fairness protocol, verifier, analysis and worker: MIT. Server and client: AGPL-3.0-or-later.