Deploy to a VPS

This page walks through the path deploy/ automates: one Hetzner box running Caddy, the game server and an analysis worker in Docker, with gammonchain.com proxied through Cloudflare. Nothing here needs a git remote, a container registry or CI. The source is copied over SSH and the image is built on the box.

laptop ── rsync over SSH ──► /opt/gammonchain on the box
                               docker compose (deploy/docker-compose.prod.yml)
                                 ├── caddy    binds 80/443, TLS, routes by hostname
                                 ├── server   expose 8080 on the `edge` network only
                                 └── worker   gammonet with gnubg

1. Create the server

In the Hetzner Cloud console: New project → Add server. The choices recommended in DEPLOY.md:

field	choose	why
Location	Falkenstein / Nuremberg / Helsinki (EU), Ashburn / Hillsboro (US)	latency to your players
Image	Ubuntu 24.04 LTS	deploy/setup.sh targets Debian/Ubuntu
Type	Shared vCPU → x86 → CX22	2 vCPU, 4 GB, 40 GB: the safe default
Volumes	none	40 GB is already far more than the records need
Firewall	create one (step 2)	the layer Docker cannot bypass
SSH key	paste your public key	never enable password login
Backups	optional, +20%	the app also backs up its database to S3-compatible storage

You need an IPv4 address. Hetzner bills it separately.

2. Add Hetzner's Cloud Firewall

Create a Cloud Firewall and attach it to the server. Allow inbound 22/tcp, 80/tcp, 443/tcp and 443/udp (HTTP/3), and nothing else. This matches deploy/ports.conf. Why both layers are needed is explained on the Firewall page.

3. Provision the box

Run once, as root, from your laptop:

ssh root@<ip> 'bash -s' < deploy/setup.sh

setup.sh does the following, in order:

1. Packages. Installs ca-certificates curl git ufw unattended-upgrades and checks that ufw and unattended-upgrades really installed. On Ubuntu 26.04, ufw once reported "config-files" (present in dpkg but not installed) and the failure only showed up later.
2. Docker, through the official get.docker.com script if it is missing.
3. The shared edge Docker network, created once. Caddy and every project's containers join it.
4. An unprivileged user (gammon by default, or APP_USER), added to the docker group.
5. The firewall. Runs deploy/firewall.sh if it is already on the box, and installs a systemd unit, gammon-firewall.service, that re-applies it after docker.service on every boot (skipped while the file does not exist).
6. SSH hardening (below).
7. Automatic security updates with unattended-upgrades.
8. /opt/gammonchain, owned by the app user. If you set REPO_URL, it clones that repository there. Otherwise it leaves the directory empty for push.sh.

On a fresh box the firewall is not active yet after setup.sh

Without REPO_URL the source is not on the box when setup.sh runs, so step 5 prints SKIPPED: /opt/gammonchain/deploy/firewall.sh is not on the box yet with the command to run, and carries on. The systemd unit is enabled but not started, so the container firewall would otherwise only take effect at the next boot. After your first push, run it by hand (or re-run setup.sh):

ssh root@<ip> 'bash /opt/gammonchain/deploy/firewall.sh'

SSH hardening, and the guards against locking yourself out

Port 22 is open to the internet and scanned constantly. With a key already on the box, password authentication only adds a brute-force surface.

• Copy the key first. /root/.ssh/authorized_keys is copied to the app user before anything is disabled. If root has no authorized_keys, the script refuses to disable password authentication and tells you to add a key and re-run.

• Then harden. Only if the app user now has a non-empty authorized_keys, it writes /etc/ssh/sshd_config.d/99-hardening.conf:

  PasswordAuthentication no
  KbdInteractiveAuthentication no
  PermitRootLogin prohibit-password

  Root can still log in, but only with a key.

• Validate before reloading. sshd -t checks the config. If it fails, the file is removed and SSH is left as it was.

Both guards exist because the failure they prevent, locking yourself out of a box you have no console for, cannot be fixed over the same SSH connection you just broke.

4. Put the secrets on the server

Secrets live on the server only:

ssh gammon@<ip> 'cp /opt/gammonchain/deploy/.env.example /opt/gammonchain/deploy/.env'
ssh gammon@<ip>          # then edit /opt/gammonchain/deploy/.env

deploy/.env.example lists what to fill in: PUBLIC_URL, the Mailjet key pair and sender, MAIL_DAILY_CAP, CRON_KEY and GAMMONET_KEY (long random strings, for example openssl rand -hex 24), error reporting, and the BACKUP_S3_* settings.

If you push before creating the file, push.sh stops with instructions. The .env file is not in the repository, and push.sh never overwrites it.

deploy/docker-compose.prod.yml loads the whole file into the server with env_file: .env, so every variable you set there reaches it, including SENTRY_DSN, STRICT_ANALYSIS, MAIL_HOUR, LIMIT_*, RELEASE, BACKUP_INTERVAL_H and SERVER_KEY_PKCS8. A few values in its environment: block take precedence over .env: DATA_DIR=/data, TRUST_PROXY=1 and DEEP_ENGINE=gnubg-1.07 are fixed; PUBLIC_URL, BOT_ENGINE, MAIL_PROVIDER, MAIL_FROM, MAIL_DAILY_CAP, ERROR_PROVIDER and BACKUP_S3_REGION take their value from .env and fall back to a default when unset.

The worker does not get the whole file: it receives only GAMMONET_KEY and GAMMONET_NAME, so the mail, backup and signing-key secrets stay out of it.

5. Deploy from your laptop

./deploy/push.sh gammon@<ip>
SSH_KEY=~/.ssh/hetzner ./deploy/push.sh gammon@<ip>     # pin the key to use

push.sh:

1. rsyncs the working directory to /opt/gammonchain (APP_DIR) with --delete. It never copies .git, node_modules, any .env or .env.* at any depth (except .env.example), data/, projects/*/data/, records/, annotations/, analysis-cache/, *.db (and -shm/-wal/-journal), or server-key.json; excluded files are also never deleted. Those files live on the server. Overwriting the real .env or the production signing key from a laptop would be the two worst things a deploy script could do. --delete is also forbidden from removing anything under projects/ or any deploy/sites/*.caddy, so other projects on the box survive, and the script refuses an APP_DIR that is /, a top-level directory or a home directory.
2. Checks that deploy/.env exists on the server.
3. Builds and restarts: docker compose -f deploy/docker-compose.prod.yml --env-file deploy/.env up -d --build.
4. Reloads Caddy (caddy reload). Caddy's config is bind-mounted, so editing Caddyfile or sites/*.caddy does not change the container definition, and up -d leaves the old config running. The script's comment records that this once cost an hour chasing a Cloudflare 525 error that was really Caddy never having read the fix.
5. Waits up to 60 seconds for /healthz at the PUBLIC_URL read from the server's .env, then prints the response.

Re-run it for every change. rsync sends only what changed, so later deploys take seconds.

Why SSH_KEY exists

On a machine with many keys in ~/.ssh, ssh can hit the server's MaxAuthTries before it offers the right key. The result is a bare Permission denied (publickey) from inside rsync. SSH_KEY passes -i <key> -o IdentitiesOnly=yes.

If you prefer git: set REPO_URL when running setup.sh, then deploy with

ssh gammon@<ip> 'cd /opt/gammonchain && git pull && \
  docker compose -f deploy/docker-compose.prod.yml --env-file deploy/.env up -d --build'

6. TLS: Caddy behind Cloudflare

The committed site file, deploy/sites/gammonchain.caddy, is written for gammonchain.com, which Cloudflare proxies with SSL mode Full:

gammonchain.com, www.gammonchain.com {
	encode zstd gzip
	tls internal
	reverse_proxy server:8080 {
		transport http {
			dial_timeout 5s
		}
	}
	log { ... }
}

• tls internal: Caddy serves a certificate from its own local CA. Cloudflare's Full mode connects to the origin over HTTPS but does not verify the certificate, so this is enough. It also needs no ACME challenge, which the proxy was blocking.
• WebSockets need no configuration. Caddy proxies the upgrade and imposes no idle timeout on a proxied connection. A 9-point match can sit quiet for minutes while someone thinks, and a proxy that closes idle connections at 60 s would drop players mid-game.
• X-Forwarded-For is left at Caddy's default, which appends the immediate peer. An earlier version set header_up X-Forwarded-For {remote_host}, which replaces the header. Behind Cloudflare, {remote_host} is a Cloudflare edge address, so every client on the internet would have shared a handful of rate-limit buckets. With the default, Cloudflare's original client address stays the left-most entry, which is what the server reads when TRUST_PROXY=1.
• dial_timeout 5s makes a dead backend fail quickly with a 502 instead of hanging a browser tab.

Full encrypts, but it does not authenticate the origin

Cloudflare Full does not check the origin's certificate. The comment in the site file suggests upgrading to a Cloudflare Origin Certificate with Full (strict), which needs no renewals (those certificates last 15 years).

Not behind Cloudflare?

tls internal certificates are not trusted by browsers. If DNS points straight at the box, remove the tls internal line so that Caddy obtains a Let's Encrypt certificate automatically. In that case, point the domain at the box before starting Caddy, because Let's Encrypt will not issue a certificate until DNS resolves to it.

The app sets CSP (including frame-ancestors), X-Frame-Options, X-Content-Type-Options and Referrer-Policy itself, so the site file does not repeat them. Nothing in the stack sets Strict-Transport-Security; enable it at Cloudflare if you want it. See Security model.

7. Check it

curl -s https://gammonchain.com/healthz

Then work through the checklist from DEPLOY.md:

• [ ] /healthz returns ok: true and the botEngine you expected
• [ ] PUBLIC_URL exactly matches the live URL, scheme included
• [ ] TRUST_PROXY=1 if anything terminates TLS in front of the process (the prod compose file sets it)
• [ ] /healthz → backup.configured: true and, after a few minutes, backup.lastOk is set
• [ ] /healthz → errors.configured: true for the provider you chose
• [ ] a real registration delivers a link that signs you in
• [ ] rate limits work: for i in $(seq 20); do curl -so/dev/null -w"%{http_code} " URL/api/analysis/x; done shows 429 from the 16th request (the analysis burst is 15)
• [ ] /privacy.html and /terms.html load and name you as the operator
• [ ] deleting a test account really removes it
• [ ] two browsers on different networks can find each other in the lobby
• [ ] a match survives 10 minutes (proxy idle timeouts are the usual culprit)
• [ ] records and server-key.json survive a redeploy
• [ ] POST /api/cron/daily?key=… returns a result, if you rely on external cron