Monitoring

/healthz

GET /healthz is the health check. The Dockerfile's HEALTHCHECK calls it every 30 seconds. It always returns 200 while the process is serving, with these fields:

field	meaning
ok	always true
uptime	seconds since the process started
online	connected WebSocket sessions
matches	live match actors
rssMB, heapMB	memory, in MB
botEngine	the engine the hard bot actually got: gnubg-1.07, a BGBlitz id, or builtin-heuristic-1
dice	the dice protocol, hmac-sha256/hashchain-reveal-v2
serverKey	the server's Ed25519 public key, published so anyone can pin it
queue	analysis jobs waiting for a worker
errors	{ provider, configured, sent, dropped, failed } from the error reporter
backup	{ configured, intervalH, runs, lastOk, lastError }
limiterKeys	number of rate-limit buckets in memory

Check botEngine, errors and backup after every deploy

All three can be "configured" and quietly not working. If botEngine says builtin-heuristic-1, your "hard" bot is the heuristic. If errors.failed is climbing, your error sink is rejecting reports. If backup.lastError is set, your backups are not happening. DEPLOY.md says these fields are exposed specifically so a deploy check can catch "configured but silently failing", which otherwise looks exactly like "nothing has gone wrong".

GET /gammonet/status reports the analysis queue in more detail: depth, jobs queued, served, dropped and rejected, cache hit rate, and annotation store statistics.

Error reporting

src/server/errors.js works the same way as the mailer: provider-agnostic, over plain fetch, with no SDK. The code comment points out that an SDK would pull in a dependency tree larger than the server.

ERROR_PROVIDER=console            # default: structured JSON on stderr, nothing sent
ERROR_PROVIDER=sentry   SENTRY_DSN=https://<key>@<host>/<project>
ERROR_PROVIDER=webhook  ERROR_WEBHOOK=https://…   # Discord, Slack, anything
RELEASE=v1.2.3                    # attached to reports (default "dev")

provider	sends to
console	nowhere. Errors are written to stderr as JSON
sentry	Sentry's store endpoint (/api/<project>/store/), built from the DSN, authenticated with X-Sentry-Auth
webhook	a JSON POST with the same text in both content (Discord) and text (Slack)

The environment reported is NODE_ENV (production in the Docker image).

What gets reported:

• every uncaught error in an HTTP route. The client gets a bare 500 {"error":"internal error"}, because the message may quote a path, a token or an address, and that belongs in the report, not the reply;
• unhandled promise rejections;
• uncaught exceptions. These are reported, and then the process exits after 250 ms so the supervisor restarts it. A clean restart is better than a process limping on with corrupt state;
• failed backups.

Always on stdout first

Every error goes to stderr before anything is sent anywhere. A remote sink that is down, rate-limited or misconfigured must never be the reason an incident left no trace. capture() never throws and never rejects.

Scrubbing

Before anything leaves the process, both message and stack go through scrub():

pattern	replaced with
anything that looks like an email address	<email>
any hex string of 32+ characters (sign-in tokens, unsubscribe tokens, seeds, chain links)	<hex>

A stack trace that quotes a sign-in token would otherwise hand the reader a live credential. The scrubbing also applies to the local stdout line. The context object is not scrubbed. The server only passes route kind, method and path in it.

Deduplication

One incident is usually thousands of identical errors. Reporting each one buries the signal and uses up a free tier within minutes. Errors are fingerprinted by name and the first 200 characters of the message, and each fingerprint may send at most 5 reports per 5 minutes. Later duplicates are counted in errors.dropped but still logged locally.

With deploy/docker-compose.prod.yml, set ERROR_PROVIDER=sentry, SENTRY_DSN and optionally RELEASE in deploy/.env; the whole file reaches the server.

Logs

With docker compose, logs are docker compose -f deploy/docker-compose.prod.yml logs. Caddy writes access logs to /var/log/caddy/access.log in its own volume, rotated at 10 MiB with 5 files kept. At boot the server logs its data directory, the server key prefix and source, the dice protocol, the bot engine and the deep engine.