On-Call Runbook

What to check when Wardline is paging you, in order. This complements High Availability and Observability (how to configure) with what to actually do when something’s wrong — failure-mode behavior confirmed by real chaos testing (killing Postgres, killing a replica, and killing a federation peer mid-load), not guesswork.

First: is it actually down, or just degraded?

curl -s -o /dev/null -w '%{http_code}\n' http://<replica>/healthz   # liveness -- always 200 once started
curl -s -o /dev/null -w '%{http_code}\n' http://<replica>/readyz    # readiness -- 503 during shutdown, or if postgres_storage is on and Postgres is unreachable

/healthz returning anything but 200 means the process itself is wedged or dead — restart it. /readyz returning 503 while /healthz is 200 means the process is alive but has a real dependency problem (see “Postgres is unreachable” below) or is mid-graceful-shutdown — check the log for "shutting down" before assuming the latter.

Log-line signatures and what they actually mean

Confirmed against the real binary, not inferred from source alone:

Postgres is unreachable

Confirmed behavior end to end (killed the container mid-load, watched recovery): the process does not crash. Budget checks fail open (requests admitted, not enforced). Audit writes fail loudly and that window’s entries are lost, not queued. Once Postgres is reachable again, no restart is needed — the shared pool reconnects on its own and behavior returns to normal automatically, typically within a few seconds of Postgres accepting connections again.

Action: fix Postgres. There is no Wardline-side lever to pull here beyond that — this is the documented availability-over-durability tradeoff (see Budget Enforcement’s own coverage of the shared Postgres pool, and internal/platform/pgpool’s doc comment), not a bug to patch around.

One replica died

Confirmed behavior (violently killed a replica — SIGKILL, no graceful shutdown — mid-load, load continued against the survivor): the other replica(s) are completely unaffected — no shared-lock contention, no crash propagation, no double-counting or gap in the shared Postgres-backed budget/audit state once the dead replica’s in-flight requests finish erroring out client-side. A load balancer health-checking /healthz will route around it once it stops responding.

Action: standard “why did the process die” investigation (OOM kill, node eviction, kubectl get events) — the cluster’s behavior around the death is already correct; there is nothing Wardline-specific to fix reactively.

A federation peer is unreachable

Confirmed behavior: the local instance’s own proxy hot path and anomaly detection are entirely unaffected. That specific peer stops receiving this instance’s anomaly summaries (logged loudly per tick, see the log-line table above) until it’s reachable again — and that gap is not backfilled: a tick’s data that failed to send is not resent once the peer recovers. Once both sides are healthy, the next anomaly on either side correlates normally.

Action: fix the peer’s reachability (network, DNS, the peer process itself). If you need the missed correlation for compliance/forensics, it’s genuinely gone for that specific window — this is a documented limitation (see Federation), not something to retry after the fact.

Latency degraded, but success rate is still 100%

Confirmed via load testing on a real machine (see Benchmarks): under real overload, Wardline degrades gracefully — throughput plateaus and latency climbs, but it does not start erroring out. If you’re seeing 100% success with climbing p99, you are past the machine’s (or container’s CPU limit’s) real sustained-throughput ceiling, not looking at a bug.

Action: this is a capacity question — check CPU usage against the container’s CPU limit/request first (a proxy sharing a core with a noisy neighbor looks exactly like this), then scale replicas horizontally (each replica’s own request-handling capacity is independent; only Postgres-backed state is shared) rather than searching for a code-level fix.

Postgres backup/restore

Wardline itself has no backup mechanism — postgres_storage is a consumer of your Postgres instance, not an operator of it. Its own data (audit_entries, budget_buckets, SCIM tables, anomaly baseline/ tenant/churn tables) is ordinary relational data with no special recovery procedure beyond your normal Postgres backup story. Confirmed with a real drill (dump the live database, restore into a separate database, verify row-for-row, then boot a real Wardline instance against the restored copy):

# Backup (pg_dump custom format -- the standard input to pg_restore)
pg_dump -h <host> -U <user> -d <db> -Fc -f wardline_backup.dump

# Restore (into a NEW database -- never restore over a live one you
# haven't already decided to discard)
createdb -h <host> -U <user> wardline_restored
pg_restore -h <host> -U <user> -d wardline_restored wardline_backup.dump

# Verify before cutting traffic over: row counts should match exactly
psql -h <host> -U <user> -d <db> -c "SELECT count(*) FROM audit_entries;"
psql -h <host> -U <user> -d wardline_restored -c "SELECT count(*) FROM audit_entries;"

Point audit.postgres_dsn at the restored database and start Wardline normally — no special recovery flag or startup mode is needed; the same CREATE TABLE IF NOT EXISTS idempotent-schema logic every Postgres-backed adapter already uses on every startup runs unconditionally, so a restored (non-empty) database behaves exactly like a fresh one that already has data in it. One real timing wrinkle to know about, not a restore bug: fixed-window budget counters (budget_buckets) reset on their own normal schedule based on wall time, so a bucket’s count in the restored copy may already read as “reset” by the time you point traffic at it, if enough real time passed between the backup and the restore relative to window_seconds — the restored count is correct for the window it reflects, it just may not be the window you expect if the gap was long.

Automating it: Wardline ships no backup scheduler of its own — same reasoning as not bundling Postgres itself (see mTLS/SPIFFE Bootstrap’s known limitations for the same “we don’t operate your infrastructure” boundary applied elsewhere). On Kubernetes, a CronJob running the exact pg_dump command above against your Postgres service, writing to durable storage, is the standard shape — for example:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: wardline-postgres-backup
spec:
  schedule: "0 3 * * *"   # daily at 03:00 -- pick a cadence against your own RPO
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: pg-dump
              image: postgres:16   # match your Postgres server's major version
              command:
                - sh
                - -c
                - >
                  pg_dump -h $(PGHOST) -U $(PGUSER) -d $(PGDATABASE) -Fc
                  -f /backup/wardline-$(date +%Y%m%d-%H%M%S).dump &&
                  find /backup -name 'wardline-*.dump' -mtime +14 -delete
              envFrom:
                - secretRef:
                    name: wardline-postgres-credentials   # PGHOST/PGUSER/PGPASSWORD/PGDATABASE
              volumeMounts:
                - name: backup-storage
                  mountPath: /backup
          volumes:
            - name: backup-storage
              persistentVolumeClaim:
                claimName: wardline-postgres-backups

The find ... -delete line is the retention policy — 14 days here, tune to your own compliance/storage requirements. PGPASSWORD in the environment is the same widely-used pg_dump/psql convention as every Postgres client tool; keep the secret scoped to a role with read-only backup privileges, not the same credential Wardline itself uses to write.

Rolling upgrades / mixed versions

A rolling deploy always has a window where an old and a new binary run at once against the same shared Postgres schema (see Budget Enforcement’s shared-pool coverage) — confirmed safe with a real drill, not assumed: built the latest tagged release and the current binary, pointed both at the same fresh database with postgres_storage and budget_enforcement on, ran one alone first (20 requests, proving the older binary’s own idempotent CREATE TABLE IF NOT EXISTS schema init), then started the newer binary against that same already-initialized database while the older one kept running, and fired 60 more requests split evenly across both at once. Result: both binaries served every request (200s throughout, no schema error in either log), and the shared budget_buckets row landed on exactly one combined count (80 — the 20 before plus the 60 concurrent split) with no drift or duplicate row — the same no-double-counting guarantee the HA runbook entry proves for same-version replicas, holding across versions too.

This works because every Postgres-backed adapter’s schema init is CREATE TABLE IF NOT EXISTS with no destructive migration step — an older replica never has a schema the newer one can’t also read, and neither one alters a table underneath the other. If a future change ever needs a genuine breaking schema migration (a column drop/rename, not an addition), this exact assumption stops holding and a real expand/contract migration plan becomes necessary before a rolling deploy — nothing in the codebase enforces that automatically today.

Escalate immediately vs. let it self-heal

Self-heals, don’t page a human at 3am for it alone:

• A burst of "budget check failed open" / "audit write failed" lines that stops once Postgres recovers.
• A burst of "federation publish failed" for one peer that stops once that peer recovers.
• Climbing latency under a real traffic spike that returns to normal once the spike passes.

Escalate:

• /healthz failing (the process itself is wedged/dead).
• Postgres unreachable for longer than your compliance/enforcement tolerance allows (budget silently unenforced, audit silently incomplete, for real — not a Wardline-side timer to tune, a Postgres availability problem).
• Any log line NOT in the table above at ERROR level, repeating.