Operations runbook

One droplet, one SQLite file, three timers. Boring on purpose: this is a system a single person has to be able to fix on a Sunday, from a laptop, without remembering anything.

Everything below assumes you are root on the droplet or sudo-capable, and that the service account is circuit.

What runs nightly

02:10  circuit-refinery.timer     pull the shared national artifacts
02:40  circuit-backup.timer       snapshot the graph, prune to 14 days
03:20  circuit-maintenance.timer  circuit pipeline

The order is by clock, not by unit dependency, and that is deliberate. If the refinery pull overruns or fails, maintenance still runs — on yesterday's artifacts, which is the correct trade. Chaining them would let a slow download silently skip the night's work.

What each one does:

• refinery — checks the manifest on the Makers Map box, downloads whatever changed (Companies House bulk, ICO register, ONSPD, the BCP and Dorset rates lists), and prints how old the oldest artifact is. Fails soft: a network error, an expired token or a missing manifest leaves the last good copy where it is and exits 0.
• backup — sqlite3 .backup into a gzip, deletes anything over 14 days old. Runs before the pipeline so the snapshot is of a state somebody has actually seen.
• maintenance — circuit pipeline: refinery check, ingest, enrich, score, publish, in one idempotent command that skips work it has already done.

Expect the whole night to take tens of minutes, most of it in the crawl and the enrichment, and to use most of the 2GB when a bulk join is running.

The verbs this runbook and the systemd units both assume

If the CLI ever renames one, deploy/systemd/circuit-maintenance.service and this table both need changing — and the unit is regenerated from maintenance=(...) in deploy/README.circuit.md, not edited in place.

Two wiring gaps that used to live here are closed, and each is pinned by a test so it cannot quietly reopen:

1. circuit purge exists and applies crawl_retention_days (90). It runs first in nightly maintenance (maintenance=("purge", "pipeline")) and the pipeline carries its own purge step just before scoring, so the notice's retention promise is kept even on a night the pipeline dies early. tests/test_cli_contract.py asserts every rendered maintenance verb exists as a CLI subcommand.
2. circuit preflight runs all eighteen checks — the platform's fourteen plus the four Circuit ones. tests/test_cli_contract.py:: test_preflight_runs_site_checks pins it; python -m circuit.preflight remains an equivalent way to run the same set.

After pulling these changes onto the droplet: re-render the deploy kit (deploy/README.circuit.md has the exact render block) and redeploy, so the maintenance unit picks up the purge verb.

The morning check

Sixty seconds, and it answers "did last night work?".

systemctl list-timers | grep circuit
journalctl -u circuit-maintenance.service --since yesterday --no-pager | tail -40
journalctl -u circuit-refinery.service   --since yesterday --no-pager | tail -10
sudo -u circuit /srv/circuit/venv/bin/circuit stats
ls -lh /srv/circuit/backups | tail -3

Healthy looks like: three timers with a LAST inside the last 24 hours, a maintenance run ending in a publish line with a plausible count, a refinery line saying refinery fresh (oldest N days), and a backup from this morning that is roughly the size of yesterday's.

The durable record is in the database rather than the journal, which is what makes "did it run?" answerable a month later:

SELECT job, started_at, duration_s, status, substr(result, 1, 120)
  FROM job_run ORDER BY started_at DESC LIMIT 10;

SELECT source_id, started_at, status, rows_in, entities_new, entities_updated
  FROM ingest_run ORDER BY started_at DESC LIMIT 10;

Read those in Datasette rather than over SSH — a question that takes five minutes to answer gets answered from memory instead, and wrongly.

Fixing a failed run

The first move is always the same: run it again. Every pipeline stage is idempotent, commits incrementally, and skips work it has already done. A failed overnight run is fixed by re-running it, and a re-run costs minutes rather than starting over.

sudo -u circuit /srv/circuit/venv/bin/circuit pipeline

If it fails the same way twice, work down this list.

The refinery is stale or unreachable

The journal says REFINERY STALE: ch_bulk (oldest 51 days, limit 45).

This is not an error and not an outage. The site is serving; its reference data is ageing. Exit code 0 is intentional — "too old" is a judgement about this site's promises, not about an HTTP request, so the alarm belongs outside the box (the meta-monitor) rather than inside it.

What it means and what to do:

1. Check the Makers Map droplet is up and that its refinery build timers ran. That box is the only shared dependency, and it is ingest-time only.
2. Check the token: CIRCUIT_REFINERY_TOKEN in /srv/circuit/shared/site.env. An expired token looks exactly like a network failure from here.
3. Run the pull by hand and read the output: sudo -u circuit /srv/circuit/venv/bin/circuit refinery pull
4. Under about a fortnight stale, do nothing else: a month-old national snapshot answers almost every question this month's would. Over a month, the hiring and incorporation freshness claims on the site start being wrong, which is a product problem rather than an ops one.

Never give the web process network access to the refinery to "fix" this. A page render that can block on somebody else's HTTP server will one day time out for a reason nobody can see from here.

The pipeline dies partway

• Disk full. df -h. Usual culprits: /srv/circuit/backups (retention should hold it to 14 files), refinery artifacts, and the journal. journalctl --vacuum-size=200M is safe.
• database is locked. A long ingest holds a write lock and something else timed out behind it. busy_timeout is 30s; if two things are genuinely running at once, find the other one (systemctl list-jobs, ps -u circuit) and let it finish rather than killing it mid-transaction.
• A metered API is capped. The Serper and Places caps are enforced in code and hitting one is a normal end to a stage, not a failure — a loop bug is meant to cost nothing. The run says so; the next run continues where it stopped.
• A crawl target is blocking us. Expected and fine. The crawler honours robots.txt, identifies itself, and moves on. If a business asks to be left alone, the robots.txt rule on /about/bot is the self-service route and a removal request is the stronger one.
• A migration failed. circuit init is idempotent and reports the version it reached. Never fix a schema by hand: write a numbered migration (site migrations start at 1000) and deploy it, because a hand-patched box and a fresh box diverge silently from that moment on.

Publication refuses

refusing to publish live data: the privacy notice cannot identify the controller

Working as designed. Set CIRCUIT_CONTROLLER_NAME, _CONTROLLER_ADDRESS and _ICO_REGISTRATION in /srv/circuit/shared/site.env and redeploy. Do not disable the check.

If publish runs but the count drops sharply, read the review queue before changing a threshold: is_public is recomputed from scratch every run, so a tightened threshold or a stale score moves records out, and that is the behaviour that makes thresholds worth having.

The site is down

systemctl status circuit.service
journalctl -u circuit.service -n 100 --no-pager
curl -sS -o /dev/null -w '%{http_code}\n' http://localhost:8000/healthz
systemctl status caddy

In order: is the app up, is Caddy up, is the certificate valid, is Cloudflare serving an error of its own. A restart (systemctl restart circuit) costs one request in flight and is a reasonable first move; a redeploy is safe at any time and preserves the database and the secrets.

Backups and restore

Nightly, sqlite3 .backup (not cp — that cannot safely copy a database being written to), gzipped, 14 days retained.

Most of the graph is rebuildable from public sources in an afternoon. What is not: claims, accounts, consent records with their wording versions, removal requests, and the suppression list. Losing the suppression keys means the next ingest recreates every record somebody asked to have taken down, and nobody finds out until they complain a second time.

Restore, tested before launch and again whenever the process changes:

systemctl stop circuit-maintenance.timer circuit.service
cp /srv/circuit/data/circuit.sqlite /srv/circuit/data/circuit.sqlite.broken
gunzip -c /srv/circuit/backups/circuit-<stamp>.sqlite.gz \
  > /srv/circuit/data/circuit.sqlite
chown circuit:circuit /srv/circuit/data/circuit.sqlite
sudo -u circuit /srv/circuit/venv/bin/circuit init      # migrations, idempotent
systemctl start circuit.service circuit-maintenance.timer

Keep the broken copy until the restore is confirmed. Test a restore somewhere else — a local machine is fine — rather than on the live box, because an untested backup is a hope and a first test during an incident is two incidents.

Deploys

CIRCUIT_HOST=<circuit-domain> ./deploy/deploy.sh <ip>

Idempotent and safe to run repeatedly. It preserves the database and the secrets, regenerates nothing that would log everybody out, and only loads fixtures into an empty graph. After any deploy that changes hostnames, DNS or mail, re-run circuit preflight.

Things that arrive from outside

A removal request. Honour it, whether or not the reasoning is agreed with. The form at /your-data does the work; by email, action it the same day. It writes suppression keys that survive re-ingestion, and the suppression_holds preflight check proves the mechanism still works. Confirm to the person that it is permanent.

A correction. A first-party fact, which outranks everything inferred. If the correction is about a sector or a website, treat it as a bug report against the classifier too: look at the signals that produced the wrong answer, because the same fault is usually sitting on other records.

An objection to the processing. Legitimate interests carries an absolute right to object here. Act on it; do not weigh it. Log it — an objection rate above 5% of notices sent is an explicit DPIA review trigger, because it would mean the reasonable-expectations reasoning in the LIA is wrong.

A complaint, or contact from the ICO. Both are DPIA review triggers. Reply within a working day, and write down what changed as a result even if the answer is nothing.

Suspicion of a breach. Assume the clock started when it was noticed: 72 hours to assess and, if it meets the threshold, notify. Take a backup copy of the current state before touching anything, because the timeline is part of the assessment. The system holds no special category data, no payment data and no credentials belonging to anybody listed, which shapes the assessment but does not remove it.

Kill switches

Quickest first. Each is one change and each is reversible.

Monthly

• Read the review queue rather than skimming it; it carries the machine's actual reasons for holding a record back.
• Check the refinery ages, the backup sizes and the disk.
• Check the gates on the admin dashboard against the plan's thresholds.
• Re-read docs/validation-protocol.md's doubt list from the last batch: it is where the next scoring change comes from.
• Quarterly, and after any change to the scoring weights or the publication thresholds: run a fresh validation batch.