The refinery

The shared national data plane. One host builds the heavyweight, geography- independent artifacts; every site pulls what has changed, at ingest time, and serves from local files afterwards.

Some inputs are national, expensive and identical for everybody: the 5.5m-row Companies House snapshot, the ONS Postcode Directory, a normalised council lookup. Downloading and rebuilding those on each site's droplet costs the same bandwidth and the same hour every month, multiplied by the number of directories, and produces byte-identical results.

The whole thing is a URL and a manifest. That is not an understatement of the design, it is the design — and it is why this can graduate to a dedicated ingestion VM later without a single consumer noticing.

Both halves live in one module, placegraph.sources.refinery, because they are one contract and a contract with its two ends in different files drifts.

build side      build_manifest(artifacts, out_dir=...)   on the refinery host
consumer side   pull(cfg) / staleness(cfg)               on each site

1. What is built, and where

Today the refinery role runs on the existing Makers Map VM, which gains only the build timers and a served directory. That is deliberate: standing up a second box to save bandwidth on the first would be a strange trade at two sites.

There are two kinds of artifact, and the distinction is not cosmetic — it follows from whether the data is national or scoped.

Built databases are shared whole. ch_bulk is the only one today. ch_bulk.load() produces a standalone SQLite file and every entry point takes a db_path override, precisely so the refinery can build it somewhere other than a site's data directory:

from placegraph.sources import ch_bulk

zip_path = ch_bulk.download()                              # cached; monthly
ch_bulk.load(zip_path, db_path="/srv/refinery/out/ch_bulk.sqlite")
ch_bulk.build_address_density(db_path="/srv/refinery/out/ch_bulk.sqlite")

That snapshot lives in its own file rather than in the graph. 5.5m rows would bloat the nightly backup and the exploration instance, and they are a disposable derivative that can be re-downloaded. bulk_db_path() resolves to {data_dir}/ch_bulk.sqlite on a consumer unless overridden — exactly the file the manifest ships, so a site that pulls it never loads it at all.

load() drops and recreates the table rather than using CREATE TABLE IF NOT EXISTS. A load is always a full replacement, so there is nothing to preserve, and keeping an older table shape around silently breaks the load when a column is added. That happened once, when address_line1 was introduced.

Source files are shared as files, and each site loads its own slice. ONSPD, ICO and NNDR all write into the site's own graph — postcode, entity, rated_premises — and every one of those loaders is scoped: load_onspd(conn, path, areas=cfg.postcode_areas) reads 2.7 million rows and keeps a few tens of thousands. There is no national loaded database to share, because a national postcode table is not what any site wants: the graph is filtered by construction and the filter is per-vertical.

So the refinery's job for these is fetching, decompressing and normalising once — the expensive, identical part — and each consumer runs its own filtered load afterwards. Publishing them as loaded databases instead would ship every site 2.7m rows to delete 2.65m of them.

2. The manifest

Written by build_manifest, atomically, beside the artifacts:

{
  "version": 1,
  "generated_at": "2026-08-13T02:04:11Z",
  "artifacts": [
    {
      "name": "ch_bulk",
      "path": "ch_bulk.sqlite",
      "sha256": "9f2c…64 hex chars…",
      "bytes": 1734405632,
      "built_at": "2026-08-13T01:58:02Z",
      "source_url": "https://download.companieshouse.gov.uk/en_output.html",
      "cadence": "monthly",
      "notes": "BasicCompanyDataAsOneFile-2026-08-01.zip, loaded and indexed"
    }
  ]
}

Two properties of the write are load-bearing:

• It is written to manifest.json.part and renamed. A site polling mid-write would otherwise read half a manifest and conclude every artifact had vanished.
• A missing artifact is a hard SystemExit at build time. Publishing a manifest that names a file nobody can fetch breaks every site at once, so the build refuses rather than shipping a manifest it cannot back up.

Build script

from pathlib import Path

from placegraph.sources.refinery import Artifact, build_manifest

OUT = Path("/srv/refinery/out")

manifest = build_manifest([
    Artifact("ch_bulk", OUT / "ch_bulk.sqlite",
             source_url="https://download.companieshouse.gov.uk/en_output.html",
             cadence="monthly",
             notes="Company Data Product, loaded and indexed. Consumers open "
                   "this read-only; there is nothing to load."),
    Artifact("onspd", OUT / "onspd.zip",
             source_url="https://geoportal.statistics.gov.uk/",
             cadence="quarterly",
             notes="The published release, unmodified. Each site runs "
                   "load_onspd(conn, path, areas=cfg.postcode_areas)."),
    Artifact("ico_register", OUT / "ico_register.csv",
             source_url="https://ico.org.uk/ESDWebPages/DoSearch",
             cadence="weekly",
             notes="Fetched and decompressed once. Each site runs "
                   "ico.ingest(conn, path) against its own scope."),
], out_dir=OUT)

print(f"{len(manifest['artifacts'])} artifacts, "
      f"generated {manifest['generated_at']}")

Serving

Caddy, from the existing droplet, under an authenticated path:

handle_path /refinery/* {
    @authorised header Authorization "Bearer {env.REFINERY_TOKEN}"
    handle @authorised {
        root * /srv/refinery/out
        file_server browse
    }
    respond 403
}

rsync-over-SSH is the equally valid alternative and is also one config stanza — pick one. If you pick rsync, the consumer half of this document does not apply and you own the checksum verification yourself, which is the reason HTTP plus a manifest is the default.

3. The consumer contract

Four rules. Every one of them is about what happens when the refinery is unavailable, because that is the only interesting case.

Pull at ingest time only

from placegraph.sources import refinery

result = refinery.pull(cfg)          # top of the ingestion job, nowhere else

Never from a request handler, never from application startup, never from a template. placegraph.sources.refinery imports nothing from placegraph.web, and nothing in placegraph.web may import it. That import boundary is the enforcement mechanism, and it is worth keeping deliberately.

A page render that can block on someone else's HTTP server is a page render that can time out for a reason the site operator cannot see. A site whose directory stops loading because a shared host is down has traded a real dependency for an imaginary saving — and the entire premise of this platform is that at runtime a site depends on nothing but its own box.

Fail soft, always

pull() never raises. It returns a status dict in every case, including catastrophe:

{"status": "ok" | "stale" | "unconfigured",
 "updated": [...], "unchanged": [...], "failed": [{"name": ..., "reason": ...}],
 "bytes": 0, "error": None,
 # plus everything staleness() returns
 "artifacts": {...}, "age_days": 12, "max_age_days": 45, "stale_artifacts": []}

Network error, DNS failure, expired token, malformed JSON, corrupt download, missing manifest, filesystem error — all of them land in one except and all of them leave the existing files exactly where they are. 401/403 is called out separately in the message ("the refinery refused the token") because a rotated token is the most likely cause of a long, quiet staleness.

An ingestion run with last month's snapshot is a good run. A month-old Companies House snapshot answers almost every question this month's would.

Keep the last good copy

data/ch_bulk.sqlite          the artifact
data/ch_bulk.sqlite.part     during a download, deleted on any failure
data/refinery-state.json     what we last successfully pulled

• Each artifact streams to <name>.part, is hashed as it streams, and is only os.replaced into position once the digest matches. A checksum mismatch keeps the existing copy. The failure this replaces is a 400 MB SQLite file that opens fine and is missing a month.
• A failed download never leaves a plausible-looking file behind — the .part is unlinked in a finally.
• An artifact whose recorded sha256 matches and whose file is present at the right size is skipped entirely. Re-running the pull is cheap.
• A corrupt or absent state file reads as empty, costing one redundant download and never any data.
• Manifest path values are sanitised before use: anything absolute, containing .., or containing a drive-letter colon is rejected outright, and the file lands under data_dir by its basename. The manifest arrives over HTTP from another host, and "write wherever this remote JSON says" is not a sentence worth writing.
• Both remote-input volumes are capped in code, from config: {PREFIX}_REFINERY_MAX_ARTIFACTS (default 32) and {PREFIX}_REFINERY_MAX_BYTES (default 8 GB).

Alarm on staleness, outside the box

refinery.staleness(cfg)
# {'artifacts': {'ch_bulk': {'age_days': 61, 'built_at': ..., 'present': True,
#                            'stale': True, 'cadence': 'monthly'}, ...},
#  'max_age_days': 45, 'age_days': 61, 'stale_artifacts': ['ch_bulk']}

Age comes from the refinery's stated built_at, because re-downloading an unchanged file does not make its contents any younger. stale is true when the age exceeds {PREFIX}_REFINERY_MAX_AGE_DAYS (default 45) or the file is missing.

Staleness is a number, not an error. {{SLUG}}-refinery.service prints

REFINERY STALE: ch_bulk (oldest 61 days, limit 45)

to the journal and exits 0 on purpose, so the pipeline behind it runs on what is already there. "Too old" is a judgement about a particular site's promises, not about an HTTP request, so the alarm belongs to the meta-monitor watching the box — which is why registering the site with it is a line on the deploy checklist.

Configuration

4. Why the web process is never given network access to it

Three independent layers say the same thing, because one of them will eventually be edited by someone who does not know why it was there.

1. The import boundary. refinery.py is in placegraph-sources, which placegraph-web does not depend on. A route that wanted to pull would have to add a dependency, in a pull request, in front of somebody.
2. The unit split. The pull is {{SLUG}}-refinery.service, a oneshot on its own timer with TimeoutStartSec=1800. The web service is a long-running unit that never runs it.
3. systemd hardening. Both units run ProtectSystem=strict with ReadWritePaths naming only the data directory, so the web process can write the graph and nothing else — including nothing the refinery pull would need.

The risk being managed is named explicitly in the build plan: the refinery becomes a hidden runtime dependency. The mitigation is the contract, and the contract is only real while it is enforced structurally rather than remembered.

5. Graduating to a dedicated ingestion VM

The refinery is a URL and a manifest, so the move is:

1. Stand up the new box. Install the build environment.
2. Run the build timers there until manifest.json is complete and the checksums match what the old host is serving.
3. Serve it under a hostname of its own, with the same token semantics.
4. On each consumer, change one line in /srv/<slug>/shared/site.env: {PREFIX}_REFINERY_URL=https://refinery.example/refinery.
5. Run <slug> refinery by hand once. Every artifact reports unchanged, because the checksums are identical — which is the proof the cutover worked.
6. Retire the build timers on the old host.

No consumer code changes. No coordinated restart. If step 4 lands on one site before another, the two sites are pulling byte-identical files from two hosts, which is not a state anything can detect or care about.

The one thing to keep stable is the artifact name field. It is the key in every consumer's state file, and renaming it makes every site download the same file again under a new key while the old entry ages into a false staleness alarm. Path can move freely; name cannot.

6. The nightly order

Ordering is by clock, not by unit dependency, and that is the robustness argument rather than an accident:

Each timer carries Persistent=true (a box that was off catches up) and a randomised delay (300s for the refinery, 600s for maintenance) so several sites do not hit the refinery in the same second.

The units also declare, belt and braces:

# {{SLUG}}-refinery.service
Before={{SLUG}}-maintenance.service

# {{SLUG}}-maintenance.service
After={{SLUG}}-refinery.service

That ordering only bites if anything ever queues both units in one transaction. The timer is what actually guarantees the sequence — and, crucially, the After= is ordering only, not a Requires=. If the pull overruns its half-hour gap or fails outright, maintenance still runs, on yesterday's artifacts. Stale reference data beats no maintenance.

Retention runs before scoring, so scoring never reads crawl text that should already have been dropped.

7. Operator commands

On a consumer

# What the timers think they are doing.
systemctl list-timers | grep <slug>

# The last pull, in full.
journalctl -u <slug>-refinery.service -n 100 --no-pager

# Only the thing the meta-monitor cares about.
journalctl -u <slug>-refinery.service --since "-2 days" | grep REFINERY

# Pull now, as the service account, with the same environment.
sudo systemctl start <slug>-refinery.service

# Pull now, in the foreground, and see the JSON.
sudo -u <slug> /usr/local/bin/<slug>-refinery.sh

# One artifact only.
sudo -u <slug> /srv/<slug>/venv/bin/python -c "
import <slug>
from placegraph.core import settings
from placegraph.sources import refinery
print(refinery.pull(settings.active(), only={'ch_bulk'}))"

# How old is everything, without touching the network.
sudo -u <slug> /srv/<slug>/venv/bin/python -c "
import json, <slug>
from placegraph.core import settings
from placegraph.sources import refinery
print(json.dumps(refinery.staleness(settings.active()), indent=2))"

# Force a re-download of one artifact: forget what we hold, then pull.
sudo -u <slug> python - <<'PY'
import json, <slug>
from placegraph.core import settings
from placegraph.sources import refinery
cfg = settings.active()
state = refinery.load_state(cfg)
state.pop("ch_bulk", None)
refinery.state_path(cfg).write_text(json.dumps(state, indent=2))
PY

# Is the snapshot the shape this version of the code expects?
sudo -u <slug> /srv/<slug>/venv/bin/python -c "
import json, <slug>
from placegraph.sources import ch_bulk
print(json.dumps(ch_bulk.snapshot_info(), indent=2, default=str))"

assert_snapshot_current() is the same check with teeth: it exits with a message naming the missing columns or the wrong layout version, and telling you the exact reload command. The ingestion sources call it before they start, so a stale snapshot fails at the top of a run rather than three hundred thousand rows in.

On the refinery host

# Rebuild everything and republish the manifest.
sudo -u refinery /srv/refinery/venv/bin/python /srv/refinery/build.py

# What is currently published.
curl -s -H "Authorization: Bearer $REFINERY_TOKEN" \
     https://refinery.example/refinery/manifest.json | jq '.artifacts[]
     | {name, bytes, built_at, cadence}'

# Verify what is on disk matches what the manifest claims.
sudo -u refinery /srv/refinery/venv/bin/python -c "
from pathlib import Path
from placegraph.sources.refinery import read_manifest, sha256_of
root = Path('/srv/refinery/out')
for a in read_manifest(root / 'manifest.json')['artifacts']:
    ok = sha256_of(root / a['path']) == a['sha256']
    print(('ok  ' if ok else 'BAD '), a['name'])"

The last one is the check to run after any manual file operation on the refinery host. A manifest whose checksums no longer match its files does not break loudly: every consumer downloads, fails verification, keeps its old copy, and reports stale — which looks exactly like the refinery being down.