Starting a third vertical
Makers Map took eight founder-weeks from zero. Circuit took three, and most of that saving was this platform rather than practice. This document is the rest of the saving: it is what the scaffold cannot generate, in the order the decisions have to be made.
Read contract.md first — it is the API everything below builds against. Read this alongside scripts/new-vertical.py; the script writes the skeleton, this explains the choices inside it.
The honest summary of the work: four files are the product and none of them can be generated. Everything else is a command.
taxonomy.py | What this directory classifies things into, in two layers |
scores.py | This vertical's hard question, and how it is answered |
gates.py | What would make you stop |
theme/ + copy | Brand scope matching data scope |
0. Before the slug: is this actually a vertical?
A vertical is one brand, one geography, one taxonomy, one database, one droplet. It is not a facet of an existing site. The test is whether the hard question changes:
- Makers Map asks does this business exist at all? — most of its subjects are unincorporated and no register lists them.
- Circuit asks does this company actually operate here? — nearly all of its subjects are incorporated and registered offices lie in both directions.
Same machinery, different question, so the question is data. If your candidate asks the same question as an existing site over a different set of postcodes, it is a config change to that site, not a new vertical.
The second test is the audience. Two directories that would send the same person two emails about the same business are one directory with a filter.
1. The slug and the env prefix
The slug is load-bearing far beyond the package name. It becomes:
farm the Python package and the console script
/srv/farm the install root
farm.service and farm-maintenance/backup/refinery.timer
farm.env the environment file on the box
data/farm.sqlite the graph
farm_session, farm_admin, the cookie names (SiteConfig.session_cookie etc.)
farm_venue
So: lowercase, letters/digits/underscore, starting with a letter — the scaffold enforces this because it becomes a Python package name. Short. Internal. "Kiln" and "Circuit" are working names; the brands are "Makers Map" and "Bournemouth Tech". Keeping them separate means renaming the brand costs a config line rather than a migration.
The env prefix defaults to the slug uppercased and can differ (Makers Map's slug is makersmap and its prefix is still KILN during the backport — see backport-kiln.md). Nothing in platform code reads a hardcoded prefix; settings.env_reader(prefix) takes it as a parameter, which is the entire point of that module.
Credentials shared between sites on one machine — COMPANIES_HOUSE_KEY, GOOGLE_PLACES_KEY, SERPER_KEY, CLOUDFLARE_* — keep their conventional unprefixed names. SiteConfig.from_env reads a prefixed name first and falls back to the shared one. A laptop holds one Companies House key, not one per directory.
2. Scaffold
python scripts/new-vertical.py \
--slug farm --brand "Wessex Farm Map" \
--dest ../farm \
--lad E06000059=Dorset --lad E07000091="New Forest" \
--postcode-area DT --postcode-area BH \
--town Dorchester --town Blandford --town Wimborne \
--domain wessexfarmmap.co.uk
That writes pyproject.toml, README.md, .gitignore, .env.example, farm/{__init__,config,taxonomy,scores,gates}.py, farm/migrations/1000_site.sql, and placeholder directories for theme/, static/, content/, docs/ and tests/.
It deliberately does not invent the vertical's judgement. The taxonomy, the scores and the copy come out as questions with the shape of answers. Generating plausible sectors would produce a directory that looks finished and classifies nothing, which is worse than an empty file.
Two files it does not write and you need before the first deploy — the worked skeleton below supplies both:
farm/asgi.py, becauseplacegraph.web.create_appis a factory and the site package has to be imported first (importing it is what configures the platform). The deploy kit'sASGI_APPtoken defaults tofarm.asgi:app.farm/cli.py, because the generatedpyproject.tomldeclaresfarm = "farm.cli:main"and the deploy kit shells out tofarm init,farm stats --json,farm seed,farm scoreandfarm purge.
3. config.py
One SiteConfig, built once, installed with settings.configure() at import of the site package. Everything downstream reads settings.active().
Two rules, and both have teeth:
- Never capture
settings.active()in a module-level constant. Tests swap the config withsettings.using(...), and a captured value defeats that. Read it inside the function.SignalContext.taxonomyis a property for exactly this reason. - Facts about the product live in code; facts about a deployment live in the environment. Brand, geography and taxonomy are the product. Hostname, secrets, caps and publication gates are the deployment, and the deploy writes those into
/srv/farm/shared/farm.env.
Geography
target_lads={"E06000059": "Dorset"}, # what the product promises
neighbour_lads={"E07000091": "New Forest"}, # configured now, activated later
wider_lads={"E06000058": "BCP"}, # named correctly, never listed
postcode_areas=["DT", "BH"],
listed_lads is neighbours plus targets and is what publication checks; all_lads includes the wider set so a postcode that resolves outside the coverage is named rather than mislabelled.
Ingest by postal area, attribute by ONSPD, filter and count by lad_code. A postcode district slice cuts across councils — about 8% of SO14–SO19 is not Southampton, and BH straddles three authorities — and the buyer's entire product is counting properly.
The taglines and the scope
tagline states the real scope. A headline naming one town over listings that cover three counties is the fastest way to lose a reader's trust and the second fastest to lose a council's. If the data covers neighbours, say so, and put them behind an explicit "surrounding area" facet.
4. taxonomy.py — the two layers
Two things are kept deliberately apart, because conflating them produced the worst bug in the platform's history.
Layer 1, reconciliation. A published statistical definition, as a list of SIC codes. It matters because counts have to tie back to official statistics before a council economic development team can put them in a bid. Makers Map reconciles against the DCMS creative industries definition; Circuit against the DSIT digital sector definition minus the content/broadcast codes, so the two properties do not double-count the same agencies.
Layer 2, market. The sectors the audience actually recognises. Most of them will not map onto SIC at all. The directory covers them anyway, because they are the audience. Taxonomy.reconcile_report() exists so the gap between the two layers is a published number rather than an argument.
The rule that keeps SIC honest
A SIC code assigns a category only when the code is specific to it.
SIC 96.09 "Other personal service activities n.e.c." covers tattooing — and also mediation services, dating agencies, pet care and shoe repair. Mapping the whole code to one category labelled 332 companies a tattoo studio, one of them a family mediation firm.
So there are three buckets, and each entry carries its reason:
| Bucket | Declared as | Effect |
|---|---|---|
| Specific | Category(sic=("23.41",)) | Assigns the category outright, confidence ≥ 0.9 |
| Broad | broad_sic={"6201": "Computer programming activities"} | Widens discovery; nudges a keyword match by +0.1; never classifies alone |
| Excluded | excluded_sic={"9609": "…contains tattooing and 330 things that are not"} | Out of scope entirely, whatever the sub-activity looks like |
A broad code with no keyword match leaves the category unset and the record goes to the review queue rather than being guessed at publicly. That is category_from_sic in placegraph.scoring.engine, and it is the behaviour the reason strings exist to justify.
The reason is not decoration. It is what gets quoted when someone asks why their company was or was not counted, and writing it down is what stops a code being added on a hunch.
Five digits, then four
Taxonomy._lookup checks the five-digit code before the four-digit one:
47781 retail in commercial art galleries <- creative
47782 retail by opticians
47789 other retail n.e.c. <- vape shops, phone shops
Truncating to four admitted all three, which is how a vape retailer once reached a creative directory. Write five-digit codes where the split matters and four-digit codes everywhere else; the lookup handles both, and '74100', '74.10' and '7410' are all accepted spellings.
Keywords
Multi-word phrases are stronger evidence than single words and are scored higher (4 vs 3 in the trading name, 2 vs 1 in body text), so prefer them where they read naturally. Matching is word-boundary-anchored with a bounded inflection set — ceramic matches ceramics and ceramicist, media does not match mediation, design does not match designated.
The trading name is weighted far above body text because it is much stronger evidence: "Moonlight Tattoo & Piercing" is a tattoo studio; the word "tattoo" somewhere on a page might be a passing mention.
Validate the keyword lists against real data in week one. The first ingest tells you which categories dominate and which lists are starving. A category with three keywords in a market that has four hundred businesses in it is not a category, it is a gap.
5. scores.py — this vertical's hard question
Name the hard question before you write anything else. Getting it wrong is not a tuning problem, it is building the wrong product.
A score is never a model output. It is a list of named signals with weights, summed, squashed, and written to entity_signal with the detail that caused each one. That explainability is a compliance requirement, not a nicety: under Article 14 we have to be able to say how we characterised someone, and "the model said so" is not an answer. It is also the only reason weights ever get fixed — when a misclassification is one query away from the signals that caused it, tuning is evidence-led rather than superstitious.
What a collector is handed
SignalContext loads lazily and once per entity, and is shared by every score in the profile:
| Attribute | What it is |
|---|---|
ctx.entity | the entity row |
ctx.facts / ctx.fact(k) / ctx.fact_values(k) | current facts by key |
ctx.sources / ctx.from_source(id) | which sources contributed |
ctx.audit | the latest website audit that completed — an error row is a record of a failed fetch, not evidence about the business |
ctx.pages / ctx.page_role("careers") / ctx.text() | every crawled page, newest first; text bounded at 200k chars |
ctx.accounts | the most recent filed accounts |
ctx.premises | matched NNDR rated premises |
ctx.address_roles() | {'registered_office'} and nothing else is the classic "registered at the accountant's" shape |
ctx.sic_codes(), ctx.taxonomy, ctx.config |
Reach for those rather than issuing your own queries — a profile with five scores that each look at the crawl pages should read them once.
ctx.signal(name, detail) looks the weight up from the definition currently running and raises on an undeclared name, because scoring a typo as zero would hide it behind a plausible number. Pass weight= explicitly only where the strength is genuinely data-dependent (a distance, a proportion).
detail is stored and shown to humans, so it says what was observed — "site discloses 07351114" — not a restatement of the signal name.
Weight bands
Keep the strong/medium/weak bands honest, and make sure a single near-conclusive signal outweighs an accumulation of medium ones. Makers Map learned this the expensive way: a name ending "Ltd" is near-certain evidence of incorporation (using the suffix without being registered is an offence), so its weight is -1.20 — enough to beat no_ch_match (0.35) plus ico_without_company_number (0.30) plus footer_lacks_company_number (0.20) plus personal_name_pattern (0.18) combined. Without that, a company we merely failed to match got published as a sole trader, which is the misclassification the DPIA rates most serious.
The headline score, thresholds and kind
headline_scorenames the one score that orders the directory and the admin dashboard.db.set_scoredenormalises it ontoentity.headline_score; the authoritative copy is alwaysentity_score.publish_thresholdsis{score_name: minimum}. Prefix a keymax_to invert it — the site declares a score calledregistered_office_onlyand configuresmax_registered_office_only: 0.45. The prefix is not part of the score's name.ScoreProfile(kind_rules=[("contractor_likelihood", 0.6, "contractor")])lets a score implyentity.kind. A profile that declares rules owns the column outright: a record that stops clearing any threshold falls back tounknownrather than keeping a classification it has stopped earning. Declare none and the platform leaveskindalone.ScoreProfile(classifier_facts=(...))names the fact keys fed to the classifier. The default is("category_hint", "sic_description")— the two the platform writes itself. Source-specific keys are yours to add.
Every score is recomputed from scratch, never accumulated. A signal that disappears — a company number found on a later crawl, a heuristic tightened — has to be able to move a record back out of a bucket. A classification that only ratchets one way is worse than none.
6. gates.py — what would make you stop
Gates are data because Kiln's four were the right four for Kiln and the wrong four for a directory of technology firms. Declare them, and the dashboard computes them continuously so nobody discovers the answer in week eight.
placegraph.metrics ships parameterised gates — precision_gate, attribution_gate, claim_rate_gate, waitlist_gate, mapped_gate, published_gate, score_gate — where the query is the platform's and the target is yours.
Three properties worth understanding before you set numbers:
passis three-valued.Nonemeans "not measurable yet" — no sample hand-checked, nothing published. Reporting that as a failure in week one is how a project talks itself out of continuing before it has any evidence.direction="max"for gates that pass by staying under a target: cost per claim, complaint rate.required_passesdefaults to all but one. A single gate that has not moved — a council meeting not held, a sample not checked — should not veto a decision the other evidence has already made.
Precision is conventionally the kill criterion, and it must include whatever dimension this vertical finds hardest. A directory can be 100% correct that every listed business exists and still be worthless if half of them are not really here. precision(conn, dimensions=(...)) reads the JSON checks column of validation_sample, so adding a dimension is a hand-check convention, not a migration.
7. Publication
placegraph.scoring.publish.publish(conn) is the only thing in the platform that sets entity.is_public, and it recomputes the whole set from scratch every run. Tighten a threshold and records leave the directory on the next run, which is the only behaviour that makes a threshold worth setting.
It enforces, in order:
demo_modepublishes only synthetic records,publish_live_dataonly real ones. With both off nothing is published — an empty directory is recoverable, publishing personal data before the compliance groundwork is not.- Live publishing with
cfg.missing_controller_fields()non-empty raisesSystemExit. A notice that cannot name the controller, its ICO registration and a postal address is not a notice, so this is a hard stop rather than a warning. - Every configured threshold,
>=or<=per themax_prefix. A record with no such score has not met it in either direction and stays internal. - Status in
candidate|verified|claimed, alad_codeinlisted_lads, and a category abovemin_category_confidencewhenrequire_category_to_publish. status='claimed'overrides the thresholds but never the gates or the geography. The owner asking to be listed is the strongest evidence a record is right, but nobody can consent their way past groundwork that has not been done, and "the owner asked us to" does not make the geography true.removed,suppressedandremoval_pendingare always hidden.
Everything that did not publish and has never been looked at goes to the review queue. review_queue() returns each record with its scores, its top signals and publication_reasons() — the machine's actual working, phrased for a human, so a reviewer decides from the queue rather than from four other screens.
8. Theme
Templates resolve through ChoiceLoader([theme_dir, placegraph/web/templates]) and static files through the equivalent. Ship only what genuinely differs.
The alternative — copying the base layer into every vertical — means a fix to the removal flow has to be applied by hand n times, which is how the removal flow came to differ between two deployments of the same code.
- Override
home.htmlandabout.html; inherit the other twenty. - Put brand tokens in
static/style.cssas custom-property overrides. Forking the whole stylesheet means never getting its fixes. - Add routes by registering them on the app
create_appreturned, before anything else matches. Route modules never import each other, so replacing one surface never means forking two. - Mark any number your theme renders with
data-count="<definition>"so preflight's counts check can see it. One definition, computed in one place, rendered everywhere — a home page claiming 163 of something while the filter said 46 was real, both numbers were true of different populations, and a council that spots two of our own pages disagreeing has no reason to believe the third.
9. Sources
Every source the platform ships is in placegraph.sources, imported lazily — running an NNDR load must not drag in a search client and an iXBRL parser. Register any of your own with core.sources.register() before init_db(), because every fact carries a foreign key to source and the row carries the licence and terms that the profile page, the crawler disclosure and the Article 14 notice all read.
Choose by what the vertical's hard question needs:
| Source | SOURCE_ID | Reach for it when |
|---|---|---|
ch_bulk | ch | Discovery by SIC × postcode, and exact negative/positive matching. Bulk beats REST: it turns a rate-limited fuzzy search into a SQL join |
companies_house | ch | On-demand detail for individual records |
accounts | ch_accounts | Size banding, dormancy, employee counts from filed iXBRL |
ico | ico | Fee-payer register; the sole-trader goldmine, and corroboration everywhere else |
nndr | nndr | The presence oracle. A rated premises is the strongest evidence a business occupies somewhere; redacted ratepayer rows are the unincorporated occupiers |
serper / discover | serper | Statutory registers carry no URLs, and without them the classifier starves |
crawl | crawl | Category text, footer company numbers, careers pages, office towns |
audit | crawl | Platform, tracking, compliance furniture, credibility |
places | gplaces | Shopfront discovery. Hard-capped |
directories | partner_directory | Partner listings, with visible credit links |
refinery | — | The shared national artifacts. See refinery.md |
seed | seed | Synthetic fixtures for local development and demo mode |
Every one of them opens a run with start_run and closes it with finish_run, commits every ~250 rows with printed progress, skips work already done, and honours its cap in code rather than by convention — so a loop bug costs nothing. Set google_places_call_cap and serper_query_cap in config; both are overridable per deployment.
upsert_entity returns (None, False) for a suppressed record. Always check for None before using the id: a suppressed record silently skipped is the correct behaviour, an AttributeError is not.
10. Migrations, from 1000
Platform migrations occupy 0001–0999. Site migrations start at 1000. The two streams merge by version number without a registry and without either having to know about the other, and a site migration can safely reference any platform table because every platform version is lower. A duplicate version across the two directories is a hard SystemExit naming both files.
farm/migrations/1000_site.sql
farm/migrations/1001_grazing_licences.sql
Point migrations_dirs=(HERE / "migrations",) at it and ship it as package data.
Rules that apply to both streams:
- Numbered SQL files with
-- +up, and-- +downwhere reversal is genuinely possible. Where it is not, say the word "irreversible": a-- +downsection containing only prose is parsed as a script, run as an empty one, and recorded as a successful reversal that changed nothing. The runner only refuses when the text says so and contains no DDL. - Do not put
BEGIN/COMMITorPRAGMA foreign_keysin a migration. The runner wraps each file in its own transaction, so an innerBEGINfails and the pragma is a silent no-op — which makes the textbook SQLite table-rebuild actively dangerous here, becauseDROP TABLEwith enforcement still on cascades through every child row. UseALTER TABLE ... RENAME COLUMNandDROP COLUMN(SQLite 3.35+) instead; see backport-kiln.md for the worked case. - A
-- description:header explaining why. Six months on that is the only record of the reasoning and it is worth more than the DDL. - No
CREATE TABLE IF NOT EXISTSschema evolution anywhere. It silently skips a table that already exists, so a new column never reached a live database and the first index referencing it failed at deploy time. - Editing an applied migration is detected by checksum and refused. Applied migrations are history; add a new one.
11. Compliance instances
The shape of the LIA, DPIA, Article 14 notice and crawler disclosure is platform data; the answers are yours. Sections whose answer depends on the machinery carry a platform default (the crawler's identity, the retention window, the source list with its licences); sections that depend on the vertical's judgement are left as prompts, because a default that is 80% right is how a wrong assessment gets signed.
python -m placegraph.compliance.documents --site farm --doc dpia --check
python -m placegraph.compliance.documents --site farm --doc dpia \
--answers docs/dpia-answers.json --out docs/dpia.md
--check exits non-zero while a required section is unanswered. That check is the point: "the DPIA is done" is otherwise an assertion nobody can test.
Register your consent wording at import, with an immutable version id (v1-2026-08). account.consent_wording_ver stores the id, so a complaint two years later is answered with the text somebody actually saw rather than today's text. Re-registering an id with different text raises.
12. Deploy, and preflight
python -m placegraph.deploy render --site farm --dest deploy
renders the whole kit — provision.sh, bootstrap.sh, deploy.sh, Caddyfile.template, the five systemd units and their three timers, the backup and refinery scripts, and CHECKLIST.md. Everything is named after the slug, so a second box shares no names with the first. The output is regenerable and is not a working copy: re-render after a config change and the diff is the change.
The timers run by clock, not by unit dependency: refinery 02:10, backup 02:40, maintenance 03:20. If the refinery pull overruns or fails, maintenance runs anyway on yesterday's artifacts, which is the correct trade.
Then, against the running site and before PUBLISH_LIVE=1:
python -m placegraph.compliance.preflight --site farm
It checks the failures that are silent until they are expensive — pre-ticked consent boxes, missing DKIM/DMARC/SPF, canonicals naming the host you happened to deploy from, old hosts that 302 instead of 301, public counts that disagree with each other, a crawler advertising a disclosure URL that 404s, privacy surfaces that are not live. Skips are counted separately from passes; a failed blocker sets a non-zero exit code so it can sit in a deploy script.
CHECKLIST.md covers the once-per-domain jobs that need a human in a browser: DNS, the M365 mailbox behind the notice's contact address, the separate campaign sending subdomains, Search Console, Bing, and registering the site with the meta-monitor so refinery staleness has somewhere to alarm.
The worked skeleton
Copy this. It is a complete, runnable third vertical minus the judgement.
farm/__init__.py
"""Wessex Farm Map — a placegraph vertical.
Importing this package configures the platform. Everything downstream reads
`placegraph.core.settings.active()`, so this import must happen before anything
touches the database — which is why the CLI and the ASGI module both do it first.
"""
from __future__ import annotations
from placegraph.core import settings
from .config import CONFIG
settings.configure(CONFIG)
__all__ = ["CONFIG"]
__version__ = "0.1.0"
farm/config.py
"""Wessex Farm Map — site configuration.
Facts that are true of the *product* live here in code; facts that are true of a
*deployment* — hostname, secrets, caps, publication gates — are overridable by
`FARM_*` environment variables, and the deploy writes those.
"""
from __future__ import annotations
from pathlib import Path
from placegraph.core.settings import AffiliateLink, SiteConfig
from . import gates, scores, taxonomy
HERE = Path(__file__).resolve().parent
CONFIG = SiteConfig.from_env(
slug="farm",
env_prefix="FARM",
site_name="Wessex Farm Map",
tagline="Every farm gate, dairy and smallholding across Dorset "
"and the New Forest",
base_url="https://wessexfarmmap.co.uk",
contact_email="hello@wessexfarmmap.co.uk",
subject_plural="farms and food producers",
# Article 13/14. Publishing live data refuses until all three are set.
controller_name="",
controller_address="",
ico_registration="",
# Ingest by postcode AREA, attribute by ONSPD, filter and count by
# lad_code. A district slice cuts across councils.
target_lads={"E06000059": "Dorset"},
neighbour_lads={"E07000091": "New Forest"},
wider_lads={"E06000058": "Bournemouth, Christchurch and Poole"},
postcode_areas=["DT", "BH"],
region="South West",
towns=("Dorchester", "Blandford Forum", "Wimborne Minster", "Sherborne"),
taxonomy=taxonomy.TAXONOMY,
score_profile=scores.SCORE_PROFILE,
headline_score=scores.HEADLINE,
publish_thresholds={
scores.HEADLINE: 0.55,
# `max_` inverts the comparison and is not part of the score's name.
"max_dormant_likelihood": 0.40,
},
require_category_to_publish=True,
min_category_confidence=0.45,
theme_dir=HERE / "theme",
static_dir=HERE / "static",
migrations_dirs=(HERE / "migrations",),
# The click is measured whether or not a deal exists; that number is what
# the demand test needs long before the contract does.
affiliates=(
AffiliateLink("insurance", "Farm and smallholding insurance",
"https://example.invalid/farm-insurance",
"Public liability for farm shops and gate sales."),
),
extra={
"gates": gates.GATES,
"previous_hosts": (), # preflight asserts these 301 to base_url
"sending_domains": ("notices.wessexfarmmap.co.uk",),
"venue_worklist": (), # placegraph ships none; this is local
},
)
farm/taxonomy.py
"""Wessex Farm Map — what this directory classifies things into.
Two layers, kept apart:
1. Reconciliation — Defra's agricultural SIC classes, so counts tie back to
published statistics before a council or a LEP uses them.
2. Market — what a reader recognises: farm shops, dairies, cider makers. Most
of these have no code of their own, and the directory covers them anyway.
A SIC code assigns a category only when the code is SPECIFIC to it.
Broad codes widen discovery and corroborate a keyword match. They never
classify on their own.
"""
from __future__ import annotations
from placegraph.scoring import Category, Taxonomy
CATEGORIES: list[Category] = [
Category(
code="dairy",
label="Dairy",
sic=("01.41", "10.51"),
keywords=("dairy", "raw milk", "milk vending", "creamery",
"cheesemaker", "cheese making"),
blurb="Milk, cheese and butter made on the farm.",
),
Category(
code="farm_shop",
label="Farm shops & gate sales",
# No code of its own: farm retail sits inside broad retail classes
# alongside everything else. The name and the website are the only
# reliable signal, which is exactly why `sic` is allowed to be empty.
sic=(),
keywords=("farm shop", "farm gate", "honesty box", "pick your own",
"veg box", "farmers market"),
blurb="Produce sold direct, at the gate or from a shop.",
),
# TODO: the rest of the real categories, validated against a DT/BH ingest
# in week one.
]
# Genuinely inside the definition, far too broad to say what someone does.
# These widen discovery and lend a little confidence to a keyword match.
BROAD_SIC: dict[str, str] = {
"0111": "Growing of cereals, leguminous crops and oil seeds",
"0150": "Mixed farming",
"0161": "Support activities for crop production",
}
# Explicitly out of scope, however tempting a sub-activity looks. Note that a
# few classes split meaningfully at FIVE digits and only one branch qualifies.
EXCLUDED_SIC: dict[str, str] = {
"4631": "Wholesale of fruit and vegetables — distributors, not producers",
"5610": "Restaurants and mobile food service — a farm café is a café",
}
TAXONOMY = Taxonomy(
CATEGORIES,
broad_sic=BROAD_SIC,
excluded_sic=EXCLUDED_SIC,
reconciliation_name="SIC 2007 Section A (agriculture, forestry and fishing)",
)
farm/scores.py
"""Wessex Farm Map — the named scores, and the questions they answer.
The hard question here is neither existence nor presence: farms are on the land
registry and everyone knows where they are. It is **does this holding sell to
the public at all**, because a directory of farms nobody can buy from is an
atlas, and the reader wants a Saturday morning.
Each score accumulates weighted signals and persists both the number and the
signals that produced it, because under Article 14 we have to be able to explain
how we characterised someone.
"""
from __future__ import annotations
from placegraph.scoring import ScoreDefinition, ScoreProfile, Signal, SignalContext
HEADLINE = "sells_to_public_likelihood"
DORMANT = "dormant_likelihood"
def collect_sells_to_public(ctx: SignalContext) -> list[Signal]:
"""Signals that this holding trades with the public. Only what fired."""
out: list[Signal] = []
if ctx.premises:
# A rated premises with a retail description is the strongest evidence
# there is: somebody is paying business rates on a shop.
descriptions = " ".join(
(row["vo_description"] or "").lower() for row in ctx.premises)
if "shop" in descriptions or "retail" in descriptions:
out.append(ctx.signal("rated_retail_premises", descriptions[:120]))
else:
out.append(ctx.signal("rated_premises", descriptions[:120]))
if ctx.entity["website"]:
out.append(ctx.signal("has_website", ctx.entity["website"]))
text = ctx.text().lower()
for phrase, name in (("opening hours", "publishes_opening_hours"),
("farm shop", "describes_a_shop"),
("honesty box", "describes_a_shop")):
if phrase in text:
out.append(ctx.signal(name, f"site mentions {phrase!r}"))
break
if ctx.from_source("gplaces"):
out.append(ctx.signal("has_places_listing",
"listed on Google with a public address"))
if ctx.from_source("qr") or ctx.from_source("member"):
out.append(ctx.signal("self_registered",
"told us themselves, via QR or a claim"))
# Absence is evidence too. A holding whose only address is a registered
# office is being farmed by a company that lives somewhere else.
if ctx.address_roles() == {"registered_office"}:
out.append(ctx.signal("registered_office_only",
"no address other than the registered office"))
return out
def collect_dormant(ctx: SignalContext) -> list[Signal]:
"""Is this holding still trading? A closed farm shop is worse than none."""
out: list[Signal] = []
accounts = ctx.accounts
if accounts is not None and accounts["is_dormant"]:
out.append(ctx.signal("accounts_dormant",
f"dormant accounts to {accounts['made_up_date']}"))
if not ctx.pages:
out.append(ctx.signal("no_page_fetched", "nothing crawled successfully"))
audit = ctx.audit
if audit is not None and audit["http_status"] and audit["http_status"] >= 400:
out.append(ctx.signal("site_error",
f"site returned {audit['http_status']}"))
return out
SCORES = [
ScoreDefinition(
name=HEADLINE,
question="Can a member of the public buy something here?",
weights={
# Strong.
"rated_retail_premises": 0.60,
"self_registered": 0.55,
# Medium.
"describes_a_shop": 0.35,
"publishes_opening_hours": 0.30,
"rated_premises": 0.25,
"has_places_listing": 0.20,
# Weak.
"has_website": 0.05,
# Negative: keep this strong enough to beat two mediums, or a
# holding registered at an accountant's office publishes as a shop.
"registered_office_only": -0.90,
},
collector=collect_sells_to_public,
),
ScoreDefinition(
name=DORMANT,
question="Is there any sign this business has stopped trading?",
weights={
"accounts_dormant": 0.70,
"no_page_fetched": 0.25,
"site_error": 0.40,
},
collector=collect_dormant,
),
]
SCORE_PROFILE = ScoreProfile(
SCORES,
# A score may imply entity.kind above a threshold. Declaring any rule means
# the profile owns the column: a record that stops clearing every threshold
# falls back to 'unknown' rather than keeping a classification it has
# stopped earning.
kind_rules=(),
# The two platform defaults plus the keys this vertical's sources write.
classifier_facts=("category_hint", "sic_description", "gbp_types"),
)
farm/gates.py
"""Wessex Farm Map — the go/no-go gates.
Defined as data so the dashboard computes them continuously and nobody
discovers the answer in week eight.
"""
from __future__ import annotations
from placegraph.metrics import (GateSet, attribution_gate, claim_rate_gate,
mapped_gate, precision_gate, score_gate)
GATES = GateSet(
gates=[
# The kill criterion, and it includes the dimension this vertical finds
# hardest: being right that a farm exists is worthless if the public
# cannot buy anything there.
precision_gate(0.80, dimensions=("exists", "category", "lad", "sells")),
attribution_gate(0.90),
mapped_gate(400),
score_gate(threshold=0.55, target=0.45),
claim_rate_gate(0.08),
],
required_passes=4,
)
farm/asgi.py
"""The ASGI entry point the deploy kit points uvicorn at.
`create_app` is a factory and the site has to be configured before it runs, so
importing this package is the first thing that happens here.
"""
from __future__ import annotations
import farm # noqa: F401 (configures the site)
from placegraph.web import create_app
app = create_app()
farm/cli.py
"""The site's console script.
Thin on purpose: everything here is an argument-parsing shim over platform
functions. The deploy kit shells out to `init`, `stats --json`, `seed`, `score`
and `purge`, so those five keep their names.
"""
from __future__ import annotations
import argparse
import json
import farm # noqa: F401 (configures the site)
from placegraph.core import db, joblog, settings
def cmd_init(args: argparse.Namespace) -> int:
db.init_db()
print(f" {settings.active().db_path} is up to date")
return 0
def cmd_seed(args: argparse.Namespace) -> int:
from placegraph.sources import seed
with db.session() as conn:
print(json.dumps(seed.generate(conn, count=args.count)))
return 0
def cmd_score(args: argparse.Namespace) -> int:
from placegraph.scoring import publish
cfg = settings.active()
with db.session() as conn:
cfg.score_profile.run_all(conn)
db.rebuild_fts(conn)
publish(conn)
return 0
def cmd_purge(args: argparse.Namespace) -> int:
from placegraph.sources import crawl
with db.session() as conn:
print(json.dumps(crawl.purge_expired(conn)))
return 0
def cmd_stats(args: argparse.Namespace) -> int:
from placegraph import metrics
cfg = settings.active()
with db.session() as conn:
data = metrics.dashboard(conn, cfg.extra.get("gates"))
# The deploy reads {"coverage": {"mapped": N}} and treats a missing verb as
# "unknown", which is the safe direction.
print(json.dumps(data, indent=None if args.json else 2, default=str))
return 0
def cmd_refinery(args: argparse.Namespace) -> int:
from placegraph.sources import refinery
cfg = settings.active()
print(json.dumps(refinery.pull(cfg), default=str))
print(json.dumps(refinery.staleness(cfg), default=str))
return 0
def cmd_serve(args: argparse.Namespace) -> int:
import uvicorn
uvicorn.run("farm.asgi:app", host=args.host, port=args.port,
reload=args.reload)
return 0
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(prog="farm", description=__doc__)
sub = parser.add_subparsers(dest="command", required=True)
sub.add_parser("init", help="create or migrate the database").set_defaults(
fn=cmd_init)
p = sub.add_parser("seed", help="synthetic fixtures for local development")
p.add_argument("--count", type=int, default=220)
p.set_defaults(fn=cmd_seed)
sub.add_parser("score", help="rescore, reindex and republish").set_defaults(
fn=cmd_score)
sub.add_parser("purge", help="apply the retention window").set_defaults(
fn=cmd_purge)
sub.add_parser("refinery", help="pull the shared national artifacts"
).set_defaults(fn=cmd_refinery)
p = sub.add_parser("stats", help="coverage, gates and demand")
p.add_argument("--json", action="store_true", help="one line, for scripts")
p.set_defaults(fn=cmd_stats)
p = sub.add_parser("serve", help="run the web app")
p.add_argument("--host", default="localhost")
p.add_argument("--port", type=int, default=8000)
p.add_argument("--reload", action="store_true")
p.set_defaults(fn=cmd_serve)
args = parser.parse_args(argv)
# Joblog or it didn't happen: durable run records and a rotated log file,
# wired in from the first ingest rather than after the first 12-minute
# mystery.
#
# `init` and `serve` sit outside it, and neither is fussiness. joblog opens
# the database and creates `job_run` itself if it is missing, so running it
# before the migrations have run leaves platform migration 0001 facing a
# table it did not create and failing with `table job_run already exists`
# on every fresh install. `serve` is a long-lived process and would hold
# the job lock for the lifetime of the server.
if args.command in ("init", "serve"):
return args.fn(args)
with joblog.job(args.command):
return args.fn(args)
if __name__ == "__main__": # pragma: no cover
raise SystemExit(main())
farm/migrations/1000_site.sql
-- description: Wessex Farm Map site migrations start here.
--
-- Platform migrations occupy 0001-0999 and site migrations start at 1000, so
-- the two streams merge by version number without a registry. A site migration
-- can safely reference any platform table, because every platform version is
-- lower.
-- +up
CREATE TABLE grazing_licence (
id INTEGER PRIMARY KEY,
entity_id INTEGER NOT NULL REFERENCES entity(id) ON DELETE CASCADE,
holding_no TEXT,
issued TEXT,
source_id TEXT NOT NULL REFERENCES source(id),
created_at TEXT NOT NULL DEFAULT (datetime('now')),
UNIQUE (entity_id, holding_no)
);
CREATE INDEX idx_grazing_entity ON grazing_licence(entity_id);
-- +down
DROP TABLE IF EXISTS grazing_licence;
Then
python -m pip install -e ../farm
farm init
farm refinery
farm seed && farm score
farm stats
farm serve
What is shared, and what is deliberately duplicated
Shared, because the failure of duplicating it is concrete and has happened:
| Shared | The failure it prevents |
|---|---|
| Graph schema and provenance writes | Two directories describing the same suppression mechanism differently, one of them out of date |
| Normalisers and entity resolution | Two definitions of "the same business" |
| The two-layer taxonomy framework | Re-deriving the broad-code rule after re-making the 332-tattoo-studios mistake |
| The score engine and its explanations | An unexplainable number in front of the ICO |
| Every source adapter | Re-learning that bulk beats REST, per vertical |
| ONSPD attribution | Counting by postcode prefix and being 8% wrong |
| The base template layer, routes and a11y checks | A removal flow that differs between two deployments of the same code |
Compliance document skeletons and preflight | A pre-ticked consent box shipping twice |
| The deploy kit | Sixty hand-edited occurrences of a slug, one of which is missed |
| The refinery client | Two copies of a 5.5m-row snapshot |
Deliberately duplicated, and staying that way:
Brand · templates beyond the base layer · taxonomy · scoring weights · DPIA and LIA instances · venue lists · guide content · DNS and email.
Each of these is where the verticals actually differ, and an abstraction over two data points is a guess. A shared taxonomy would mean one directory's categories constraining another's; shared scoring weights would mean tuning Makers Map's sole-trader heuristics moved Circuit's presence score; a shared DPIA would be a document about no particular processing.
The rule of three governs promoting anything else. Two implementations are a coincidence; three are a pattern, and only the third one tells you which parts of the first two were incidental. When a third vertical wants the same theming mechanism the first two hand-rolled, then theming beyond the base layer gets promoted — with three examples to design against rather than two.
The corollary is the discipline that makes it work: when something is promoted, it moves, it does not get copied. A shared thing that still has a local fork is a shared thing that is about to diverge.