What to configure on each project

Notes on how to instrument the sites this app watches, and what each thing is actually worth. Opinionated on purpose — everything here is optional.

The short version

If you do nothing else, do these four, in this order:

1. Cloudflare Web Analytics on every zone. Free, no script tag needed (Cloudflare can inject it), no cookie banner implication, and it gives you real-user Core Web Vitals. Queryable from the cf_graphql collector.
2. NEL — but check which collector you are already using first. NEL reports DNS failures, TLS errors and dropped connections: the outages no client-side analytics can ever see, because the page never loaded. Cloudflare already sets the header on every proxied zone and collects the results itself, so on most zones this is one toggle (enable the nel_reports dataset), not a header change. Point NEL at this app's /ingest/reports/... only where you want the raw report bodies — and read the section below first, because doing both at once gets you neither cleanly.
3. Content-Security-Policy-Report-Only, pointed at the same endpoint. Free intelligence about what third-party scripts your ad networks are actually pulling in, with zero risk of breaking the page.
4. expectText on every uptime monitor. The homepage assertion first, then one on /ads.txt set to your publisher line — silent ads.txt drift after a template change or CDN cache purge kills programmatic revenue and shows up nowhere else for weeks. Without a body assertion an uptime check only proves that something answered, which a parking page answers just as well as your site. See Content assertions.

Layers, and what each one catches

They overlap deliberately. A traffic drop that appears in Search Console but not in edge logs is a reporting artefact; one that appears in both is real.

Headers to set

Set these once per zone with a Cloudflare Rules → Transform Rules → Modify Response Header rule, so they apply regardless of what your origin does.

Reporting-Endpoints: default="https://meta.example.com/ingest/reports/blog?t=YOUR_INGEST_TOKEN"

Report-To: {"group":"default","max_age":86400,"include_subdomains":true,"endpoints":[{"url":"https://meta.example.com/ingest/reports/blog?t=YOUR_INGEST_TOKEN"}]}

NEL: {"report_to":"default","max_age":86400,"include_subdomains":true,"success_fraction":0.0,"failure_fraction":1.0}

Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
Referrer-Policy: strict-origin-when-cross-origin
Permissions-Policy: camera=(), microphone=(), geolocation=(), interest-cohort=()
Cross-Origin-Opener-Policy: same-origin-allow-popups

Three things people get wrong here:

• Report-To is not redundant. It is the deprecated v0 Reporting API and Chrome still uses it for NEL specifically. Reporting-Endpoints (v1) does not drive NEL. You need both headers, and the report_to group name in the NEL header must match a group defined in Report-To.
• success_fraction: 0.0 matters. Set it above zero on a busy site and you will ingest a report for every successful request.
• Setting your own NEL header takes reports away from Cloudflare's collector, and setting it on only some responses splits them unpredictably between the two. See below. This is the one that costs you data without ever looking like an error.

Replace blog with the project id from config/monitors.json — the ingest route rejects unknown ids.

NEL has two destinations, and a browser picks one

Cloudflare already sets NEL and Report-To on every proxied zone, pointing at its own collector (a.nel.cloudflare.com, group cf-nel). Those reports are what the nel_reports Log Explorer dataset contains. So NEL is already on — what you are choosing is where it goes, not whether it happens.

Default to Cloudflare's. It needs no header changes, it cannot be broken by a bad transform rule, and it keeps working when your own Worker is the thing that is down — which is exactly when failure reports matter. Point NEL at /ingest only when you want the raw per-report bodies rather than counts.

A NEL policy is stored per origin, not per response, and the most recent response wins. The NEL header names exactly one report_to group, so a browser reports to one collector at a time — but which one depends on which response it saw last. Override the header on your HTML and leave Cloudflare's on everything else and you do not get both collectors, you get an unpredictable split, changing every time a client fetches something served by a different rule.

The flagship zone is exactly this case: / and most paths return a custom "report_to":"default" pointing at its own collector, while /robots.txt still returns Cloudflare's cf-nel. The nel_reports dataset for that zone is not empty — it still receives about a thousand reports a day — it is just receiving an arbitrary fraction of them. Neither collector has the whole picture.

So pick one and apply it uniformly. Before assuming which a zone is on:

curl -sI https://example.com/ | grep -i '^nel:'
curl -sI https://example.com/robots.txt | grep -i '^nel:'

Two different answers means the zone is split.

DNS logs

dns_logs records the queries Cloudflare's authoritative nameservers answered for the zone. It is a layer below everything else here: these are lookups, not requests, so it sees clients that never got as far as opening a connection.

Worth having because of one query in particular. dns_nxdomain lists the names your zone was asked for and had no record for — which is both the scanners walking mail., vpn., dev. looking for something soft, and, more usefully, the real hostname somebody deleted a record for. A missing A record produces no HTTP request, no NEL report and no 5xx: nothing else in this app can see it, because from the edge's point of view the traffic simply stopped existing. dns_rcodes is the same signal aggregated, and worth watching as a ratio: a NOERROR share that steps down is a record that went away.

dns_names is the low-drama one — it tells you which subdomains are actually looked up, and how many resolvers ask for HTTPS/SVCB records rather than just A.

Volume is higher than nel_reports but far lower than http_requests, since resolvers cache. All three queries are aggregations, so cost scales with distinct names rather than query count.

CSP

Start in report-only and leave it there for a couple of weeks:

Content-Security-Policy-Report-Only: default-src 'self'; script-src 'self' 'unsafe-inline' https:; img-src 'self' data: https:; frame-src https:; report-to default

That policy is deliberately loose. The point of the first pass is not to block anything — it is to find out, from real traffic, what your pages actually load. On an ad-monetised site the answer is usually startling: Adsterra and Monetag both inject chains of third-party scripts that change without notice, and a tight CSP written from first principles will break your revenue within a day.

Read the results on the dashboard (Overview → Browser reports), tighten script-src to the hosts you actually see, and only then rename the header to Content-Security-Policy. Keep a report-only header alongside the enforcing one so you keep seeing what a tighter policy would have blocked.

frame-ancestors 'none' (or your own origin) is the one directive worth enforcing immediately — it costs nothing and is what the frame_protection check looks for.

Cloudflare-side configuration

Product analytics: PostHog, GA4, or neither

Cloudflare Web Analytics first. It is free, server-side sampled, needs no consent banner in most jurisdictions because it sets no cookies, and gives you Core Web Vitals. For a content site it may be all you need, and it is already wired into this app via cf_graphql.

PostHog if you need funnels, retention, session replay or feature flags — things page-level analytics cannot answer. Two setup notes:

• Reverse-proxy it through your own domain. A /ingest/* route on your zone proxying to PostHog's endpoint stops roughly a third of traffic being eaten by ad blockers. PostHog documents this; Cloudflare Workers is the easy way to do it. Without it your PostHog numbers and your edge-log numbers will disagree permanently and you will never work out why.
• PostHog's HogQL query API is straightforward, so a posthog collector here is a natural next addition — pull daily uniques and top paths into the same observations table and you can chart them against Search Console clicks.

GA4 only if something external requires it — an advertiser, a client, or AdSense reporting you want cross-referenced. It is the weakest of the three for your own decision-making: sampled, aggressively blocked, and the Data API is awkward. If you do run it, the Data API v1 runReport endpoint fits this app's collector interface fine.

Do not run all three. Two sources of truth is a reconciliation problem; three is a hobby.

SEO

• Search Console data lags ~2 days and Google revises the last few days. This app re-fetches a 10-day trailing window every run and upserts, so the numbers self-correct. The most recent two buckets on any chart will drift up.
• Cross-reference search_crawl_budget (what Googlebot actually fetched) with Search Console coverage. Crawl traffic dropping before impressions drop is the earliest warning you get.
• Bing Webmaster Tools has an API too and takes about ten minutes to add as a collector. Worth it if Bing is non-trivial for you — it also feeds ChatGPT search results.
• Monitor /sitemap.xml and /robots.txt with the uptime collector and an expectText assertion. Both are one bad deploy away from being a 404, and neither will alert you.

Indexing, which is the layer under all of that

Every figure above is about pages that are in the index. On a programmatic site the expensive failure is one layer down: pages that never got in. A thousand published pages Google crawled and declined to index produce no impressions, no clicks, no crawl errors and no 5xx — they look, in every chart on this dashboard, exactly like a thousand pages you never wrote.

gsc_index is the check for that, and there are three things to know before trusting its numbers.

It is a sample, not a census, because Google offers nothing else. The Page Indexing report you can see in the Search Console UI has no API and never has had one. URL Inspection is the only programmatic route and it takes one URL per call, so the coverage totals here are accumulated by inspecting a rotating slice of the sitemap and remembering the verdicts. The dashboard card says so, and counts URLs it has not reached yet under (not yet inspected) rather than dropping them — a coverage chart that quietly showed only the pages it had got to would read as a much smaller site, which is the one misreading that would make you relax.

Your numbers will not match the Search Console UI. They are not measuring the same population: the UI counts everything Google knows about, including URLs that are not in your sitemap at all. Compare trends, not totals.

The report grades the states, and the grading is the opinionated part. Not every non-indexed page is a problem — "Alternate page with proper canonical tag" and "Page with redirect" are how a correct site looks — so they are counted and not alarmed on. Three things do get raised:

"Pages not indexed" is deliberately a chore rather than an incident. A site publishing faster than Google indexes will have that list every morning forever, and a subject line that says action needed every morning forever is a subject line nobody reads.

Two things it does not do, and you should not assume otherwise:

• It cannot tell you why a page is not indexed. "Crawled — currently not indexed" is Google declining to say. Thin content, duplication and low authority all land in the same bucket, and no API distinguishes them.
• It is blind to pages outside the sitemap, since that is where the URL pool comes from. Orphaned pages, URLs only reachable by internal link, and anything a broken build dropped from the sitemap are invisible to it — which is one more reason the sitemap checks above matter.

Ad networks

Watch RPM, not revenue. Revenue moving with traffic is normal; RPM moving on flat traffic means something broke — a policy strike, an ads.txt problem, a failed ad script, or a demand-side change. The dashboard shows both.

Concrete things that silently cost money:

• ads.txt drift. Add an uptime monitor on https://example.com/ads.txt with expectText set to your publisher line. This is the highest-return check on this page.
• CSP or Permissions-Policy blocking ad scripts. If you enforce a CSP, a mistake here shows up as an RPM cliff with no traffic change. Cross-check against the CSP reports in this app before blaming the network.
• Adsterra/Monetag script hosts changing. Visible in CSP reports as new script-src violations days before it shows up in revenue.

Revenue rows are attributed by the network's own site/placement dimension rather than by project, which is why the ad collectors live on the synthetic _account project in the example config.

Content assertions on uptime checks

An uptime monitor without expectText asserts exactly one thing: something answered with the status you expected. That is a weaker claim than it looks — all of these return a perfectly healthy 200:

• a registrar parking lander, after DNS drifts or a site is never deployed
• a CDN or framework holding page
• a broken deploy that serves the app shell with no content in it
• a cached copy of a page whose origin is now failing

Set expectText on the homepage monitor of every project, not just on /ads.txt. The /ads.txt assertion protects revenue; the homepage assertion is the one that notices your site has stopped being your site while still returning 200.

It is a plain substring match against the raw response body, so:

• Pick something that proves the page rendered to the end. A footer string, a company number, a licence line. A <title> is weaker than it looks — an empty shell usually still has a correct <head>.
• Choose stability over specificity. A registered company number or a legal disclaimer outlives a tagline. Avoid anything containing a year, a count, or a price.
• Avoid HTML entities. The match is against source bytes, so a title containing & or ' will not match the string you typed. Pick a span of plain ASCII prose instead.

Verify the string against the live page before committing it, using the same user agent the collector sends:

curl -s -A 'meta-monitor/0.1 (+uptime check)' https://example.com/ | grep -c 'your expected string'

A zone that challenges non-browser user agents will fail that check with a 403 before expectText is ever considered — read the string out of a real browser instead, and add a WAF skip rule for the monitor's user agent.

The one case you get for free

Cloudflare serves origin failures as a bare error code: NNN body — 16 bytes, no markup. The collector treats any body matching /^error code: \d+$/ as down whatever status came with it, and tags the observation error=CloudflareErrorBody, so that case is caught even on a monitor with no expectText. It is the only such case. Everything in the list above still needs the assertion.

Only the third row has no fallback. That is the row expectText exists for.

Gaps this app does not cover

Be honest about these rather than assuming you are covered:

• True availability. The uptime collector runs inside Cloudflare. If Cloudflare is down, so is your monitor. Use an external prober (UptimeRobot, Better Stack, Healthchecks.io free tiers are all fine) for the one check that must not share fate with the thing it watches.
• Certificate expiry. Workers fetch does not expose the peer certificate, so this app cannot see cert expiry. Use Cloudflare Notifications for certs. Domain registration expiry is covered — the domain_expiry collector reads it over RDAP, no credentials — and is the half that actually gets forgotten, because a lapsed registration is measured in weeks to recover rather than minutes. Watch the number, not the renewal email: it goes to whichever address was on the registrar account when the domain was bought.
• Whether the cron itself is running. A Worker whose cron silently stops looks identical to a Worker whose data sources all returned zero. The watchdog below cannot help here — it runs inside the daily report, so it shares fate with the thing it would be reporting on. Point a dead-man's-switch (Healthchecks.io) at the daily lane, or check the Health & setup tab's run history occasionally.
• Anomaly detection on the numbers themselves. The watchdog below answers "is this collector still collecting", not "is this figure strange". A generic robust pass over observations — trailing median plus MAD per (project, source, query_id, dims_hash, metric), compared same-hour-of-day — is the natural next addition, along with ratio rules a spike detector cannot see: 5xx rate per hostname, cache-hit floor, verified-crawler error rate.

Is the monitoring still monitoring?

src/watchdog.ts runs as part of the daily email. It exists because "Failed collections" only reports a collector that errored on its most recent attempt, and the most expensive failures here make no error row at all.

Two checks, kept apart because neither finds the other's case:

Three details are load-bearing, and each was got wrong first:

• Liveness is measured on collected_at, never on bucket. The daily Log Explorer queries (top_paths, top_user_agents, countries, errors_5xx, search_crawl_budget, hotlinked_assets) carry a consistent two-day source lag: the bucket collected today is dated the day before yesterday. Judging freshness by MAX(bucket) reads all six as "stopped writing two days ago" on a permanently healthy system — six false alarms on day one, which is how a section gets skimmed forever after.
• The cadence is learned, not read from config. schedule: "daily" is collected by the six-hourly cron, so a "daily" monitor runs four times a day and a threshold derived from the config word would be eight times too lax.
• Gaps are measured between attempts that produced rows, not between attempts. top_referrers sees nothing on a quiet morning and dns_nxdomain nothing on a well-configured zone; judging those against how often they run flags healthy keys for the crime of two quiet hours. Learning the productive interval scales the threshold to each key's own sparsity and leaves dense keys where they were.

The "never collected" half is the one that reads on a Log Explorer dataset that was never enabled for a zone — which, per the table above, returns zero rows rather than an error and is otherwise indistinguishable from a quiet week. It is grouped by query rather than by project, because that is how the cause clusters: one unticked dataset produces the same three barren queries on every zone it was missed on.