How it works

The full method behind the ledger: which sources are read and why, how documents become structured data, how national codes are matched to a shared reference, how English is used as the comparison language, and how the gap between countries is measured. Tables marked live are read from the database when you open this page.

Contents — 1 Goals and coverage · 2 Sources and selection · 3 Pipeline · 4 Audit ledger · 5 English baseline · 6 Ontology · 7 Matching · 8 Semantic layer · 9 Gap score · 10 EU standards · 11 Limitations

1. Goals, and how far the current build meets them

The project has two goals. (1) An ongoing, auditable ledger of how EU rules are implemented in each Member State, how to map between the implementations, and how large the gap between them is, whether linguistic or procedural. (2) An MCP server and API that give cancer patients the best EU care options, with costs and conditions referenced to official documents.

RequirementStatusWhat exists nowWhat is missing
Log every site, API and file accessed, with timestamps, before processingdoneWrite-ahead, hash-chained audit ledger in D1 (§4); originals stored by SHA-256.External anchoring outside Cloudflare.
Track changing rules over timedoneEach re-fetch is compared by hash. Changed bytes create a new document version and change events; unchanged bytes create none.History deepens as the hourly schedule keeps running.
Store original, translation and compact referencedoneOriginal bytes, extracted text, English translations (two methods) and an LLM reference card per legal text, all hash-linked.Full-document translation is opt-in per source (budget).
EU directive → national implementation mappingpartialDirective 2011/24/EU in 4 official languages and Regulation 883/2004 are ingested and split into articles.National transposition acts (NL, DE, RO) are not yet sources, so the transposition table is empty.
Size of the gap (linguistic / procedural)partialCoverage, granularity, linguistic distance and translation disagreement, per concept and country (§9).Procedural gap (prices, authorisation rules) needs DE/RO tariffs.
MCP server / API for patientsdone7 read-only MCP tools and a REST API, every answer with provenance.Patient-friendly wording; per-country contact-point info.
Costs referenced to official documentspartialNL: national average sale price per care product (NZa) — pipeline ready.DE (InEK DRG × state base rate) and RO (CNAS) prices.
Conditions (cross-border routes)partialS2 vs Directive route with citations to the ingested EU texts.National lists of care requiring prior authorisation.
"Best EU care" for a patientnot yet—Needs quality data (e.g. ECIBC-certified breast units, outcomes). Out of the current scope; listed honestly.

2. Sources checked, and why they were selected live

Selection rules: (a) the publisher is the body legally responsible for the code system, tariff or law, or its official portal; (b) the content is public and its licence allows at least storing and citing it; (c) it covers the worked example (breast-cancer surgery) or the EU rule being implemented; (d) it can be fetched automatically within the free plan, or through the heavy lane for large files. When the best available copy is not from the official publisher (Romania), that is stated and the source is kept private.

WherePublisher and sourceWhy it was selectedLicence

2a. Romania: why the procedure list is hard to pull, and the official sources used instead

Not an Australian dependency. RO DRG v1 is a Romanian classification (Ordin MS 1199/2011, MO 542/2011), but its content is derived from Australia's ICD-10-AM / ACHI / AR-DRG under a government licence. The only consequences are the licence caveat and the code format (letter + 5 digits, e.g. Q00701).

Why it fails at the edge. No ministry or SNSPMS download or API was found; the list exists only as a 7.8 MB, ~310-page PDF hosted by hospitals. The free plan's 10 ms CPU limit and the edge PDF converter returned about 6,000 of ~869,000 characters, so the full text comes from the heavy lane (pdftotext on GitHub Actions).

Official sources added.

2b. Reference catalogue (portable, for processing outside Cloudflare)

Table document_reference lists every official act, annex and classification PDF behind a Romanian figure: legal reference (e.g. Ordin MS/CNAS 1018/528/2025), publication reference, URL, host (official, official portal, mirror, third party, none), what it holds and where (annex, DRG codes). The view v_document_reference adds the live fetch outcome, last fetch time, SHA-256 and size from the ledger.

Exports: CSV, SQLite script (sqlite3 refs.db < document_reference.sql), JSON, and the MCP tool document_references. The repo holds the same catalogue as data/references/references.sqlite (built from the migrations by scripts/build-reference-db.mjs). scripts/fetch-references.mjs downloads and extracts the PDFs on any machine: it checks robots.txt, writes the attempt before each request, hashes the file, runs pdftotext, and appends results to local_fetch.

Open request: a Law 544/2001 information request to SNSPMS / the Ministry of Health for an official, machine-readable RO DRG v1 copy and its reuse terms (docs/requests/RO-544-2001-RO-DRG-request.md).

3. Pipeline: what runs, when, and why

An hourly schedule (minute 17, UTC) queues every enabled source whose refresh window has passed. Work then runs as small, idempotent jobs, one at a time, so crawling stays polite and each step fits the free plan's 10 ms CPU limit. Every job has a deterministic key, so repeating work is a no-op. The diagram and live counts are on Data model & pipeline.

StepTriggered byWhat it doesLedger events
fetchhourly schedule; parser discovering child pages; admin runrobots.txt check (RFC 9309) → write-ahead log → conditional GET (ETag / Last-Modified) → SHA-256 → content store. Files over 4 MB are routed to the heavy lane.robots_fetched, fetch_started, fetch_completed / fetch_not_modified / fetch_skipped / fetch_failed
extracta new document versionHTML, PDF text layer, DOCX → Markdown with Workers AI Markdown Conversion; CSV and text kept as-is. Scanned PDFs with no text layer are flagged for OCR instead of guessed.extracted / ocr_required
parseextracted textA deterministic parser per source type (OPS pages, EUR-Lex articles, NZa CSV, RO DRG list) produces codes, provisions or tariffs. Large files are read in 96 KB chunks. A hash of the parsed records is logged, so the parse can be re-checked.parsed, parse_schema_mismatch
mapnew codesProposes code → concept mappings with the keyword rules and parent chaining (§7).mapped
translatenew non-English codes; backfillTwo English renderings per label: literal MT baseline and glossary-guided LLM (§5).translated, translation_rejected
compactnew legal / guideline version; backfillLLM reference card (title, scope, key points, rights and obligations), built from the parsed articles.compacted
gapnew mappings or translationsRecomputes the gap score; skipped when its inputs hash is unchanged.gap_computed
anchordaily 03:17 UTCRe-verifies new ledger rows and stores the head hash as a separate object.anchor
backfillevery hourly runBrings older data up to the current method (new model, new glossary) through normal jobs, never deleting.(the events of the steps it queues)

4. The audit ledger

5. English as the comparison baseline: how translation is done

English is used only as a pivot for comparison. The original-language label is always kept and shown. There are three kinds of English text, each recorded with its method:

TextMethodWhy
EU legal textsOfficial language versions fetched from EUR-Lex (EN, NL, DE, RO). No machine translation.All language versions of EU law are equally authentic, so the EU's own translations are the reference for linguistic comparison.
National code labels — baseline@cf/meta/m2m100-1.2b, literal machine translation, one label at a time.A neutral, reproducible baseline. It is visibly weak on medical Latin: German Mamma (breast) became "mother" in the first live run.
National code labels — preferredLLM (MODEL_COMPACT) with a curated medical glossary live in the prompt, temperature 0, JSON output checked for one translation per label.Correct terminology. The glossary's hash is logged with every batch, so the output can be reproduced. Output that fails the check is rejected and logged, not stored.

Where the two renderings differ, that difference is itself measured as translation disagreement (§9): it shows how much of an apparent gap may come from language rather than substance.

Glossary used (live)
LangTermEnglishNote

6. The ontology on top of the ledger

National systems are not mapped to each other directly (that would need n×(n−1) crosswalks). Each one is mapped to a small reference concept tree that this project defines and versions in git (migration 0002), with EU/ECIBC guidance as the scope reference and WHO ICHI planned as the external anchor. Layers:

LayerTablesMeaning
Jurisdiction → authority → sourcejurisdiction, authority, sourceWho publishes what, for which country or the EU.
Evidenceaudit_ledger, artifact, document, document_version, renditionWhat was read, when, and every derived text.
National vocabulariesprocedure_system, system_version, code, code_labelEach country's code systems (procedure, billing product, DRG), versioned.
Reference conceptsconcept (tree), concept_referenceThe shared meaning that national codes are compared through.
Mappingsmapping (SSSOM shape, SKOS predicates)"National code X relates to concept Y in way Z", with justification, confidence, author and review status.
Law, money, rightslegal_act, provision, transposition, tariff, entitlement_ruleThe EU rule, its national implementation, prices, and patient routes.

Current concept tree live

7. Matching algorithm (national code → concept)

  1. Lexical rules. Each concept has regular-expression keywords per language live, written for the original-language label (not the translation), so translation errors can't create false matches. Each rule carries the SKOS relation it proposes.
  2. Negation guard. A keyword preceded by zonder / ohne / fără / without does not count ("Mastectomie zonder okselklierdissectie" is not an axillary dissection).
  3. Most-specific wins. If a label matches both a concept and its child, only the child is proposed.
  4. Parent chaining. A code with no match of its own inherits its parent code's concepts as skos:broadMatch (the code is narrower), confidence 0.4, justification semapv:MappingChaining.
  5. Never auto-confirmed. Every mapping starts as proposed. A person confirms or rejects it through the admin API, and that review is itself logged.

Direction is always code → concept: closeMatch means about equivalent, broadMatch means the concept is broader than the code, narrowMatch means the concept is narrower.

Keyword rules (live)
ConceptLangPatternProposes

8. Is there a semantic layer?

Yes, but deliberately a thin one. Matching (§7) is rule-based and explainable. Semantics comes in only for measuring: English labels and concept labels are embedded with @cf/baai/bge-m3, a multilingual embedding model, and compared by cosine similarity. Embeddings are computed when a gap is recalculated and are not stored; there is no vector database yet. Embedding-based candidate suggestions (always as proposals) are the planned next step, once a review interface exists.

9. Gap score

ComponentQuestionHow it is computed
CoverageDoes the country have any code for this concept or its sub-types?1 if none, else 0.
GranularityHow well does the closest code fit?exact 0 · close 0.25 · code more specific 0.4 · code broader 0.6 · related 0.75 (best mapping wins).
LinguisticHow far is the national wording, in English, from the reference concept?1 − highest cosine(bge-m3) between the concept label and the preferred English labels.
Translation disagreementHow unstable is the English rendering?Mean (1 − cosine) between the MT baseline and the glossary LLM translation of the same labels.
ProceduralDo price or authorisation rules differ?Column exists; filled once DE and RO tariffs are ingested.
Composite1 if no coverage; otherwise the mean of granularity and linguistic.

These scores are a screening signal for where to look, not a clinical equivalence judgement. They become meaningful once the proposed mappings have been reviewed.

10. EU and international standards referenced live

The EU has no common procedure-code system, so the "European standard" in this project is threefold: EU law (the common rule, in its authentic language versions), EU clinical guidance (ECIBC, which defines the scope of the reference concepts), and international interoperability standards (SKOS/SSSOM for mappings now; ICHI, SNOMED CT and the EHDS exchange format planned).

StatusStandardIssuerHow Rosetta uses it

11. Known limitations