Architecture

Context

flowchart LR
    subgraph Sources["External sources (per docs/source-catalogue.md)"]
        S1[GENESIS Bayern S-01]
        S2[TED S-05 - verified live]
        S3[CORDIS S-06]
        S4[DOE S-04 pending]
        SX[Synthetic fixtures S-16/S-17]
    end
    Sources --> P[IngoShift pipeline]
    P --> WH[(Warehouse)]
    WH --> CLI[CLI status / detect-changes / report]
    WH --> EX[XLSX / CSV exports with provenance]
    WH --> RP[Weekly report - byte-reproducible]
    WH --> DB[Self-contained HTML dashboard]
    WH -. optional future .-> API[API server]

Layered data flow

flowchart TB
    A[acquire: fixture or verified live] --> R[RAW: immutable, content-addressed SHA-256,\npartitioned by source/date, meta sidecar]
    R --> B[BRONZE: pydantic-validated records,\nlineage block mandatory]
    B --> S[SILVER: region filter, normalization,\nenrichment, dedup - drops counted, never silent]
    S --> Q{Quality gate\ncritical / warning / info}
    Q -- critical fails --> X[NO marts, exit 3,\nfailures persisted as evidence]
    Q -- pass --> G[GOLD: ordered tested SQL marts]
    S --> ER[Entity resolution: exact key + domain +\nmanual override; review queue; NO fuzzy auto-merge]
    ER --> CD[Change detection over content-hashed snapshots]
    CD --> G

Star schema (core)

flowchart LR
    subgraph Dims
        dsrc[dim_source]
        dloc[dim_location]
        dskill[dim_skill]
        dtheme[dim_theme]
        dcomp[dim_company]
    end
    fj[fact_job_posting] --> dloc & dsrc
    fj --- bjs[bridge_job_skill] --- dskill
    fe[fact_employment] --> dloc & dsrc
    ft[fact_tender] --> dsrc
    ft --- btt[bridge_tender_theme] --- dtheme
    fcs[fact_company_snapshot] --> dcomp
    dcomp --- bct[bridge_company_theme] --- dtheme
    frp[fact_research_project] --- bpo[bridge_project_organisation] --- dcomp
    frp --- bpt[bridge_project_theme] --- dtheme
    fce[fact_change_event] --> dcomp
    fsr[fact_source_run] --> dsrc
    fdq[fact_data_quality_run] --> fsr

Grain, keys, and every column: see the generated data dictionary. Every fact carries the lineage block (source_id, run_id, raw_hash, retrieved_at, parser_version, data_class) — the lineage walk mart → fact → raw bytes is an automated test.

Source ingestion lifecycle

stateDiagram-v2
    [*] --> catalogued: entry in docs/source-catalogue.md
    catalogued --> verified: representative request recorded (date + result)
    catalogued --> rejected: legal/ToS review fails
    verified --> adapter_built: parser + fixture + contract tests
    adapter_built --> integrated: REAL payload replays through the same tests
    adapter_built --> adapter_built: fixture-only (GENESIS / CORDIS)
    integrated --> integrated: TED live refresh
    integrated --> stale: freshness threshold exceeded -> dashboard banner

No source may skip a state; "integrated" is a test outcome, not a claim.

Deployment

See deployment: the local SQLite demo and dashboard generation are verified. GitHub Pages automation is prepared but its first public run requires a configured remote. Docker/PostgreSQL remains future work.