Architecture Decision Records

Format: context → decision → consequences. Status accepted unless noted. All decisions revisitable; changes append a superseding ADR, never silent edits.

ADR-001 — Dual-target warehouse: DuckDB default, PostgreSQL full profile

Context: Brief prefers PostgreSQL, allows DuckDB demo mode; demo must run from a clean clone without infrastructure. Decision: DuckDB is the zero-setup default (make demo); PostgreSQL runs via Docker Compose as the full profile; both targets exercised in CI. Consequences: SQL must stay dialect-portable; dbt adapters (ADR-002) absorb most differences; slight extra CI cost, large accessibility win.

ADR-002 — dbt-core for gold marts

Context: Brief allows dbt "where it materially improves warehouse modelling". Dual-target SQL, model tests, docs, and a dependency graph are exactly that case. Decision: bronze→silver transformations that need Python (parsing, entity resolution) live in the Python package; silver→gold marts live in dbt with dbt tests. Consequences: one more tool in the chain, but lineage graph, portability, and tested SQL come nearly free; dbt docs feed the data dictionary.

Amendment 2026-07-28 (Phase 2): dbt-core is not installable in the build environment (PyPI blocked). Gold marts are implemented as ordered, tested SQL models (ingoshift.warehouse.MARTS) — the brief's explicitly allowed alternative. dbt migration stays on the table for a network-enabled environment; the SQL is already dialect-portable to ease it.

ADR-003 — Prefect 3 over Dagster/Airflow

Context: Orchestration must not dominate a demo-sized system; brief excludes Airflow without documented need. Decision: Prefect 3, flows runnable in-process (ingoshift run-all works without any server). Consequences: lineage is not delegated to the orchestrator — it lives in our own fact_source_run / fact_change_event tables, keeping the honesty story independent of tooling.

Amendment 2026-07-28 (Phase 2): Prefect is not installable in the build environment. Pipeline steps are plain, flow-shaped functions (ingoshift.pipeline) run by the CLI; the Prefect wrapper is added when an index-reachable environment exists. Because run tracking lives in fact_source_run, nothing about lineage or reproducibility waits on the orchestrator.

ADR-004 — Streamlit + Plotly as the one public dashboard

Context: Brief demands one polished dashboard over frontend complexity. Decision: Streamlit multipage app; Plotly charts; every page carries the source/coverage/quality banner component. Consequences: fastest path to a deployable, screenshot-able product; a React frontend remains possible later since all data flows through the FastAPI layer anyway.

Amendment 2026-07-29 (post-review): The deployable public product is a static, self-contained HTML dashboard generated from the warehouse (ingoshift dashboard). It uses inline CSS/SVG plus a small inline script for text and NUTS filtering; there are no external runtime resources. Company profiles and tender/source details use native HTML disclosure elements. The file is hostable on GitHub Pages or a file share and clearly labels its data classes. Byte identity is guaranteed when rendering the same stored warehouse; a new pipeline run intentionally has a new run ID and retrieval timestamps. Streamlit is no longer a prerequisite. An API-backed app is optional future work only when a client needs server-side refresh or access control.

Amendment 2026-07-30 (portfolio UI): The dashboard adopts a product-style shell with compact navigation, editorial hierarchy, responsive cards, and reduced technical noise above the fold. English remains the international portfolio default; users can switch the entire interface to German. German presentation text uses native Unicode umlauts and ß, while immutable source values remain unchanged in storage.

ADR-005 — Quality: Pandera schemas + internal severity rules engine

Context: Great Expectations/Soda add weight; the brief requires critical/warning/info severities and a publication-blocking gate. Decision: Pandera for frame/schema contracts at layer boundaries; a small internal rules engine for cross-record, semantic, freshness, and anomaly rules; all results persisted to fact_data_quality_run; critical failure = non-zero exit = no publish. Consequences: full control and transparency; we own the rule DSL (kept tiny).

ADR-006 — Tooling: uv + lockfile, Typer CLI, Ruff, mypy, pytest

Decision: Python 3.12, uv for locked reproducible envs, Typer for the CLI, pre-commit with Ruff (format+lint), mypy strict-ish on core packages, pytest with per-package coverage thresholds. Consequences: standard, boring, portable.

Amendment 2026-07-28 (Phase 1): baseline lowered to Python ≥3.11 (build environment ships 3.11; CI covers 3.11 and 3.12). CLI implemented with click instead of Typer: PyPI was unreachable from the build sandbox, click was available, and at this CLI size Typer adds only sugar — one dependency saved. Revisit only if the CLI grows complex option models. The uv lockfile is deferred until an index-reachable environment can resolve it (tracked in backlog F-01 note).

Amendment 2026-07-28 (Phase 2, ADR-001 scope): DuckDB is likewise not installable here; the warehouse v1 runs on SQLite behind a thin wrapper (ingoshift.warehouse), with dialect-portable SQL (ISO-TEXT dates, no autoincrement, no backend-specific date functions). The active backend string is printed in every run-all/status output. DuckDB (demo default) and PostgreSQL (full profile) are restored as soon as an environment can install them; the swap is contained in one module by design.

ADR-007 — Language and licensing

Decision: Code, docs, and dashboard UI in English; German source data preserved verbatim (skills/labels keep original strings alongside canonical forms). MIT for code; per-source data licenses documented and enforced via dim_source attribution fields; raw data never redistributed. Consequences: portfolio reaches international clients; regional authenticity preserved in the data itself.

ADR-008 — Posting-level job data is flag-gated; official aggregates are the real signal

Context: BA offers no official postings API; the bund.dev wrapper is reverse-engineered with no stated terms (verified 2026-07-28). Decision: the public demo's real labour-market data = official BA + GENESIS aggregates. The posting-level extraction pipeline (skills NLP, dedup, co-occurrence) is built and tested against clearly-labelled synthetic/fixture postings. The wrapper client may exist behind INGOSHIFT_ENABLE_UNOFFICIAL_SOURCES=true with a terms warning, off by default, excluded from published outputs. Consequences: demo stays legally clean; skills module is honest about data class; if official terms appear, the switch is one flag plus a source-catalogue update.

ADR-009 — Company universe: curated seed + open identifiers, no registry scraping

Context: No free, legal, current bulk registry exists (Handelsregister restricted; OffeneRegister dump is 2019-vintage). Decision: company dimension starts from a curated seed (~150–250 Region-10 organizations) built from public self-published information, Wikidata QIDs where available, and evidence accumulated from tenders/research/jobs records; every field carries source evidence; OffeneRegister dump used only as a labelled fixture for entity-resolution tests. Consequences: smaller N, stated coverage, zero legal exposure; the auditable- evidence model becomes a selling point rather than a workaround.

ADR-010 — Conservative, auditable entity resolution from day one

Decision: deterministic normalization (legal-form stripping, umlaut/case folding) + exact keys (domain, QID) + fuzzy matching only above conservative thresholds; below threshold → review queue, never auto-merge. Tables company_entity_map, company_merge_log, company_manual_overrides from the first schema version. Consequences: fewer merges, no silent wrong merges; every merge explainable in the dashboard.

ADR-012 — Extraction layering: dictionaries first, models only when justified

Context (2026-07-28, Phase 3): the brief mandates a layered extraction strategy (deterministic → structured metadata → NLP → optional flagged LLM → review queue) and forbids a mandatory LLM. Decision: layer 1 ships now: versioned taxonomy variants matched case-insensitively on word boundaries; observed strings retained on the bridge; method recorded per mention; seniority/occupation inference labelled via *_method. Layers 2+ (spaCy model, flagged LLM assist, review queue) are added only with a written value justification once real posting text exists — on synthetic fixtures a model would only demonstrate overfitting to our own inventions. Consequences: under-extraction is possible (inflections, synonyms) and is made visible by the JOB-W1 coverage warning; negation-blindness is documented in limitations.

ADR-013 — Component indicators instead of a composite "Transformation Index"

Context (2026-07-28, Phase 4): the brief permits a composite index only with published formula, component weights, inspectability, and sensitivity analysis, and demands interpretable indicators. Decision: publish component-level indicators only (mart_transformation_indicators: per theme — companies_total, companies_current, postings_linked; all 13 themes always present, including zeros). No weighted score anywhere. Consequences: users compare themes on stated facts; if a client ever needs a composite, it is built per the brief's full transparency requirements as new, separate work.

ADR-011 — Raw layer: immutable hashed Parquet/files partitioned by source and date

Decision: every collector writes raw responses verbatim (plus fetch metadata: URL, timestamp, status, content hash, parser version) to data/raw/<source>/<yyyy-mm-dd>/; bronze parses raw without re-fetching; replays are deterministic. Consequences: change detection and audits are possible years later; storage is cheap, trust is not.