Skip to content

Phase 2 Review — Core Data Platform

Date: 2026-07-28 · Four-review standard applied to the data-platform build.

1. Engineering review

Executed and green: ruff clean, mypy strict clean (21 source files), 79 tests passed (was 36), mkdocs build --strict green, and the real demo: ingoshift run-all → gate PASS → 4 marts built → ingoshift status reads the warehouse back (12 employment facts, 3 tenders, 12 quality-rule results).

Findings: - Backend deviation (ADR-001 amendment): DuckDB could not be installed (PyPI 403), so the warehouse wrapper runs on SQLite with dialect-portable SQL (ISO-TEXT dates, no autoincrement, no backend-specific date functions). The backend string is printed in every run/status output so nobody mistakes the fallback for the target architecture. - dbt and Prefect deferred (ADR-002/003 amendments): not installable here. Marts live as ordered, tested SQL constants (the brief's allowed alternative); pipeline steps are plain functions designed to be wrapped by Prefect later. Lineage never depended on the orchestrator, so nothing honesty-relevant is lost. - Raw-store immutability is enforced (content-addressed, verify-don't-overwrite) and validate replays from raw without re-fetching — replay capability is real, not a slogan.

2. Data review

  • Lineage proven end to end (C-09): an automated test walks a mart number (Ingolstadt, 2023-06-30, 101 870) → fact row → raw hash → recomputed SHA-256 of the bytes on disk → equality with the committed fixture. This is the core promise of the project and it is now a failing-able test, not prose.
  • Gate proven: a poisoned fixture (negative employment) blocks publication: marts are not built, the CLI exits 3, and the failed rule is persisted to fact_data_quality_run as evidence.
  • All data is data_class=fixture and every fact row says so. The GENESIS marker conventions (- = semantic zero, x = suppressed, …) are handled and tested; the region-total mart publishes completeness flags instead of silently summing over suppressed cells.
  • Finding (carried): both parsers are written against constructed fixtures. S-01/S-05 remain NOT integrated until a real export/response is replayed through the same contract tests (C-03 stays open; sandbox cannot fetch).

3. Business review

  • What a user can already do: run one command, get a quality-gated warehouse with employment facts, tender facts, deadline views, and a persisted quality report — the "pipeline that explains itself" story now demos end to end.
  • What this phase does NOT yet answer: skills questions (Phase 3), company intelligence (Phase 4). The employment mart answers US-05's shape but with fixture values — stated on every output via data_class.

4. Honesty review

  • run-all --live fails with exit 4 and an explanation naming the catalogue — fixture data can never silently pose as live data (mode travels with every raw record and fact row). ✅
  • Fixture files themselves carry FIXTURE markers in names, values, and READMEs; constructed-not-real is stated at every layer. ✅
  • make demo now claims exactly what it does; make serve still fails honestly. ✅
  • Checked: no claim of DuckDB/Postgres/dbt/Prefect being in use anywhere in README or docs — all marked deferred with dates. ✅

Gate decision: Phase 2 passes. Carried obligations: C-03 local source re-verification (needs a normal network), first CI run, DuckDB/Postgres backend restoration, uv lockfile. Proceed to Phase 3 (Jobs & Skills Intelligence).