Backlog
Priorities: P0 = next phase, P1 = after, P2 = core modules, P3 = productization,
P4 = optional (Phase 7, value justification required). Status values: todo /
in-progress / done / blocked.
P0 — Phase 1 Foundation (built 2026-07-28)
| ID |
Item |
Status |
| F-01 |
Repo packaging: pyproject.toml, uv lock, src layout, ingoshift package |
done — except uv.lock, deferred: PyPI unreachable in build sandbox (see reviews/phase-1) |
| F-02 |
Typed config: Pydantic v2 loader with validation errors that name file and field |
done (12 schema tests) |
| F-03 |
Structured logging (JSON option), run-ID correlation |
done |
| F-04 |
CLI skeleton (click, per ADR-006 amendment): status, config validate, --version; non-zero exits |
done |
| F-05 |
Test framework: unit, contract, integration and e2e suites; coverage enforced via CI flags |
done locally |
| F-06 |
CI: GitHub Actions — ruff, mypy, pytest matrix (3 OS × py3.11/3.12), docs build |
workflows written and updated 2026-07-29 — first GitHub run pending |
| F-07 |
Validate AGS/NUTS codes against official tables; pin by test |
in-progress — codes pinned by test; official-table verification still open (R-14; six fetch attempts failed on infobox extraction) |
| F-08 |
Docker: compose with PostgreSQL profile; app Dockerfile; .env.example |
done — docker compose config valid; image build/run not executed (no daemon in sandbox) |
| F-09 |
Makefile: bootstrap/test/lint/typecheck/docs/demo/artifacts/live-ted + honest failing API serve stub |
done locally; clean-clone proof transfers to first CI run |
| F-10 |
MkDocs Material site with existing docs; strict build |
done (mkdocs build --strict green) |
| F-11 |
pre-commit: ruff + mypy local hooks, secrets/large-file checks |
config written — hooks not executed in sandbox (pre-commit unavailable) |
Phase 1 acceptance: config validate catches seeded errors ✅ (tested); docs site
builds ✅; stubs honest ✅; tests green locally ✅ — green on all three OS and clean
install from clone transfer to the first CI run (carried-forward obligation).
| ID |
Item |
Status |
| C-01 |
Raw storage abstraction (ADR-011): hashed, timestamped, partitioned, idempotent, replayable |
done (5 unit tests) |
| C-02 |
dim_source + fact_source_run schema and writer; source registry from config/sources/sources.yml |
done |
| C-03 |
Re-verify candidate live sources and record evidence |
partial — TED S-05 verified live 2026-07-29; S-01/S-04/S-08/S-10 remain open |
| C-04 |
Collector 1: GENESIS Bayern → raw → bronze → silver, fixture contract tests |
done as fixture adapter (8 contract tests) — source NOT integrated until a real export replays (C-03) |
| C-05 |
Collector 2: TED → raw → bronze → silver, region filter |
done and verified live 2026-07-29; anonymous official API, raw archived, API-v3 contract covered |
| C-06 |
Quality gate v1: severities, fact_data_quality_run, publication block on critical |
done + proven (poisoned-fixture test: marts not built, exit 3) |
| C-07 |
Warehouse DDL v1 dims/facts |
done on SQLite fallback (ADR-001 amendment); DuckDB/Postgres restoration pending installable env |
| C-08 |
Gold marts |
done as ordered tested SQL models (ADR-002 amendment); dbt migration pending installable env |
| C-09 |
Lineage test: mart number → fact → raw hash → bytes on disk |
done (automated e2e test, exact fixture bytes verified) |
| C-10 |
CLI wired: ingest, validate (raw replay), run-all; make demo real |
done (transform as separate command arrives when multi-step transforms exist in Phase 3) |
P2 — Core modules (Phases 3–5)
M1 Jobs & Skills (Phase 3, built 2026-07-28)
| Item |
Status |
Versioned taxonomy config/taxonomies/skills.yml (30 skills, variant-collision guard) |
done |
| Synthetic postings fixture S-16 per ADR-008 (33 postings, visibly fake employers) |
done |
| Extraction layer 1: dictionary matching, observed strings retained, method recorded (ADR-012) |
done (extraction contract tests incl. all Power-BI spellings) |
Normalization: titles, companies (legal forms), seniority + occupation inference with *_method labels |
done |
| Dedup (normalized title+company+Kreis, earliest wins) |
done + tested |
Warehouse: dim_skill, fact_job_posting, bridge_job_skill |
done |
| Marts: top skills, monthly trends, co-occurrence, company hiring |
done (pinned-number tests) |
| Jobs quality rules incl. taxonomy referential integrity |
done (7 rules) |
| Excel/CSV export with provenance README/manifest (US-15) |
done + tested |
| Dashboard page (Skills Explorer) |
done in the self-contained dashboard, including text filtering and data/source details |
API endpoints (/api/v1/skills, /skills/trends, /jobs) |
open; not required by the deployed static demo |
| Extraction layers 2+ (NLP model, flagged LLM, review queue) |
deferred with written justification (ADR-012) — pointless on synthetic text |
M2 Company Monitor (Phase 4, built 2026-07-28)
| Item |
Status |
| Synthetic two-snapshot company fixture S-17 (merge/override/queue/change cases by construction) |
done |
| Entity resolution (ADR-010): exact key + domain + manual overrides, union-find, NO fuzzy auto-merge |
done (7 unit tests + e2e) |
Audit tables: company_entity_map, company_merge_log, company_review_queue; overrides in config/overrides/company_overrides.yml |
done — CMP-C4 makes missing audit rows a critical failure |
dim_company + content-hashed fact_company_snapshot (first/last_seen historization) |
done |
Change detection: new / removed_from_source / changed_description / new_theme (fact_change_event) |
done + ingoshift detect-changes |
Transformation theme taxonomy config/taxonomies/themes.yml (13 themes, keyword+exclusion rules, matched keyword stored) |
done |
| Transparent indicators mart (ADR-013: no composite index) + company overview mart with cross-module hiring link |
done (pinned-number e2e) |
| Companies quality rules (7) incl. merge auditability + theme referential integrity |
done |
| Curated-real company seed with verifiable evidence |
open; fixture stands in, visibly synthetic (ADR-009) |
| Company Monitor dashboard page + API |
dashboard done with filterable detail profiles; API open |
M3 Tender & Research Radar (Phase 5, built 2026-07-28)
| Item |
Status |
| CORDIS-shaped parser (header-sniffed projects/organisations CSVs, per-line errors) + constructed fixture |
done (6 contract tests) — S-06 NOT integrated until a real export replays (C-03) |
| Regional link step: participations entity-matched to dim_company; projects without regional partner dropped WITH visible count |
done (RES-C5 filter-integrity critical rule) |
fact_research_project, bridge_project_organisation (canonical_id nullable for unmatched partners), bridge_project_theme |
done |
Tender relevance: bridge_tender_theme + mart_tender_themes (matched keyword stored) |
done |
Relationship network as relational edges (vw_network_edges; graph DB deliberately not added per brief) |
done |
| Research quality rules (8) incl. referential + filter integrity |
done |
| Live tender source S-04 (Datenservice Öffentlicher Einkauf) |
still pending-local-check (C-03) |
| Live tender source S-05 (official TED Search API) |
done and verified 2026-07-29; above-threshold coverage only |
P3 — Phase 6 Productization (in-environment scope built 2026-07-28)
| Item |
Status |
Reproducible weekly report from warehouse (ingoshift report weekly; as-of from stored runs, zero wall-clock values, byte-identical regeneration tested) |
done + sample committed in reports/ |
Data dictionary generated from live schema (scripts/gen_data_dictionary.py, 35 objects) |
done |
| Architecture doc with Mermaid (context, layers, star schema, ingestion lifecycle) |
done |
| Case study (brief's 10 points) |
done |
| Analysis report (demo-labelled, lineage path per finding, alternative explanations) |
done |
| Upwork/portfolio assets (no prices, per brief) |
done |
Sample exports committed (data/samples/: XLSX + CSVs + manifest, regenerable) |
done |
Security pass (secret-pattern sweep clean, .env ignored) + performance note (demo run ~1s) |
done |
| Deployment recipes |
documented; local dashboard verified; Docker profile remains unproven |
| Dashboard screenshot and committed demo |
done; interactive filters/detail views added 2026-07-29 |
| GitHub Pages workflow |
done locally; first public deployment pending repository remote |
| OpenAPI link |
open because no API is implemented or claimed |
| Excel/CSV exports |
done since Phase 3; Parquet variant deferred until pyarrow installable |
P4 — Optional (each needs written value justification first)
Knowledge graph (NetworkX export first) · source-grounded document assistant with
citation + abstention tests · forecasting with rolling-origin evaluation (only if
history suffices — else a written "why not" note) · alerting/email reports.