Skip to content

Build Log

Chronological, append-only. Every entry states what was actually executed and what was not.

2026-07-28 — Phase 0: Discovery and feasibility

Done: - Read and decomposed the master brief; produced the required first response (understanding, feasibility, risks, architecture, plan, default decisions). - Source discovery with live verification where the environment allowed: - Fetched GENESIS-Online Bayern product page (downloads confirmed); webservice endpoint returned 403 to the sandbox fetcher. - Fetched bundesAPI/jobsuche-api README (reverse-engineered status + static key confirmed; no terms) → ADR-008. - Fetched TED API documentation (Search API anonymous + free confirmed). - Fetched CORDIS H2020 dataset metadata via data.europa.eu API (EC reuse notice, CSV/XLSX distributions confirmed). - Confirmed existence of BA AA-Ingolstadt statistics pages, Stadt Ingolstadt statistics publications, OffeneRegister data page, Förderkatalog UI via search. - Attempted oeffentlichevergabe.de, thi.de, statistikdaten.bayern.de webservice, query.wikidata.org — blocked/timeout from sandbox; recorded as pending-local-check, not as source failures. - Wrote Phase 0 deliverables: charter, source catalogue (15 sources with statuses and evidence), data governance, user stories (16), backlog, risk register (14), ADRs (11), definition of done, limitations, region config, README, LICENSE. - Ran the four-review standard on Phase 0 itself → docs/reviews/phase-0.md. - Initialized git repository; first commit.

Not done / explicitly deferred: - No code, no pipeline, no tests exist yet (by design — Phase 0 is documentation). - Licenses for S-01/S-02/S-05 recorded as expectations, to be confirmed at integration time. - Four sources await re-verification from a non-sandboxed network (backlog C-03).

Next: Phase 1 Foundation per backlog P0 (F-01…F-11).

2026-07-28 — Phase 1: Foundation

Executed and verified (exact commands, build sandbox): - ruff format . / ruff check . (ruff 0.15.11) — clean, 12 files. - mypy strict — Success: no issues found in 7 source files. Caveat: the sandbox mypy (1.20.2) runs in an isolated tool env where the pydantic.mypy plugin cannot import; the run used an equivalent strict config resolving the system interpreter's packages. CI runs the committed config with plugin. - pytest (9.0.3) — 36 passed (config schema 12, loader 7, logging 4, CLI 7, plus parametrized cases), via pytest pythonpath=src and the system site-packages (pydantic 2.13.3, click 8.3.3, PyYAML 6.0.3). - mkdocs build --strict (mkdocs-material 9.7.6) — green. - docker compose config — valid (with .env from .env.example).

Environment constraint hit: PyPI returned 403 (direct and via proxy) → no pip install / no uv lock possible in this sandbox. Handled by: click instead of Typer (ADR-006 amendment), coverage flags moved to the CI command so plain pytest works without pytest-cov locally, lockfile deferred (F-01 note).

Explicitly NOT executed (defined only, no success claimed): make bootstrap venv/pip path, GitHub Actions matrix, Docker image build (no daemon), pre-commit hooks (tool unavailable). First GitHub push proves the first three.

F-07 attempt: six fetches for official AGS/NUTS confirmation failed on infobox/table extraction; codes stay pinned-by-test + flagged (R-14).

Next: Phase 2 core data platform (C-01…C-10) — raw storage abstraction, source-run tracking, first two collectors (fixture-tested), quality gate v1.

2026-07-28 — Phase 2: Core data platform

Executed and verified (build sandbox, exact commands): - ruff format . / ruff check . — clean (32 files). - mypy strict (sandbox-equivalent config, system-interpreter packages) — Success: no issues found in 21 source files. - pytest — 79 passed (raw store 5, GENESIS contract 8, TED contract 8, quality 10, warehouse 4, e2e pipeline 8, CLI 8, plus Phase-1 suites). - Real demo run from repo root: python -m ingoshift run-all → run b37e7dac9c3e, genesis_bayern raw=1 bronze=18 silver=12 (6 out-of-region filtered), ted raw=1 bronze=4 silver=3, 12 quality rules, 0 critical failed, gate=PASS, 4 marts built; ingoshift status reads the warehouse back (fact_employment=12, fact_tender=3, gate PASS). - mkdocs build --strict — green (with phase-2 review in nav).

Built: immutable content-addressed raw store with replay; source registry (config/sources/sources.yml); GENESIS Bayern and TED fixture adapters with parser contracts (GENESIS marker conventions handled: -→semantic zero, x/./...//→masked with marker retained); silver region filter + dedupe; SQLite warehouse v1 (dims, facts, source runs, quality runs) with portable SQL; quality gate v1 (12 rules, critical/warning/info) that blocks mart builds and exits 3 on critical failure; ordered SQL marts incl. completeness-flagged region totals; lineage e2e test (mart value 101870 → raw sha256 → fixture bytes); CLI ingest/validate(replay)/run-all; make demo now real.

Deviations, all dated in decisions.md: SQLite instead of DuckDB (ADR-001), SQL-model marts instead of dbt (ADR-002), plain functions instead of Prefect (ADR-003) — every one caused by the blocked package index, none silent.

Explicitly NOT done: live collection for any source (S-01/S-05 adapters are fixture-only and say so; POST-based TED search not possible from sandbox); C-03 local re-verification; CI first run; DuckDB/Postgres restoration; uv.lock.

Next: Phase 3 Jobs & Skills Intelligence (module 1) — taxonomy, layered extraction on labelled fixtures per ADR-008, trend/co-occurrence marts, dashboard page, API endpoints, export.

2026-07-28 — Phase 3: Jobs & Skills Intelligence (Module 1)

Executed and verified (build sandbox): - ruff clean, mypy strict clean (28 source files), pytest 145 passed (taxonomy 5, normalization 26 parametrized, extraction contract 13, jobs parser 4, jobs quality 7, module e2e 10, plus all prior suites), mkdocs build --strict green. - Real run: 3 sources, synthetic_jobs raw=1 bronze=33 silver=31 (filtered=1, deduped=1), 19 quality rules, gate=PASS, 8 marts; ingoshift export --format xlsx wrote data/exports/ingoshift_marts.xlsx (README provenance sheet + 7 mart sheets); status shows dim_skill=30, fact_job_posting=31, bridge_job_skill=75.

Built: versioned taxonomy (30 skills; variant-collision guard); dictionary extraction with observed-string retention and per-mention method (ADR-012); title/company normalization incl. legal-form stripping that preserves the (SYNTHETIC) marker; seniority/occupation inference with explicit *_method labels; dedup (earliest wins); dim_skill/fact_job_posting/bridge_job_skill; marts top-skills/monthly/co-occurrence/company-hiring (pinned-number tests: python=10, sql=9, power_bi=5 over four retained spellings, python+sql=4, python monthly 2/2/2/4); 7 jobs quality rules incl. taxonomy referential integrity; XLSX/CSV export with provenance (US-15); CLI export command; validate/run-all extended to jobs.

Explicitly NOT done (environment-blocked, in backlog): Skills Explorer dashboard page (Streamlit) and API endpoints (FastAPI) — neither installable here; extraction layers 2+ deferred with written justification (ADR-012).

Next: Phase 4 Company & Transformation Monitor — company dim + snapshots, ADR-010 entity resolution, change detection, transformation taxonomy with transparent indicators.

2026-07-28 — Phase 4: Company & Transformation Monitor (Module 2)

Executed and verified (build sandbox): - ruff clean, mypy strict clean (34 source files), pytest 169 passed (entity resolution 7, change detection 3, themes 5, companies parser/quality, module e2e 9, plus all prior suites), mkdocs build --strict green. - Real run: 4 sources; synthetic_companies raw=2 (two snapshot files) bronze=18 silver=18; 10 canonical entities, 1 open review pair, 5 change events; 27 quality rules, gate=PASS, 11 marts; ingoshift detect-changes lists new/removed/changed/new-theme events.

Built: synthetic two-snapshot company fixture (S-17) with constructed merge/override/queue/change cases; deterministic entity resolution (exact name key + domain + manual override; union-find; NO fuzzy auto-merge — similar pairs go to company_review_queue); full merge audit (company_entity_map, company_merge_log; CMP-C4 makes unaudited assignments a critical failure); dim_company with first/last_seen; content-hashed fact_company_snapshot; change detection with honest semantics (removed_from_source = source-level only); 13-theme transformation taxonomy with keyword+exclusion rules and stored matched keyword; component-only indicators mart (ADR-013 — deliberately no composite index); company overview mart linking synthetic-jobs hiring via shared entity keys; 7 companies quality rules; CLI detect-changes.

Environment incident (disclosed): one ruff --fix invocation ran from the wrong working directory and modified ~3900 .py files outside the repository, including preinstalled site-packages; unused-import "fixes" broke re-export chains in openpyxl and pathspec. Repaired from local evidence (restored openpyxl xml/functions.py re-exports and drawing/geometry.py LineProperties, pathspec re2 fallback initialization; helper script documented) and verified by openpyxl write/read roundtrip, the full test suite, and mkdocs build --strict. Residual risk: other preinstalled packages we do not import may still carry removed re-exports — irrelevant to this project's toolchain, noted for transparency. Prevention: all lint/format runs now execute from the repository root only.

Fixture-earned bug fix: display-name heuristic preferred a one-character- longer typo; switched to most-frequent-then-longest (caught by e2e).

Explicitly NOT done: curated-real company seed (needs verifiable per-company evidence — impossible from sandbox); dashboards/API (unchanged blocker); no composite transformation index (by decision ADR-013, not by gap).

Next: Phase 5 Tender & Research Radar (module 3) — CORDIS research model, relationship bridges, tender/research relevance classification, deadline views.

2026-07-28 — Phase 5: Tender & Research Radar (Module 3)

Executed and verified (build sandbox): - ruff clean, mypy strict clean (35 source files), pytest 187 passed (CORDIS contract 6, research quality 6, research e2e 6, plus all prior), mkdocs build --strict green. - Real run: 5 sources; cordis raw=2 (projects + organisations CSVs) bronze=26; 6 projects regionally linked and kept, 2 dropped (visible count); 11 of 14 kept participations entity-match regional companies, 3 foreign partners retained unmatched; 35 quality rules, gate=PASS, 14 marts incl. mart_research_projects, vw_network_edges, mart_tender_themes.

Built: CORDIS-shaped bulk-CSV parser (header sniffing, per-line errors, nullable amounts per brief); regional link step over resolved companies with filter-integrity as a critical rule (RES-C5); fact_research_project + bridge_project_organisation (nullable canonical_id keeps unmatched partners as facts) + bridge_project_theme; tender theme relevance (bridge_tender_theme, matched keyword stored; exactly one regional notice classifies — word-boundary under-matching preferred over silent over-matching); the relationship network as relational edges (no graph DB, per brief); 8 research quality rules.

Explicitly NOT done: S-04/S-06 live integration (C-03 unchanged); dashboards/API (unchanged blocker); no forecasting, no graph database, no composite scores.

Next: Phase 6 productization — in-sandbox scope: reproducible weekly report from warehouse data, data dictionary, architecture diagrams (Mermaid), case study, analysis report (fixture-labelled), Upwork/portfolio assets, security/limitations review; deployment recipe documented (execution needs a network-enabled environment).

2026-07-28 — Phase 6: Productization (in-environment scope)

Executed and verified (build sandbox): - ruff clean, mypy strict clean (36 source files), pytest 192 passed (reporting e2e 5 incl. byte-reproducibility), mkdocs build --strict green with expanded nav. - Generated for real: reports/weekly_report_*.md (as-of derived from stored run timestamps — zero wall-clock values), docs/data-dictionary.md via schema introspection (35 objects), data/samples/ XLSX + 11 CSVs + manifest. - Security sweep: no secret patterns; .env ignored; only full-profile secret is the local Postgres password. Performance note: full demo run ≈1 second at demo scale.

Written: docs/architecture.md (Mermaid: context, layers, star schema, ingestion lifecycle), docs/deployment.md (recipes explicitly NOT executed here, each with its verification step), docs/case-study.md (brief's 10 points), reports/analysis-report.md (demo-labelled findings with lineage paths, observation-vs-inference labels, alternative explanations), docs/portfolio/upwork.md (no prices, per brief).

Explicitly NOT done (environment-blocked, tracked): dashboard screenshots / OpenAPI link / deployed demo (no dashboard/API runnable here); Parquet export (pyarrow); everything on the standing carried list (CI first run, C-03 live checks, DuckDB/Postgres restoration, curated company seed).

Handover state: Phases 0–6 complete for everything this environment can execute; remaining work requires a network-enabled machine — see backlog and README "Roadmap". Optional Phase-7 features stay ungated until core acceptance criteria (incl. the environment-blocked items) pass, per brief.

2026-07-29 — Post-review improvements (external review response)

An independent review (external AI, ran the project from the delivered zip) confirmed the engineering claims from a clean clone — 192 tests passing, ~94 % coverage, ruff/mypy/docs clean, demo pipeline working — and correctly criticized the missing visible product. Addressed in-environment:

  • Static self-contained HTML dashboard (ingoshift dashboard, ADR-004 amendment): inline SVG charts, zero scripts/external resources, demo banner mandatory, byte-reproducible (all tested); generated sample committed at docs/demo/dashboard.html → publicly clickable via mkdocs gh-deploy on GitHub Pages. Real screenshot (headless Chromium) committed at docs/assets/dashboard.png and embedded in the README.
  • Windows quick start in the README (PowerShell, no make required) + make artifacts convenience target.
  • Demo video script (docs/portfolio/demo-video-script.md) — every spoken claim is demonstrated on screen in the same breath.
  • Author identity hygiene: commit author/committer rewritten to Velat Isik <velat-isik@users.noreply.github.com> before any public push (the portfolio alias must not leak a private email via git metadata). Note: commit hashes changed; nothing had been pushed.

Verified after changes: ruff clean, mypy strict clean (37 files), 199 tests passed, mkdocs build --strict green (dashboard + screenshot included in the site build).

Still environment-blocked (unchanged): first CI run, interactive dashboard/API, live source integration, DuckDB/Postgres, curated-real seed.

2026-07-29 — Post-review implementation

Implemented the four concrete gaps identified in the external review:

  • Integrated the official anonymous TED Search API as a real live collector. A verification run archived the raw response, parsed 38 current Region-10 notices, passed 35 quality rules, and built the marts. The record count is time-dependent and is not committed as a fixed claim.
  • Added source-selective live execution: ingoshift run-all --live --source ted.
  • Expanded the static dashboard with global text search, NUTS filtering, company profiles, tender/source details, official links, and distinct generated-at versus source-retrieval coverage.
  • Added GitHub Pages generation/deployment workflow and updated the CI actions. These workflows remain externally unproven until the repository is pushed.
  • Reconciled README, deployment, limitations, governance, catalogue, backlog, case study, and portfolio copy with the actual implementation.

Local verification: Ruff formatting/linting, strict mypy, focused collector and dashboard tests, complete pytest suite, strict MkDocs build, regenerated demo, and browser-based visual/interaction review.

2026-07-30 — Bilingual dashboard redesign

  • Reworked the generated dashboard into a product-style interface with a dark compact navigation rail, editorial hero, refined KPI hierarchy, responsive cards, and technical metadata behind disclosure controls.
  • Added a persistent EN/DE language switch for navigation, headings, controls, tables, data classes, method notes, and accessibility labels.
  • Corrected German presentation strings to native Unicode spelling, including Eichstätt, für, Qualitätsprüfung, Steuergeräte, Übersicht, Fähigkeiten, and Öffentliche Aufträge. Stored source values remain immutable.
  • Verified the German interaction state and a 390 px mobile viewport in the browser; the page has no horizontal document overflow. Final verification: 207 tests passed at 93.71% coverage, Ruff and strict mypy clean, strict MkDocs build green.