Skip to content

Data Governance and Legal Collection Rules

Status: binding for all collectors. Updated: 2026-07-29

Collection rules (checked before any collector is written)

  1. Prefer, in order: official API → open-data portal → documented bulk download → RSS/feed → public HTML pages only as last resort.
  2. Review robots directives and terms of use for every source; record the result and date in source-catalogue.md. A collector without a recorded review does not ship.
  3. Never access login-protected content; never bypass rate limits, CAPTCHAs, or technical access controls.
  4. Respect explicit rate limits; where none are stated, default to conservative throttling (≥1s between requests, exponential backoff, identifying User-Agent with contact possibility).
  5. If a source cannot be used responsibly, it is not scraped. Instead: a documented adapter interface plus a labelled fixture demonstrating the intended schema.
  6. A source is only claimed "integrated" when its collector and tests exist and a representative request has been executed and recorded.

Personal data policy

  • The business questions concern organizations, skills, and markets — not people.
  • No collection of private contact details, applicant data, or person-level enrichment. Person names occurring incidentally in source text (e.g., a contact person in a tender notice) are not extracted into structured fields.
  • Job postings, where ever ingested, are processed for employer, role, skills, location, and terms — never for individuals.
  • GDPR stance: process only data made public by organizations about organizations; document lawful-basis reasoning per source before ingestion; honor takedown requests (contact note in README once published).

License handling

  • Per-source license/terms recorded in source-catalogue.md; unclear terms = treat as restricted until clarified.
  • Attribution obligations (e.g., Datenlizenz Deutschland – Namensnennung, EC reuse notice, ODbL for OpenStreetMap, CC0 for Wikidata) are stored as machine-readable fields on dim_source and rendered wherever derived data is displayed or exported.
  • ODbL caution: OSM-derived databases trigger share-alike obligations. Decision: OSM data, if used, stays in a clearly separated layer with attribution, and we do not mix it into proprietary-style exports without legal review (see risk R-08).
  • Raw retrieved data is never redistributed via the public repository. Only small, legally redistributable, clearly labelled fixtures live in data/fixtures/.

Implemented live source

The TED Search API integration was verified end to end on 2026-07-29. It uses the official anonymous endpoint, requests only the fields required by the warehouse, stores the response before parsing, and extracts no contact-person fields. The request is bounded to configured Region-10 NUTS codes, current tender deadlines, and 100 records. Notice reuse and attribution obligations are recorded in the source catalogue and source configuration.

Historical environment constraint

Phase 0 verification ran from a cloud sandbox whose outbound fetcher is blocked by some German government portals (observed 2026-07-28: oeffentlichevergabe.de timeout, statistikdaten.bayern.de webservice HTTP 403, thi.de timeout, query.wikidata.org proxy-rejected). These are sandbox limitations, not source rejections — each affected source carries status pending-local-check in the catalogue and must be re-verified from a normal network before its collector is built. TED was subsequently re-verified successfully; the other named sources remain open.

Fixture policy

  • Fixtures are small (≤ a few hundred records), legal to redistribute, and carry a data_class: synthetic or data_class: fixture marker plus provenance comment.
  • Synthetic data is generated to be realistic but visibly fake (e.g., company names like "Beispiel Antriebstechnik GmbH (SYNTHETIC)") — it must never be mistakable for a real observation, and demo_mode: true renders a banner in every output that includes it.