Modelling & mapping suite
Stage 2 of the pipeline — the missing middle. Modelling turns an approved requirements-catalog.yaml (the discovery output) into the source-to-target mapping files the build suite consumes. On a traditional engagement this bridge between requirements and build is done by hand; here three agents chain together — SourceProfiler → Modeller → SourceToTargetMapper — passing the profile report, then the logical model, then the mapping files down the line. The newest of the three suites: its first scored runs landed on 12 July (see Runs).
The three agents
| Agent | Role |
|---|---|
SourceProfiler | reads and profiles the real source schemas (strictly read-only) to ground the catalogue's source and data-quality sections |
Modeller | turns approved requirements into entities, relationships, conformed dimensions and facts, keys, and SCD intent — the conceptual and logical model |
SourceToTargetMapper | combines the logical model with the profiled source schemas to emit the gold mapping files build consumes |
Design principles
- Source access is read-only by construction — a connector protocol with no write path: an in-memory SQLite connector for synthetic sources in tests, and a Fabric connector for real schemas and data.
- Profiling is deterministic code, not model output — null rates, cardinality, candidate keys, PII flags and referential gaps are computed, so the model reasons over facts rather than inventing them.
- The logical model schema mirrors the real catalogue — the same
logical_modelblock the Meridian worked example carries. - Fully testable offline — sources are mocked with synthetic SQLite shaped like the real systems, so the whole suite runs without a live Fabric connection.
Deep dive: what a run produces
A modelling run leaves three artefacts in the engagement repo, each consumed by the next step. The profile report is deterministic facts about the real sources — row counts, null rates, candidate keys, referential gaps — that ground everything downstream. The conceptual and logical model maps approved requirements to entities, relationships, conformed dimensions and facts, keys and SCD intent; it uses the same logical_model schema the requirements catalogue carries, so it cross-references catalogue items by ID. The source-to-target mapping files are the handover contract: one per gold table, precise enough that the build suite can turn each into a tested Fabric table without human translation.
Because the catalogue and the logical model are platform-agnostic, this whole stage would survive a change of target platform — only the mappings' consumer (build) is Fabric-specific. The scored runs under Runs show both modes: modelling from the golden catalogue, and modelling from a live discovery output.
Current limitations
The mapper currently emits gold-layer dimension and fact mappings — generating bronze-ingest and silver-conform mappings is the natural next step — and profiling-derived data-quality findings are not yet first-class typed citations. Both are tracked and honest: the run scorecards under Runs show exactly where the suite stands today.