Design: unit-aware datasets across parsers, query API, and plots

[micahpw]

Motivation

GAT's core promise is comparing results across different modeling tools (Sienna, PLEXOS, ReEDS) through one API. That comparison is only trustworthy with unit-level safety: today units are handled by convention — e.g. the convert_units() helper in quickplots, Sienna base-power normalization at ingest, and the 'ratings stored in MW' convention for branch/generator tables. A tool that natively reports MW compared against one reporting per-unit or GWh is a silent-error class this design should eliminate.

Architectural reality

No mainstream engine respects Arrow custom metadata through compute — pandas, polars, and DuckDB all strip it. So there is no 'units flow through the engine for free' design available. Units must be enforced at the boundaries and deliberately re-attached after any engine pass:

entry (parser/ingest)          in-memory compute        exit (save/publish/send)
UnitArrow ──────────────► pint-pandas ──────────────► UnitArrow ──► parquet / Arrow IPC / client
                ▲                                        │
                └----- re-hydrate after DuckDB ----------┘

• UnitArrow owns the entry and publishing points: unit metadata attached when data is parsed/ingested, and re-attached when results are saved, sent over Arrow IPC, or published.
• pint-pandas owns in-memory compute between boundaries, where the ergonomics and safety checks matter.
• Because DuckDB strips metadata, GAT needs a persisted per-dataset/column unit registry (natural home: alongside the category maps written at ingest) so query results can be re-hydrated into UnitArrow/pint after the engine pass.

Implication for DuckDB's role

Taken seriously, this contract demotes DuckDB from "the data engine" to storage + transport prep + known-shape aggregation — because a metadata-stripping engine inherently breaks the unit chain, and registry re-hydration only works when the operation's dimensional effect is known:

• Rescuable: query_grouped()-style operations (group-by + sum/mean over declared columns) — summing MW is still MW; the registry can re-hydrate outputs with confidence.
• Not rescuable: the raw-SQL paths (Scenario.query(), the server /query endpoint). A user-written SELECT a * b mints new units no registry can infer. Unit enforcement there is structurally impossible, not just unimplemented.

So under this design, unit-bearing compute happens in pint-pandas after the engine pass (or in a future UnitArrow engine), DuckDB's sanctioned surface narrows to dimensionally-safe operations, and raw SQL remains available as a documented escape hatch where the unit contract is explicitly void — caveat emptor, clearly flagged in returned metadata.

This is worth stating plainly because the v1 architecture deliberately moved toward DuckDB; the units contract doesn't reverse that for storage and retrieval, but it does cap how much compute should ever live in SQL.

The real open question

Whether a simple data engine built on top of UnitArrow (unit-respecting transforms, so the strip/re-attach dance disappears for common operations) is worth building — and if so, whether GAT consumes it. That decision belongs upstream in UnitArrow; GAT's design should work with the boundary pattern above either way, and get simpler if such an engine lands.

Contract to converge on

By the time data reaches a plotting function, it is pint-pandas or UnitArrow — never bare floats. Plot signatures become the enforcement edge; convert_units()-style helpers become internal details.

This also unlocks a uniform units= parameter on every plotting function. A handful of plots already accept units='GWh'/units='TWh' kwargs, but they assume the input is MWh by convention — a wrong assumption converts silently. With unit-carrying inputs, units= becomes a checked conversion (input_quantity.to(units)) that raises on dimensional mismatch, and it can be offered consistently across the whole plotting API instead of ad hoc per function.

Plugin contract enforcement

The comparison guarantee is only as strong as the least-disciplined parser, so unit declaration must be part of the plugin contract, not a convention. If someone contributes a PyPSA extension, declaring the units of every dataset their parser produces should be a requirement to register, on the same footing as implementing list_datasets() / get_dataset():

• The parser/system base interfaces (BaseSimulationParser, BaseSystem) gain a required unit declaration — e.g. dataset schemas carry per-column units, enforced as part of the abstract interface rather than documented as a guideline.
• Validation happens at the existing choke points: gat_ext entry-point loading and ingest. A dataset without declared units is rejected (or requires an explicit dimensionless opt-out — silence is not an option).
• This is what makes third-party tools first-class in cross-tool comparison: a Sienna-vs-PyPSA comparison is trustworthy for exactly the same reason a Sienna-vs-PLEXOS one is — both sides declared units at the boundary and GAT checked them.

The plugin development guide would document this as part of the parser authoring contract.

Open questions

• Where exactly is the pandas edge: Scenario.query() returns pint-pandas, or a .units accessor?
• Interaction with base-power normalization (Sienna) and per-tool native units
• Performance on large hourly timeseries (pint-pandas overhead per column)
• Migration: additive unit registry first, breaking plot-signature change later

Not scheduled — filed as a design discussion for the post-v0.1 roadmap.