feat(zarr-metadata): add metadata model layer

.github/workflows/zarr-metadata.yml

@@ -82,7 +82,10 @@ jobs:
       - name: Sync test dependency group
         run: uv sync --group test --python 3.11
       - name: Run pyright
-        run: uv run --group test --with pyright pyright src
+        # Pinned to the last version that types PEP 661 sentinels in class
+        # attributes correctly; 1.1.405+ regressed (microsoft/pyright#11115).
+        # Unpin when the fix lands.
+        run: uv run --group test --with 'pyright==1.1.404' pyright src
 
   zarr-metadata-complete:
     name: zarr-metadata complete

docs/superpowers/specs/2026-07-22-zarr-metadata-review-fixes-design.md

@@ -0,0 +1,118 @@
+# Zarr Metadata Review Fixes Design
+
+## Goal
+
+Correct the significant conformance and public-contract defects found by the
+independent branch review without changing the documented ability to construct
+temporarily invalid model instances for metadata repair workflows.
+
+## Scope
+
+This change fixes four areas:
+
+1. Validation of raw Zarr v2 `.zarray` and `.zgroup` documents must be distinct
+   from validation of the library's merged in-memory document representation.
+2. Public `TypeIs` predicates must narrow only values that actually inhabit the
+   type named in their return annotation.
+3. The public Zarr v2 dtype type must represent nested structured dtypes.
+4. Pydantic-generated JSON schemas must encode the structural constraints that
+   the corresponding runtime parsers enforce.
+
+The inaccurate consolidated-metadata docstring is minor and outside this fix.
+Existing model-instance pass-through in the Pydantic integration remains
+unchanged because it is part of the documented repair-state design.
+
+## Raw V2 Documents and Merged Models
+
+The public `ZarrV2ArrayMetadataJSON` and `ZarrV2GroupMetadataJSON` types describe
+an in-memory representation in which an optional `attributes` member has been
+merged from `.zattrs`. They do not describe the raw `.zarray` and `.zgroup`
+objects stored on disk.
+
+Private raw-document parsers will enforce the storage specification before any
+merge occurs:
+
+- `.zarray` accepts exactly the required array metadata members plus the
+  optional `dimension_separator` member. Every other member, including
+  `attributes`, is an error.
+- `.zgroup` accepts exactly `zarr_format`. Every other member, including
+  `attributes`, is an error.
+- `.zattrs`, when present, must be a JSON object with string keys and JSON
+  values. It is then added as the merged `attributes` member.
+- An absent `.zattrs` remains distinguishable from an explicitly empty object,
+  preserving the existing `UNSET` behavior.
+
+The existing public merged-document validators remain strict. This avoids
+silently discarding data and keeps their declared `TypedDict` contracts sound.
+
+## Sound Type Guards
+
+Validation and parsing intentionally accept abstract `Mapping` and `Sequence`
+inputs. Parsers materialize those inputs into canonical dictionaries and
+tuples. Type guards cannot make the same promise because `TypeIs[T]` asserts
+that the original object already inhabits `T`.
+
+`is_json` will therefore require recursively canonical JSON container types:
+`dict`, `list`, and `tuple`, with string dictionary keys and canonical children.
+It will reject abstract containers such as `UserDict` and `range`, even though
+`parse_json` continues to accept and normalize them.
+
+`is_metadata_field_v3` will likewise require either a string or a concrete
+dictionary whose nested values are already canonical. Parsing continues to
+accept abstract mappings and materialize them.
+
+## Recursive V2 Dtype Type
+
+`ZarrV2DataTypeMetadata` will be a named recursive alias. A structured field's
+datatype may be either a dtype string or another structured dtype tuple. The
+optional subarray shape remains a tuple of integers. Runtime validation already
+accepts this structure, so this change aligns static typing and generated schema
+with existing spec-compliant behavior.
+
+## Pydantic JSON Schema Parity
+
+The Pydantic field aliases will retain their current runtime behavior: raw
+documents are parsed through the core model constructors, and existing core
+model instances pass through unchanged.
+
+Their JSON schema input definitions will be replaced or annotated with
+constraints that express the runtime document rules, including:
+
+- non-negative shape and chunk dimensions;
+- at least one v3 codec;
+- closed named-configuration objects;
+- closed v2 merged documents and consolidated payloads where runtime parsing is
+  closed;
+- `must_understand` constraints at mandatory v3 extension points; and
+- the recursive v2 structured-dtype representation.
+
+Constraints that depend on semantic interpretation outside this package's
+structural validators will not be added. The generated schema should describe
+accepted raw document inputs, not the intentionally permissive state of an
+already-constructed core model instance.
+
+## Error Handling
+
+All new raw-v2 failures use `MetadataValidationError` and preserve precise
+locations. An illegal `.zarray` member named `attributes`, for example, reports
+the location `("attributes",)` and the existing `invalid_value` kind.
+Malformed `.zattrs` values report locations beneath `attributes` after the
+merge boundary, consistent with the public merged representation.
+
+## Testing
+
+Implementation follows red-green-refactor cycles. Regression coverage will
+prove:
+
+- raw `.zarray` and `.zgroup` reject `attributes` and other extra members;
+- valid sibling `.zattrs` still merges, including absent-versus-empty behavior;
+- `is_json(range(3))` and `is_metadata_field_v3(UserDict(...))` are false while
+  the corresponding parsers still normalize those inputs;
+- nested structured v2 dtypes type-check through Pydantic and validate at
+  runtime; and
+- generated schemas reject representative inputs that runtime parsing rejects:
+  empty codecs, negative dimensions, forbidden extra members, and unsupported
+  `must_understand: false` values.
+
+Focused package tests, formatting, linting, type checking where configured, and
+the full repository test suite will run before the implementation is committed.

packages/zarr-metadata/README.md

@@ -9,36 +9,58 @@ JSON shapes specified by the [Zarr v2](https://zarr-specs.readthedocs.io/en/late
 and [Zarr v3](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html)
 specifications, plus types for [`zarr-extensions`](https://github.com/zarr-developers/zarr-extensions/)
 and a few widely-used-but-unspecified entities (e.g. consolidated metadata).
+It also provides canonical frozen-dataclass models, structural validators,
+parsers, store-key serialization, and optional Pydantic field integrations.
+The optional integration requires Pydantic 2.13 or newer.
 
 ## What this is for
 
-These types describe the JSON shape of Zarr metadata. They are
-intended for libraries that **read, write, validate, or transform**
-Zarr metadata. Pair them with a runtime validator like
-[pydantic](https://docs.pydantic.dev/) to check JSON loaded from disk:
+The public `TypedDict` definitions describe the static JSON shape of Zarr
+metadata. For strict, loc-aware validation of JSON loaded from disk, use the
+model parser:
 
 ```python
 import json
-from pydantic import TypeAdapter
-from zarr_metadata.v3.array import ArrayMetadataV3
+from zarr_metadata.model import ZarrV3ArrayMetadata
 
 with open("zarr.json", "rb") as f:
     raw = json.load(f)
 
-metadata = TypeAdapter(ArrayMetadataV3).validate_python(raw)
+metadata = ZarrV3ArrayMetadata.from_json(raw)
 ```
 
-## What this is *not*
+The optional Pydantic integration delegates raw input to the same strict
+parser and returns the same normalized model class:
 
-- Not a parser or builder. There are no `make_array_metadata(...)` factories —
-  that surface belongs to consumer libraries.
-- Not a runtime validator on its own. Pair with `pydantic`, `msgspec`, or
-  similar to enforce shapes at decode time.
+```python
+from pydantic import TypeAdapter
+import zarr_metadata.pydantic as zmp
+
+metadata = TypeAdapter(zmp.ZarrV3ArrayMetadata).validate_python(raw)
+encoded = metadata.to_key_value()["zarr.json"]
+```
 
-Even with a runtime validator, these types only describe **structural**
-shape — they will not flag *semantically* invalid metadata, like a 3D v3
-array whose `dimension_names` has 4 entries instead of 3. That's a job
-for downstream validator routines.
+A bare `TypeAdapter` over a public document `TypedDict` is a coercive shape
+adapter, not a Zarr conformance validator; it may coerce values or discard
+members that the strict model parser rejects.
+
+## Validation boundary
+
+The model validators enforce the declared document structure and a small set
+of context-free consistency rules, including fixed format literals, finite
+JSON numbers, non-negative dimensions, non-empty v3 codec pipelines, and one
+`dimension_names` entry per array dimension. They do not interpret extension
+names or configurations, resolve codec pipelines, or decide whether a data
+type, chunk grid, codec, or storage transformer is supported. Those decisions
+belong to consumer implementations.
+
+The Pydantic integration's generated JSON Schemas express independently
+checkable document structure and field constraints, but they are not a
+replacement for runtime model validation. Standard JSON Schema treats a
+mathematically integral number such as `1.0` as an integer, while the runtime
+boundary requires Python `int` values, and it cannot express arbitrary
+same-length relations such as `dimension_names` versus `shape` or v2 `chunks`
+versus `shape`. Consumers should run the model parser after schema validation.
 
 ## Scope
 

packages/zarr-metadata/changes/4119.feature.md

@@ -0,0 +1,79 @@
+Added `zarr_metadata.model`: frozen-dataclass models (`ZarrV2ArrayMetadata`,
+`ZarrV3ArrayMetadata`, `ZarrV2GroupMetadata`, `ZarrV3GroupMetadata`,
+`ZarrV2ConsolidatedMetadata`, `ZarrV3ConsolidatedMetadata`, `ZarrV3NamedConfig`)
+that are canonical, semantically lossless representations of Zarr metadata
+documents, plus structural validators (`validate_*` / `is_*` / `parse_*`).
+Every v3 extension point (data type, chunk grid, chunk key encoding, codecs,
+storage transformers) is held as `ZarrV3NamedConfig`: a name, configuration,
+and `must_understand` obligation; nothing is interpreted. On the wire, an
+empty configuration with the default obligation uses the spec's plain-string
+shorthand. Model fields are annotated with the role alias
+`ZarrV3MetadataField` (today exactly `ZarrV3NamedConfig`), so annotations
+convey the logical meaning and stay put if the spec adds another field form.
+
+Validation is strict about what the types declare: v2 `dtype` / `order` /
+`compressor` / `filters` / `dimension_separator` shapes and the fixed
+`zarr_format` / `node_type` literals are all enforced. Every
+`ValidationProblem` carries a machine-readable `kind`
+(`missing_key` / `invalid_type` / `invalid_value` / `invalid_json`) so
+consumers can dispatch on the failure mode without matching message strings,
+and every ingestion failure — including missing store keys and undecodable
+bytes in `from_key_value` — surfaces as `MetadataValidationError`. An
+adversarial review added further structural checks: JSON booleans are not
+accepted as dimension lengths, dimensions are non-negative,
+`dimension_names` must have one entry per dimension of `shape`, `attributes`
+and `configuration` values are JSON-checked recursively (like `fill_value`),
+non-finite floats and non-standard JSON constants are rejected, abstract
+mappings and sequences normalize to encoder-safe canonical containers,
+v2 `shape` and `chunks` must have the same rank, non-null v2 filter pipelines
+contain at least one filter, document `TypeIs` guards only narrow values that
+already use the declared canonical containers,
+and the inline consolidated-metadata envelope and entries are deep-validated
+so the group validator's verdict always agrees with the model constructor.
+
+The v3 models expose `must_understand_fields`: the subset of `extra_fields`
+not explicitly waived with `must_understand: false` (fields are implicitly
+must-understand per the spec). Readers discharge the spec's fail-to-open
+duty by subtracting the extension names they recognize; the model only
+partitions by obligation, since recognition is reader-specific.
+
+Optional pydantic integration ships as `zarr_metadata.pydantic` (importing it
+requires pydantic 2.13 or newer; the core package does not depend on it): one
+`Annotated`
+field type per model, validating raw documents through `from_json`, passing
+core-model instances through unchanged, serializing via `to_json`, and
+publishing JSON Schemas derived from private constrained document types that
+mirror the independently expressible runtime rules. Cross-field cardinality
+relations still require runtime validation. The instances are the core model
+classes, so values interoperate freely with non-pydantic code.
+
+`create_default` keeps its output self-consistent: overriding `shape` without
+a chunk grid derives one regular chunk covering the array (v3
+`chunk_shape == shape`; v2 `chunks == shape`) instead of silently keeping the
+scalar default's 0-d grid.
+
+A v2 `.zarray` that omits `dimension_separator` is interpreted with the v2
+convention's default `"."` (the model previously normalized absence to `"/"`,
+which would misaddress the chunks of real-world default-separator arrays).
+The value is never null: absent, `"."`, or `"/"` are the only spellings.
+
+Optional document keys use `UNSET` — a PEP 661 sentinel
+(`typing_extensions.Sentinel`), usable directly in type expressions — never
+`None`: in a model, `None` always corresponds to a JSON `null` in the
+document (a v2 `compressor`, an unnamed dimension inside `dimension_names`),
+and `UNSET` always means the key is absent. Checker note: ty types the
+sentinel exactly; pyright needs `<= 1.1.404` until microsoft/pyright#11115
+is fixed (this package's CI pins it); mypy users need a `cast` or
+`type: ignore` at narrowing sites until python/mypy#21647 merges. This keeps semantically distinct spellings
+distinct — an absent `dimension_names` ("there are no dimension names") and
+an explicit `[null, null]` ("every dimension has a name, which is null") are
+different documents and round-trip as such. The `consolidated_metadata: null`
+written by a historical zarr-python bug is the one deliberate exception to
+faithful round-tripping: those stores remain readable, but the bug spelling
+is repaired to absence on read and never written back.
+
+The v2 models treat the `.zattrs` file's presence as part of the store:
+`attributes` is `UNSET` when no `.zattrs` file exists (and `to_key_value`
+emits none), while an explicit empty `.zattrs` is `{}` and round-trips as a
+file. Previously `to_key_value` always emitted `.zattrs`, silently adding a
+file to stores that never had one.

packages/zarr-metadata/changes/4119.removal.md

@@ -0,0 +1,23 @@
+The document (TypedDict) types are renamed to put the format version at the
+front of the name and to mark the JSON-document form with a `JSON` suffix,
+so a format version can never be misread as a class revision and the bare
+entity names are reserved for the `zarr_metadata.model` dataclasses:
+
+- `ArrayMetadataV2` → `ZarrV2ArrayMetadataJSON` (and `...Partial` accordingly)
+- `ArrayMetadataV3` → `ZarrV3ArrayMetadataJSON` (and `...Partial` accordingly)
+- `GroupMetadataV2` → `ZarrV2GroupMetadataJSON` (and `...Partial` accordingly)
+- `GroupMetadataV3` → `ZarrV3GroupMetadataJSON` (and `...Partial` accordingly)
+- `ConsolidatedMetadataV2` → `ZarrV2ConsolidatedMetadataJSON`
+- `ConsolidatedMetadataV3` → `ZarrV3ConsolidatedMetadataJSON`
+- `NamedConfigV3` → `ZarrV3NamedConfigJSON`
+- `MetadataV3` → `ZarrV3MetadataFieldJSON` (the union of the bare-name and
+  named-configuration spellings of one metadata field)
+- `ExtensionFieldV3` → `ZarrV3ExtensionField`
+- `CodecMetadataV2` → `ZarrV2CodecMetadata`
+- `DataTypeMetadataV2` → `ZarrV2DataTypeMetadata`
+- `ArrayOrderV2` → `ZarrV2ArrayOrder`
+- `ArrayDimensionSeparatorV2` → `ZarrV2ArrayDimensionSeparator`
+
+The old names are removed, not aliased. The `zarr_metadata.pydantic` field
+types take the bare entity names (`ZarrV3ArrayMetadata`, ...), matching the
+model classes they validate into.

packages/zarr-metadata/pyproject.toml

@@ -32,7 +32,11 @@ classifiers = [
 ]
 keywords = ["zarr"]
 dependencies = [
-    "typing_extensions>=4.13",
+    # >=4.16: first release where `Sentinel` pickles by reference
+    # (`__reduce__` returns the sentinel's name), so `UNSET` — and any model
+    # holding it — can cross process boundaries and be deep-copied with its
+    # singleton identity intact. 4.15 and earlier refuse to pickle sentinels.
+    "typing_extensions>=4.16",
 ]
 
 [project.urls]
@@ -43,7 +47,7 @@ Changelog = "https://github.com/zarr-developers/zarr-python/blob/main/packages/z
 Documentation = "https://github.com/zarr-developers/zarr-python/blob/main/packages/zarr-metadata/README.md"
 
 [dependency-groups]
-test = ["pytest", "pydantic>=2"]
+test = ["pytest", "pydantic>=2.13", "jsonschema"]
 
 [tool.hatch.version]
 source = "vcs"
@@ -67,9 +71,9 @@ xfail_strict = true
 addopts = ["-ra", "--strict-config", "--strict-markers"]
 filterwarnings = [
     "error",
-    # pydantic warns about ReadOnly TypedDict items not being enforced at runtime.
-    # That's expected here — we rely on type-checker enforcement, not pydantic mutation guards.
-    "ignore::UserWarning:pydantic._internal._generate_schema",
+    # Pydantic validates these public immutable-shape TypedDicts correctly but
+    # cannot enforce the type checker's ReadOnly mutation restriction.
+    "ignore:Items? .* using the `ReadOnly` qualifier.*:UserWarning:pydantic._internal._generate_schema",
 ]
 
 [tool.numpydoc_validation]
@@ -82,6 +86,9 @@ checks = [
     "PR06",
 ]
 
+# CI pins pyright==1.1.404: later versions regress PEP 661 sentinel typing in
+# class attributes (microsoft/pyright#11115), which zarr_metadata.model._sentinel
+# relies on. Use the same pin locally; unpin when the fix lands.
 [tool.pyright]
 include = ["src"]
 enableExperimentalFeatures = true