feat(python): serve the shipped ai library, so the trace generator can load its input

The Python port shipped both halves of the AI trace stack and not the metadata in
between. `runtime/llm_recorder.py` was there. The `trace-helper` generator was there
and registered for the CLI. But there was no `library/` package and no `libraries`
loader option, so `metaobjects::ai::LlmCallBase` could not be loaded on this port at
all: the documented `extends: metaobjects::ai::LlmCallBase` failed with
ERR_UNRESOLVED_SUPER. A generator shipped without its input.

The more useful part is why nothing caught it. The codegen suite hand-builds its own
abstract `LlmCallBase` — exactly what ADR-0024 already recorded ("the green tests pass
only because they bypass the shipped base with bespoke entities"). A test that builds
its own fixture proves nothing about the path an adopter follows, so the port's docs
could describe a path that could not run.

No new vocabulary — no type, subtype, or attribute — so expected-registry.json is
untouched. This mirrors the TypeScript design rather than inventing a second one: same
package names, same refs (path under library/ minus .yaml), same on-disk-first order.

  library/            library_sources(packages) returns a FileSource when the repo-root
                      library/ tree is reachable, so editing the canonical YAML takes
                      effect immediately, and falls back to the generated embed
                      otherwise (the wheel-in-site-packages case).
  scripts/            regenerates the embed. Embedded as a .py module, not shipped as
                      package data, so no build-backend config can silently drop it.
  libraries=[...]     on from_directory (hence load_directory). Opt-in and lazily
                      imported: a load requesting no libraries neither pays the import
                      nor gets extra names in its model. Sources are prepended for a
                      deterministic, TS-matching order — NOT because resolution needs
                      it; resolve_supers runs once after every root merges.
  config `libraries`  threaded into the CLI's load path. The option first landed only
                      on the loader — which is also all TypeScript exposes — so
                      `metaobjects gen` still could not load the metadata the registered
                      trace-helper generator exists to consume. The generator was
                      reachable from the command line while its input was not. The TS
                      CLI still lacks the key; that is a parity follow-up, not drift
                      introduced here.

An unknown package name is a ConfigError naming the valid ones when it comes from a
config file, and a silent skip when it comes through the API — matching TypeScript.
A name a human typed is a mistake worth failing on; an API caller asking for a package
this version does not ship should still load its own metadata.

Four gates ship with it, because each of these failed silently before:

  * the embed is byte-compared against the canonical YAML (the drift pattern already
    used for spec/metamodel/), so a stale generated module cannot reach a wheel;
  * `extends` is asserted to FAIL without the opt-in and resolve 18 inherited fields
    with it — the negative half is what proves the opt-in is doing the work;
  * ADR-0024 FIX #1 is now enforced: build_llm_call_row's keys equal LlmCallBase's
    effective fields, both directions. The ADR asked for this gate; it did not exist;
  * the acceptance test RUNS the generated helper against a capturing recorder and
    asserts every key it writes is a field the entity declares. The first version of
    that test asserted the strings voRequest/voResponse appeared in the emitted source,
    against a fixture declaring neither column — so it passed while blessing a helper
    that raises on its first write. That is the same bypass ADR-0024 warns about,
    reappearing inside a test written to prevent it. A substring assertion over
    generated code is not an end-to-end test.

Not addressed, and documented rather than changed: the opt-in also brings the library's
own concrete LlmCall entity (table llm_call) alongside the abstract base, so it appears
in codegen output and in a schema diff unless filtered. library/ai/llm-call.yaml is
shared by every port, so splitting it is a cross-port decision.

1639 tests pass; ruff clean on the changed files; no new mypy errors.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018k3CqBZLFkbP4Qs4FZhh96

Author: dmealing

CHANGELOG.md

@@ -482,6 +482,78 @@ ADR-0015 makes for schema migrations. Refusing warns rather than failing the rea
 because failing a Maven build over a file the user chose to own would punish exactly the
 person the guard protects.
 
+||||||| constructed merge base
+
+### Added — Python port serves the shipped `ai` library (loader `libraries=[...]`)
+
+**No new vocabulary — this is port parity.** No type, subtype, or attribute is added, so
+`expected-registry.json` is untouched.
+
+The Python port shipped both halves of the AI trace stack *except* the metadata they
+operate on. `runtime/llm_recorder.py` (`build_llm_call_row` / `persist_llm_call_row`) and
+the registered `trace-helper` generator were both present, but there was no `library/`
+package and no `libraries` loader option — so `metaobjects::ai::LlmCallBase` could not be
+loaded on this port at all, and the documented `extends: metaobjects::ai::LlmCallBase`
+failed with `ERR_UNRESOLVED_SUPER`. A generator shipped without its input.
+
+That is the more useful lesson: the feature was complete on both sides of the metadata
+and absent in the middle, and nothing failed loudly, because *nothing could author the
+entity that would have exercised it*. The port's own docs described the path as working.
+
+Mirrors the TypeScript design rather than inventing a second one — same package names,
+same refs (path under `library/` minus `.yaml`), same on-disk-first resolution:
+
+- `metaobjects/library/` — `library_sources(packages)` returns a `FileSource` when the
+  repo-root `library/` tree is reachable (a checkout, so editing the canonical YAML takes
+  effect immediately) and falls back to the generated embed otherwise (the ordinary
+  wheel-in-site-packages case). An unrecognised package name contributes no sources
+  rather than raising: asking for a package this version does not ship must not stop a
+  consumer loading its own metadata.
+- `scripts/generate_embedded_library.py` — regenerates the embed from the canonical
+  repo-root YAML. Embedded as a `.py` module, not shipped as package data, so no build
+  backend configuration can silently drop it.
+- `MetaDataLoader.from_directory(..., libraries=["ai"])` (hence `load_directory`) —
+  opt-in, and imported lazily: a load that requests no libraries neither pays the import
+  nor gets extra names in its model. Sources are prepended for a deterministic,
+  TS-matching order, **not** because resolution needs it — `resolve_supers` runs once
+  after every root merges, so appending resolves identically.
+- **A `libraries` key on `metaobjects.config.yaml`, threaded into the CLI's load path.**
+  The option first landed only on the loader — which is also all TypeScript exposes — so
+  `metaobjects gen` still could not load `metaobjects::ai::LlmCallBase` even though the
+  `trace-helper` generator that consumes it is registered *for the CLI*. The generator was
+  reachable from the command line while its input was not. An unknown package name in the
+  config is a `ConfigError` naming the valid ones, while the programmatic API keeps
+  TypeScript's silent skip: a name typed into a config file is a mistake worth failing on,
+  where an API caller asking for a package this version does not ship should still load
+  its own metadata. (The TypeScript CLI still lacks the key — parity follow-up, not drift
+  introduced here.)
+
+Three gates ship with it, since each of these failed silently before:
+
+- the embed is byte-compared against the canonical YAML (the drift pattern already used
+  for `spec/metamodel/`), so a stale generated module cannot reach a wheel;
+- `extends: metaobjects::ai::LlmCallBase` is asserted to fail *without* the opt-in and to
+  resolve 18 inherited fields *with* it — the negative half is what proves the opt-in is
+  doing the work;
+- **ADR-0024 FIX #1 is now enforced**: `build_llm_call_row`'s keys are asserted equal to
+  `LlmCallBase`'s effective fields, both directions. The ADR asked for this gate; it did
+  not exist. A recorder writing an undeclared column fails at persist with "Unknown
+  field", and the two drifting apart is invisible until then.
+- the acceptance test **runs** the generated helper against a capturing recorder and
+  asserts every key it writes is a field the entity declares. Worth stating why: the first
+  version of that test asserted the strings `voRequest`/`voResponse` appeared in the
+  emitted source, against a fixture declaring neither column — so it passed while blessing
+  a helper that raises on its first write. That is the same bypass ADR-0024 already warns
+  about ("the green tests pass only because they bypass the shipped base with bespoke
+  entities"), reappearing one level up in a test written to prevent it. A substring
+  assertion over generated code is not an end-to-end test.
+
+Not addressed here, and worth knowing before adopting: the `ai` opt-in also brings the
+library's own **concrete** `LlmCall` entity (table `llm_call`) in alongside the abstract
+base, so it appears in codegen output and in a schema diff unless filtered. Documented in
+the Python prompts reference rather than changed, because `library/ai/llm-call.yaml` is
+shared by every port and splitting it is a cross-port decision.
+
 ## [0.23.2] — npm `0.23.2` · PyPI `0.23.2` · NuGet `0.23.2` · Maven `7.23.2`
 
 A coordinated **PATCH** across all four registries.

agent-context/skills/metaobjects-prompts/references/python.md

@@ -119,7 +119,41 @@ one boundary, not both.
 > generator emits a `record_<entity>(recorder, input, redact=None)` helper (per
 > concrete entity extending `LlmCallBase` with a `@responseRef`/`@payloadRef`-carrying
 > `template.prompt`) that tolerantly extracts the typed response, builds the base
-> trace row, and persists it once. What's still TS-only is the **`call<Entity>`
+> trace row, and persists it once.
+>
+> `LlmCallBase` is shipped metadata, not something you author: opt in when you load,
+> then inherit from it. Without the opt-in the `extends` fails with
+> `ERR_UNRESOLVED_SUPER`.
+>
+> ```python
+> load_directory("metadata/", libraries=["ai"])   # metaobjects::ai::LlmCallBase
+> ```
+> For `metaobjects gen` / `verify`, declare it in `metaobjects.config.yaml` instead —
+> the CLI reads the same opt-in from there:
+> ```yaml
+> metadata: metadata/
+> libraries: ["ai"]
+> ```
+> ```yaml
+> - object.entity:
+>     name: AssistantCall
+>     extends: metaobjects::ai::LlmCallBase
+>     children:
+>       - source.rdb:       { table: assistant_call, role: primary }
+>       - identity.primary: { name: id, fields: ["spanId"] }
+>       # Typed columns are AUTHORED, never derived (ADR-0024 amendment).
+>       # Declare BOTH: the generated record_<entity> writes voRequest and
+>       # voResponse unconditionally, and any key the entity does not declare
+>       # raises "no field '<name>' in metadata" on the first persist.
+>       - field.object: { name: voRequest,  objectRef: MyRequestVO,  storage: jsonb }
+>       - field.object: { name: voResponse, objectRef: MyResponseVO, storage: jsonb }
+> ```
+>
+> Note the opt-in also brings in the library's own concrete `LlmCall` entity
+> (table `llm_call`) alongside the abstract base — it will appear in codegen output
+> and in a schema diff unless you filter it.
+>
+> What's still TS-only is the **`call<Entity>`
 > render→call→record convenience loop** — Python intentionally does not emit it,
 > because the `LlmClient` seam it wraps is BYO / vendor-neutral here (ADR-0024). So
 > you compose render → your LLM call → the generated `record_<entity>(...)` yourself;

server/python/scripts/generate_embedded_library.py

@@ -0,0 +1,63 @@
+#!/usr/bin/env python3
+"""Regenerate ``metaobjects/library/embedded_library.py`` from the repo-root ``library/``.
+
+Cross-port parity: mirrors ``scripts/generate-embedded-library.ts`` in the TypeScript
+port, including the ref convention (path under ``library/`` minus the ``.yaml``
+extension) so both ports key the same content identically.
+
+Why embed at all: a wheel installed into site-packages has no repo-root ``library/``
+tree, and shipping the YAML as package data is one packaging-config mistake away from a
+silently absent library. A generated ``.py`` module is ordinary source — it cannot be
+dropped by a build backend.
+
+Run from anywhere:  python server/python/scripts/generate_embedded_library.py
+"""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+HEADER = '''\
+# AUTO-GENERATED by scripts/generate_embedded_library.py — DO NOT EDIT.
+# Canonical source: repo-root library/**/*.yaml
+# Regenerate: python server/python/scripts/generate_embedded_library.py
+#
+# Embeds the canonical library files as string literals so they resolve wherever the
+# on-disk library/ directory is unavailable (a wheel installed into site-packages).
+# Keys are refs: path under library/ minus the .yaml extension.
+
+EMBEDDED_LIBRARY: dict[str, str] = {
+'''
+
+
+def repo_root(start: Path) -> Path:
+    """Walk up until a directory holds BOTH ``library/`` and ``server/``."""
+    for candidate in [start, *start.parents]:
+        if (candidate / "library").is_dir() and (candidate / "server").is_dir():
+            return candidate
+    raise SystemExit("could not locate the repo root (a dir containing library/ and server/)")
+
+
+def main() -> int:
+    root = repo_root(Path(__file__).resolve().parent)
+    library_dir = root / "library"
+    out_path = root / "server/python/src/metaobjects/library/embedded_library.py"
+
+    entries = []
+    for path in sorted(library_dir.rglob("*.yaml")):
+        ref = path.relative_to(library_dir).with_suffix("").as_posix()
+        entries.append((ref, path.read_text(encoding="utf-8")))
+
+    if not entries:
+        raise SystemExit(f"no *.yaml found under {library_dir} — refusing to emit an empty library")
+
+    body = "".join(f"    {ref!r}: {text!r},\n" for ref, text in entries)
+    out_path.write_text(HEADER + body + "}\n", encoding="utf-8")
+
+    print(f"wrote {out_path.relative_to(root)} ({len(entries)} ref(s): {', '.join(r for r, _ in entries)})")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

server/python/src/metaobjects/cli.py

@@ -235,6 +235,7 @@ def _load_root(
     metadata_dir: str,
     strict: bool = False,
     providers: list[object] | None = None,
+    libraries: list[str] | None = None,
 ) -> tuple[MetaData | None, list[str]]:
     """Load metadata; return ``(root, error_messages)``. ``root`` is None on error.
 
@@ -247,15 +248,26 @@ def _load_root(
     composed ON TOP of the core set, so a config-registered custom subtype resolves
     through the standalone CLI just as it does when an app loads the loader directly.
     Empty/None keeps the core-providers-only default (``from_directory`` convenience).
+
+    ``libraries`` — MetaObjects-shipped library packages (config key ``libraries``,
+    e.g. ``["ai"]``). Without this the CLI could not load ``metaobjects::ai::LlmCallBase``,
+    so ``metaobjects gen`` failed with ERR_UNRESOLVED_SUPER on exactly the metadata the
+    registered ``trace-helper`` generator exists to consume — the generator was reachable
+    from the CLI while its input was not.
     """
     if providers:
         from metaobjects.core_types import core_providers
 
         result = MetaDataLoader.from_directory(
-            metadata_dir, providers=[*core_providers, *providers], strict=strict
+            metadata_dir,
+            providers=[*core_providers, *providers],
+            strict=strict,
+            libraries=libraries,
         )
     else:
-        result = MetaDataLoader.from_directory(metadata_dir, strict=strict)
+        result = MetaDataLoader.from_directory(
+            metadata_dir, strict=strict, libraries=libraries
+        )
     if result.errors:
         msgs = [f"{e.code}: {e.message}" for e in result.errors]
         return None, msgs
@@ -671,7 +683,9 @@ def _cmd_gen_config(args: argparse.Namespace) -> int:
     if not providers_ok:
         return 1
 
-    root, load_errors = _load_root(config.metadata_dir(), providers=providers)
+    root, load_errors = _load_root(
+        config.metadata_dir(), providers=providers, libraries=config.libraries
+    )
     if root is None:
         print("error: failed to load metadata:", file=sys.stderr)
         for msg in load_errors:
@@ -844,7 +858,9 @@ def _verify_codegen_config(args: argparse.Namespace) -> int:
     if not providers_ok:
         return 1
 
-    root, load_errors = _load_root(config.metadata_dir(), strict=strict, providers=providers)
+    root, load_errors = _load_root(
+        config.metadata_dir(), strict=strict, providers=providers, libraries=config.libraries
+    )
     if root is None:
         print("error: failed to load metadata:", file=sys.stderr)
         for msg in load_errors:

server/python/src/metaobjects/codegen/project_config.py

@@ -71,6 +71,11 @@ class ProjectConfig:
     metadata: str
     #: Consumer provider refs (``module:symbol``), resolved config-relative.
     providers: list[str]
+    #: MetaObjects-shipped library packages to load alongside ``metadata``
+    #: (e.g. ``["ai"]`` for ``metaobjects::ai::LlmCallBase``). Without this the
+    #: CLI cannot load the metadata that shipped generators like ``trace-helper``
+    #: exist to consume, and an adopter's ``extends`` fails ERR_UNRESOLVED_SUPER.
+    libraries: list[str]
     #: Ordered run-specs (YAML map insertion order preserved).
     targets: list[TargetConfig]
 
@@ -126,6 +131,22 @@ def load_project_config(path: Path) -> ProjectConfig:
         raise ConfigError(f"{path}: 'metadata' must be a string (a directory path).")
 
     providers = _require_str_list(raw.get("providers", []), f"{path}: 'providers'")
+    libraries = _require_str_list(raw.get("libraries", []), f"{path}: 'libraries'")
+    if libraries:
+        # Validated HERE and not in library_sources(): the programmatic API skips an
+        # unknown package on purpose (cross-port parity with TS), but a name typed
+        # into a config file is a mistake worth failing on — silently skipping it
+        # resurfaces later as ERR_UNRESOLVED_SUPER pointing at the adopter's own
+        # metadata, which is the wrong place to go looking.
+        from metaobjects.library import known_packages
+
+        available = known_packages()
+        unknown = [name for name in libraries if name not in available]
+        if unknown:
+            raise ConfigError(
+                f"{path}: 'libraries' has unknown package(s) {unknown}; "
+                f"available: {available}"
+            )
 
     targets_raw = raw.get("targets")
     if not isinstance(targets_raw, dict) or not targets_raw:
@@ -156,5 +177,6 @@ def load_project_config(path: Path) -> ProjectConfig:
         config_dir=path.parent.resolve(),
         metadata=metadata,
         providers=providers,
+        libraries=libraries,
         targets=targets,
     )

server/python/src/metaobjects/library/__init__.py

@@ -0,0 +1,9 @@
+"""MetaObjects-shipped standard metadata packages.
+
+Adopters opt in through the loader's ``libraries=["ai"]`` option and then
+``extends: "metaobjects::ai::LlmCallBase"`` on their own entity.
+"""
+
+from .library_sources import known_packages, library_sources
+
+__all__ = ["known_packages", "library_sources"]

server/python/src/metaobjects/library/embedded_library.py

@@ -0,0 +1,11 @@
+# AUTO-GENERATED by scripts/generate_embedded_library.py — DO NOT EDIT.
+# Canonical source: repo-root library/**/*.yaml
+# Regenerate: python server/python/scripts/generate_embedded_library.py
+#
+# Embeds the canonical library files as string literals so they resolve wherever the
+# on-disk library/ directory is unavailable (a wheel installed into site-packages).
+# Keys are refs: path under library/ minus the .yaml extension.
+
+EMBEDDED_LIBRARY: dict[str, str] = {
+    'ai/llm-call': '# library/ai/llm-call.yaml\n# MetaObjects-shipped standard metadata. Adopters opt in via the loader\'s\n# `libraries: ["ai"]` option, then `extends: "metaobjects::ai::LlmCallBase"`.\nmetadata:\n  package: metaobjects::ai\n  children:\n    - object.entity:\n        name: LlmCallBase\n        abstract: true\n        children:\n          - field.uuid:     { name: traceId }\n          - field.uuid:     { name: spanId }\n          - field.uuid:     { name: parentSpanId }\n          - field.string:   { name: sessionId }\n          - field.string:   { name: callType }\n          - field.string:   { name: system }\n          - field.string:   { name: requestModel }\n          - field.string:   { name: responseModel }\n          - field.int:      { name: inputTokens }\n          - field.int:      { name: outputTokens }\n          - field.currency: { name: costMinor, currency: USD }\n          - field.int:      { name: latencyMs }\n          - field.string:   { name: finishReason }\n          - field.string:   { name: status }\n          - field.string:   { name: errorDetail }\n          - field.timestamp: { name: startedAt }\n          - field.string:   { name: llmRequest,  dbColumnType: jsonb }   # generic jsonb (no objectRef)\n          - field.string:   { name: llmResponse, dbColumnType: jsonb }\n    - object.entity:\n        name: LlmCall\n        extends: metaobjects::ai::LlmCallBase\n        children:\n          - source.rdb:      { table: llm_call, role: primary }\n          - identity.primary: { name: id, fields: ["spanId"] }\n',
+}