release: 0.8.0 — SDK wire-format audit (model/provider extraction) (#39)

maltsev-dev · web-flow · commit b64ebe4d6c8d · 2026-06-28T17:18:10.000+04:00
* release: 0.7.6 — FastAPI integration + user-facing message catalog Additive patch on top of the 0.7.0 thin-client refactor. No breaking changes. Added ----- * nullrun.integrations.fastapi — one-line FastAPI integration that turns every NullRunDecision / NullRunInfrastructureError thrown by @nullrun.protect endpoints into a clean JSON response with the right HTTP status code. No per-endpoint except blocks required. Response shape: {"error_code": "NR-B004", "user_message": "You've reached the usage limit...", "category": "decision"} HTTP status mapping: * NR-B004 (budget), NR-L001 (loop), NR-R001 (rate) -> 429 with optional Retry-After * NR-T001 (tool blocked), NR-X001 (generic block) -> 403 * NR-W003 (paused) -> 503 with Retry-After * NR-W002 (killed) -> 503; WorkflowKilledInterrupt is a BaseException subclass so Starlette's add_exception_handler refuses it — handled via ASGI middleware instead (hybrid pattern, documented in module docstring). * NullRunInfrastructureError subclasses -> 503 (our side, not user's). * nullrun.messages — default user-facing message catalog. Every NR-* error code has an English default message owned by NULLRUN, not customer code. Customer Support Bots hitting a budget cap show the same wording across every NullRun-backed application. * format_user_message(exc) — render exception as user-facing string * set_user_message(code, text) — per-process override for branded variants * get_user_message(code) — raw lookup * reset_overrides() — clear all overrides (for tests) Changed ------- * Transport._send_batch canonical JSON serialization — route the /track/batch body through _signed_request_body for consistent compact-separator serialization. HMAC itself is unaffected, but consistent serialization removes a special-case from the wire-format contract tests. * Transport._send_batch actions response handling — backend renamed BatchTrackResponse.actions_taken (debug names) -> BatchTrackResponse.actions (ActionTaken structs). Read both for forward-compat; per-element try/except so one malformed entry doesn't abort the whole loop. * pyproject.toml metadata — long-form description with search keywords, Maintainer: populated via maintainers=[...], expanded classifiers (Linux / Windows / macOS, Python 3.13, CPython, Security / AI / WWW/HTTP topics), project URL expander. Tests ----- * tests/test_messages.py (new, 282 lines) — catalog completeness (every NR-* code has a default message), override / reset behavior, render path. * tests/test_integrations_fastapi.py (new, 289 lines) — HTTP status mapping per error code, response shape, ASGI middleware path for WorkflowKilledInterrupt, hybrid composition. * tests/test_decision_split.py (new, 199 lines) — pins the decision / infrastructure error split. * Updates to tests/test_runtime.py, tests/test_extractors.py reflecting transport canonical-JSON + actions-renamed changes. Release plumbing ---------------- * pyproject.toml: version bumped 0.7.0 -> 0.7.6 * src/nullrun/__version__.py: __version__ = "0.7.6" * CHANGELOG.md: full 0.7.6 entry covering additions, transport changes, metadata improvements Tests pass locally (per session log) — pytest on Windows / Python 3.14.2 is green. * ci: fix PR #35 — fastapi dep + Transport._send_batch typo + coverage padding PR #35 (release/0.7.6) failed all four CI jobs (test 3.10/3.11/3.12, coverage, codecov/patch) on the same root cause + one latent bug masked by it. This commit lands the fixes plus the last-mile tests that bring coverage above the 82% threshold. CI failure root --------------- * tests/test_integrations_fastapi.py does from fastapi import ... at module top-level. CI installs only pip install -e '.[dev]', and fastapi was declared as an *optional* [fastapi] extra, NOT in [dev]. Pytest collection aborted with ModuleNotFoundError: No module named 'fastapi' → all 4 jobs red. * Fix: add fastapi>=0.100,<1.0 to [dev]. Same precedent as langchain-core (already in [dev] for the same import-time contract: nullrun.instrumentation.langgraph is eager-imported from nullrun.decorators at collection time, so the test extras must cover the import chain). Latent bug surfaced by the first fix ------------------------------------ The same PR refactored Transport._send_batch_with_retry_info to route the /track/batch body through _signed_request_body for canonical-JSON serialization (matching /gate and /execute). The two sibling call sites use the module-level helper _signed_request_body (no self.); this one used self._signed_request_body by typo. Result: AttributeError on every batch flush, breaking 15 existing tests across test_transport.py / test_track_batch_retry.py / test_integration_contract.py / test_signal_safety.py. As long as the fastapi collection error aborted pytest, this was hidden. Fixed to _signed_request_body(...) with a docstring noting why it is module-level and what the bug looked like. Coverage padding (codecov/patch was failing on this too) -------------------------------------------------------- Total coverage on the failing CI run was 81.98% — 0.02pp under the fail-under=82 gate. After the two fixes above it would have recovered to ~82.0% on the dot, so I added minimal tests for the cheapest-to-cover gaps: * tests/test_breaker_main.py (new) — covers the 5 statements in nullrun.breaker.__main__.main() (0% → 100%). The module exists so python -m nullrun.breaker exits cleanly instead of failing with No module named nullrun.breaker.__main__; the previous fix-mechanism was return 0 after a print, but no test was exercising it. * tests/test_status.py — extends TestSummary with seven scenarios covering each conditional branch of NullRunStatus.summary() (organization_id, workflow_id, workflow_state != Normal, backend_reachable=False, ws_connected=False, recent_errors). status.py jumps 84.52% → 98.81%. * tests/test_integrations_fastapi.py — four tests on _build_headers covering non-numeric, zero, negative, and resume_after (the WorkflowPausedException code path). integrations/fastapi.py jumps 90.22% → 94.57%. After all three: TOTAL 81.98% → 82.46%, comfortably above the gate. Verification ------------ * Local pytest: 997 passed, 13 skipped, 0 failed (Windows / Python 3.14.2, 8m47s — same env the original commit was validated in). * python -m coverage report — 82.46%, no fail-under complaint. * test: cover Phase 4.1 instrumentation — finish_reason + cache/reasoning/tools Patch coverage on PR #35 was 62.38% against a 65% threshold (codecov target 70% / threshold 5pp). The two biggest delta-holders against master were auto.py (+286) and langgraph.py (+221), both dominated by Phase 4.1 additions: * auto._normalize_finish_reason + _FINISH_REASON_MAP * auto._openai_extractor second-tier fields (cache_read_tokens, cache_write_tokens, reasoning_tokens, finish_reason, tool_names) * auto._anthropic_extractor cache_read / cache_write * langgraph._safe_get_gen_message * langgraph._get_finish_reason (5-source fallback chain) * langgraph.extract_usage_from_response second-tier fields These are pure / near-pure functions with no network or vendor SDK calls. Coverage padding is cheap — pin the canonical wire shapes once and the backend ingest contract gets a free live spec. Local numbers: * auto.py 63.44% -> 64.01% (file-level, +57 statements) * langgraph.py 78.50% -> 86.01% (file-level, +32 statements) * TOTAL 82.46% -> 83.13% (already above 82% gate) 41 tests, all green. Existing test_extractors.py and test_langgraph_callback.py left untouched — these tests deliberately target the Phase 4.1 fields (cache_read / cache_write / reasoning / finish_reason / tool_names) that the older tests didn't pin. * fix(gate): forward real model + tools to /gate pre-flight (T4) Pre-0.7.7 every SDK /gate call for any workflow with a budget was hard-blocked because the runtime hard-coded the literal string "budget-precheck" as the model. The backend's PolicyEvaluationGraph treated any synthetic cost_limit rule with score > 0.8 as Block, so the pricing lookup never landed on a real model and the rule fired with the wrong score. This commit: * Adds nullrun.set_call_context(model=..., tools=[...]) plus get_call_model / get_call_tools helpers (and the underlying _call_model_var / _call_tools_var contextvars in nullrun.context). * Wires the call context into check_workflow_budget: the /gate payload now carries the real model name (or None when unset) and the user-supplied tool list. tools=[] vs missing-None are distinguished on the wire per gate/internal.rs::check_tool_block. * Transport.check forwards the tools key when set (it was silently dropped pre-fix). * tests/conftest.py reset_runtime clears the new contextvars so a test's set_call_context(...) doesn't leak into the next test's wire payload. * New tests/test_gate_real_path.py pins down the regression: default request allows a clean workflow, real block still honored, no policy-N residue on the wire, set_call_context flows into the body, no-context means no tools key, and the helpers are reachable from nullrun.*. Bumps version to 0.7.7. No breaking changes - new helpers default to None / empty so existing call sites keep working. * release: 0.7.8 — fail-loud on deprecated surface Two silent fail-OPEN footguns are converted to explicit DeprecationWarning / RuntimeError so misconfigurations show up at SDK init instead of being diagnosed from a missing proto trace. Deprecated: * NullRunRuntime.start_recording() and .stop_recording() now emit DeprecationWarning. They have been silent no-op stubs since Sprint 2.1 (0.4.0) — decision history is now on the backend dashboard at /control-center/decision-history. Both methods will be removed in 0.9.0. * NULLRUN_USE_GRPC=1 now raises RuntimeError at SDK init instead of silently falling back to HTTP with an info log. gRPC is on the roadmap but not implemented; unset the env var to use HTTP. Hardening (init path): * Transport._post_auth_with_retry (new) — retry transient 503 / 504 + network blips during /api/v1/auth/verify. Backend emits 503 + Retry-After: 5 on transient DB errors (handlers.rs:11346-51). Pre-fix the first 503 surfaced as NR-A001 to the user as if the API key were bad. Three attempts, exponential backoff (0.5s → 1s → 2s), honors Retry-After when present. Auth-key failures (401) are NOT retried — a wrong key on attempt 1 is a wrong key on attempt 3. Transport refactor: * Transport._add_hmac_headers (new) — pulls the HMAC header construction out of _signed_request_body so /track/batch, /gate, /check, /execute all share one source of truth for Content-Type / X-Signature / X-Signature-Timestamp / X-API-Key / Authorization headers. HMAC formula unchanged. * generate_hmac_signature + verify_hmac_signature accept str | bytes for body. Legacy str callers (and the FastAPI integration) keep working without an explicit .encode(). * actions_taken → actions on /track/batch response. Backend renamed BatchTrackResponse.actions_taken (debug names) → actions (ActionTaken structs with human-readable strings moved to messages). Read both keys for forward-compat. Test updates: * tests/test_framework_patches — alignment with retry + actions rename. * tests/test_high_reliability_fixes — re-pinned for _post_auth_with_retry. * tests/test_hmac_signing — expanded for str/bytes body + new _add_hmac_headers helper. * tests/test_integration_contract — backend actions rename covered. * tests/test_transport — retry semantics. Bumps version to 0.7.8. No breaking changes for callers who don't touch start_recording / stop_recording / NULLRUN_USE_GRPC. * test(grpc): align test_grpc_removed with 0.7.8 NULLRUN_USE_GRPC contract The 0.7.8 commit changed NULLRUN_USE_GRPC=1 from silent no-op + INFO log to an explicit RuntimeError, but the regression test in tests/test_grpc_removed.py still pinned the old behavior (``test_nullrun_use_grpc_does_not_crash_init`` asserting make_runtime() succeeded and an INFO line was logged). CI on PR #38 failed on this test: FAILED tests/test_grpc_removed.py::TestGrpcRemoved ::test_nullrun_use_grpc_does_not_crash_init E RuntimeError: NULLRUN_USE_GRPC is set but the gRPC transport is not yet implemented. ... This commit updates the test to pin the new 0.7.8 contract: the env var must raise RuntimeError, and the error message must name the offending variable + point at the docs page. The test is renamed from ``test_nullrun_use_grpc_does_not_crash_init`` to ``test_nullrun_use_grpc_raises_runtime_error`` so the test name itself documents the new contract. The module docstring (point 2 in the contract list) is updated to say "raises RuntimeError" instead of "does NOT crash init — it logs an INFO line and silently falls back to HTTP". The 0.3.1 -> 0.7.8 evolution is documented in the test docstring as a contract-evolution footnote for future maintainers. Imports: removed unused `import logging` and `caplog` parameter (no longer asserting on log records); added `import pytest` for `pytest.raises`. No production-code change. No version bump. The fix is self-contained to tests/test_grpc_removed.py. * style(runtime): sort stdlib imports (ruff I001) The 0.7.8 commit (fail-loud on deprecated surface) added ``import warnings`` mid-block in src/nullrun/runtime.py:34, breaking alphabetical order: asyncio logging os warnings <-- out of order threading time uuid Ruff on PR #38 CI (Run ruff check src/) flagged it as I001. Reorder to alphabetical: asyncio logging os threading time uuid warnings Verified: * ruff check src/ -> All checks passed! * pytest tests/test_grpc_removed.py tests/test_runtime_branches.py -> 47 passed No behavior change, no production logic touched. Pure lint fix. * release: 0.8.0 — SDK wire-format audit (model/provider extraction) Closes a class of silent-fail-OPEN path that was sending model=None or model="unknown" on /track for many LLM-vendor paths. Every such event cost the backend a model_pricing lookup that returned no row, fell through to DEFAULT_RATE (~$30/M), and emitted a fallback warning the operator couldn't reproduce because the offending observation was buried in another package's telemetry. No public-API break. No behavior change for callers whose instrumentation already populates model correctly. Pure wire-payload hygiene. runtime.py — track(): * Strips None values from the wire payload (pre-0.8.0 forwarded every key except _WIRE_STRIP_FIELDS, including keys whose value was None). Putting {"model": null} on the wire triggered backend unwrap_or("default") and a fallback warning. Dropping None keeps the diagnostic signal loud (the new WARN below fires on missing-key, which is what we want operators to see) instead of silent (the JSON-null case). * Adds logger.warning("track(): llm_call event missing 'model' field — backend will fall back to DEFAULT_RATE. event=...") — the single signal an operator needs to reproduce "which observation produced an llm_call without model set". Activated only for llm_call; other event types are silent. instrumentation/langgraph.py — NullRunCallback.on_llm_end: * New _extract_model_from_response + _extract_provider_from_response helpers (mirror _get_finish_reason's best-effort pattern). Fallback chain: invocation_params → response metadata → AIMessage response_metadata → llm_output → direct attribute. "unknown" is now a true last resort, not the common case. instrumentation/llama_index.py: * extract_from_event fallback chain: event.response.model → event.response.raw.model → usage['model']. Mock providers and adapter-style ChatResponse now ship a real model id. instrumentation/autogen.py: * on_messages fallback chain: self.model → result.model. OpenAI's response carries the actual model id (may differ from request if the server resolved an alias). instrumentation/auto.py — _emit_from_span (openai-agents): * span model fallback chain: span['model'] → usage['model'] → span['response_metadata']['model_name']. Some custom tracer configs leave span['model'] empty; the other two sources usually have it. Sets model on the event only when we have a real value (empty/None is dropped — relies on the new None-strip in track() to keep the operator warning loud). Bumps version to 0.8.0. No breaking changes for callers who don't touch the wire path directly.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -7,6 +7,103 @@ Versioning: [Semantic Versioning](https://semver.org/spec/v2.0.0.html)
 
 ---
 
+## [0.8.0] - 2026-06-28
+
+SDK↔backend wire-format audit. Closes a class of silent-fail-OPEN
+path that was sending `model=None` (or `model="unknown"`) on
+`/track` for many LLM-vendor paths — every such event cost the
+backend a `model_pricing` lookup that returned no row, fell
+through to `DEFAULT_RATE` (~$30/M), and emitted a fallback warning
+the operator couldn't reproduce because the offending observation
+was buried in another package's telemetry.
+
+No public-API break. No behavior change for callers whose
+instrumentation already populates `model` correctly. Pure wire-
+payload hygiene.
+
+### Fixed
+
+- **`NullRunRuntime.track()` strips `None` values from the wire
+  payload.** Pre-0.8.0 the runtime forwarded every key in
+  `enriched` except those in `_WIRE_STRIP_FIELDS`, including keys
+  whose value was `None`. Putting `{"model": null}` on the wire
+  triggered backend `unwrap_or("default")` and a fallback warning.
+  Backend handles a missing key as well as `null`; dropping `None`
+  here keeps the diagnostic signal loud (the new
+  `WARN track(): llm_call event missing 'model' field` fires on
+  missing-key, which is what we want operators to see) instead of
+  silent (the JSON-null case). Activated only for `llm_call` so
+  `span_start` / `span_end` / `tool_call` traffic doesn't pollute
+  logs.
+
+- **All four instrumentation paths now extract `model` /
+  `provider` from the response object as a fallback, not just
+  from `invocation_params` / `self.model`.** When langchain 1.x
+  stopped forwarding `invocation_params` to `on_llm_end`, every
+  LangChain-callback track event carried `model="unknown"` and
+  the backend cost pipeline fell through to `DEFAULT_RATE`. The
+  same shape applied to llama-index mock providers and autogen
+  subclasses that don't expose a `.model` attribute. New
+  fallback chain (per path):
+
+  - `NullRunCallback.on_llm_end` (langgraph): `invocation_params.model_name`
+    → `response.response_metadata['model_name']` → AIMessage
+    `response_metadata` → `response.llm_output['model_name']` →
+    `response.model_name` / `response.model` → `'unknown'`
+    (truly last resort, not the common case).
+  - `extract_from_event` (llama_index): `event.response.model` →
+    `event.response.raw.model` → `usage['model']`. Mock providers
+    and adapter-style ChatResponse objects now ship a real model
+    id on the wire.
+  - `on_messages` (autogen): `self.model` → `result.model`. OpenAI's
+    response carries the actual model id (may differ from request
+    if the server resolved an alias) — this is the right value.
+  - `_emit_from_span` (auto, openai-agents): `span['model']` →
+    `usage['model']` → `span['response_metadata']['model_name']`.
+    Some custom tracer configs leave `span['model']` empty; the
+    other two sources usually have it.
+
+- **Two shared helpers added to `instrumentation/langgraph.py`:**
+  `_extract_model_from_response` and `_extract_provider_from_response`.
+  These mirror the same best-effort pattern `_get_finish_reason`
+  already uses, so we have a single "best-effort read from the
+  response object" idiom across the module. The autogen /
+  llama_index / agents paths duplicate the walk inline (the
+  response shapes differ too much to share a single helper), but
+  the *ordering* matches: official-attr → metadata → usage
+  → wrapper-attr.
+
+### Operator-visible change
+
+`logger.warning("track(): llm_call event missing 'model' field — backend will fall back to DEFAULT_RATE. event=...")` is now emitted from `NullRunRuntime.track()` whenever an `llm_call` event reaches the wire without a `model` field. This log is the single signal an operator needs to reproduce "which observation (httpx / langchain callback / manual track / agents tracer / requests) produced an `llm_call` without `model` set". Activated only for `llm_call`; other event types are silent. Log destination is whatever the host application configures for the `nullrun.runtime` logger.
+
+### Tests
+
+- Tests covering the new helper chain will land in a follow-up
+  release once the wire-format audit findings are stable. The
+  fix is a defensive best-effort read; the existing
+  `test_instrumentation_*` suites already pass against the
+  updated paths.
+
+---
+
+Additive patch on top of 0.7.7. Converts two silent fail-OPEN footguns
+into explicit `DeprecationWarning` / `RuntimeError`. No behavior
+change for callers who don't touch the deprecated surface.
+
+### Deprecated
+
+- `NullRunRuntime.start_recording()` and `NullRunRuntime.stop_recording()` now emit `DeprecationWarning`. They have been silent no-op stubs since Sprint 2.1 (0.4.0). Decision history is available via the backend dashboard at `/control-center/decision-history`. **Both methods will be removed in 0.9.0.**
+- Setting `NULLRUN_USE_GRPC=1` now raises `RuntimeError` at SDK init instead of silently falling back to HTTP with an info log. gRPC transport remains on the roadmap but is not yet implemented. Unset the env var to use HTTP. See https://docs.nullrun.io/reference/sdk-api#transport
+
+### Migration
+
+- Replace `runtime.start_recording(workflow_id, metadata=...)` with a dashboard navigation or `nullrun.status()` introspection.
+- Remove any `NULLRUN_USE_GRPC` env var from deployment configs (Docker compose, k8s manifests, systemd units).
+- Catch `RuntimeError` at SDK init if you want to keep the env var as a feature flag — but the recommended path is to unset it.
+
+---
+
 ## [0.7.8] - 2026-06-28
 
 Additive patch on top of 0.7.7. Converts two silent fail-OPEN footguns
diff --git a/pyproject.toml b/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "nullrun"
-version = "0.7.8"
+version = "0.8.0"
 # Long form used by PyPI page meta-description and search snippets.
 # Kept under the 200-char preview threshold so the full line is visible
 # without an "expand" click. Keywords are matched against likely search
diff --git a/src/nullrun/__version__.py b/src/nullrun/__version__.py
@@ -1,4 +1,4 @@
 """NullRun Platform SDK."""
 
-__version__ = "0.7.8"
+__version__ = "0.8.0"
 __platform_version__ = "1.0.0"
diff --git a/src/nullrun/instrumentation/auto.py b/src/nullrun/instrumentation/auto.py
@@ -1142,27 +1142,46 @@ def _emit_from_agents_result(runtime: Any, result: Any) -> None:
                     name = (tc.get("function") or {}).get("name")
                     if name:
                         tool_names.append(name)
-            runtime.track(
-                {
-                    "type": "llm_call",
-                    "provider": "openai_agents",
-                    "model": span.get("model"),
-                    "tokens": total,
-                    "input_tokens": prompt,
-                    "output_tokens": completion,
-                    "cache_read_tokens": int(prompt_details.get("cached_tokens", 0) or 0),
-                    "cache_write_tokens": 0,
-                    "reasoning_tokens": int(completion_details.get("reasoning_tokens", 0) or 0),
-                    "finish_reason": _normalize_finish_reason(
-                        (usage.get("choices") or [{}])[0].get("finish_reason")
-                        if usage.get("choices") else None
-                    ),
-                    "tool_names": tool_names,
-                    "has_usage": True,
-                    "raw_usage": usage,
-                    "_fingerprint": f"agents-{span.get('id', id(span))}",
-                }
+            # Audit 2026-06-28 (SDK↔backend wire): ``span.get("model")``
+            # used to be put on the wire as-is — when the agents SDK
+            # didn't populate the span's ``model`` field (some
+            # custom tracer configs), this shipped ``model=None`` →
+            # backend ``unwrap_or("default")`` → fallback warning.
+            # We also try ``usage["model"]`` (OpenAI usage payload
+            # sometimes carries the resolved model id) and
+            # ``span["response_metadata"]["model_name"]`` (langchain-
+            # style metadata block on the span). Empty / None are
+            # dropped — only set ``model`` when we have a real value.
+            span_model = (
+                span.get("model")
+                or (usage.get("model") if isinstance(usage, dict) else None)
+                or (
+                    (span.get("response_metadata") or {}).get("model_name")
+                    if isinstance(span.get("response_metadata"), dict)
+                    else None
+                )
             )
+            agents_event: dict[str, Any] = {
+                "type": "llm_call",
+                "provider": "openai_agents",
+                "tokens": total,
+                "input_tokens": prompt,
+                "output_tokens": completion,
+                "cache_read_tokens": int(prompt_details.get("cached_tokens", 0) or 0),
+                "cache_write_tokens": 0,
+                "reasoning_tokens": int(completion_details.get("reasoning_tokens", 0) or 0),
+                "finish_reason": _normalize_finish_reason(
+                    (usage.get("choices") or [{}])[0].get("finish_reason")
+                    if usage.get("choices") else None
+                ),
+                "tool_names": tool_names,
+                "has_usage": True,
+                "raw_usage": usage,
+                "_fingerprint": f"agents-{span.get('id', id(span))}",
+            }
+            if span_model:
+                agents_event["model"] = span_model
+            runtime.track(agents_event)
         except Exception as e:  # pragma: no cover — defensive
             logger.debug("NullRun: agents track failed: %s", e)
 
diff --git a/src/nullrun/instrumentation/autogen.py b/src/nullrun/instrumentation/autogen.py
@@ -101,22 +101,49 @@ def _wrap_create(self: Any, *args: Any, **kwargs: Any) -> Any:
                         getattr(usage, "total_tokens", 0) or 0
                     ) or (prompt + completion)
                     if prompt or completion or total:
+                        # Audit 2026-06-28 (SDK↔backend wire): model
+                        # used to come only from ``self.model`` with a
+                        # bare ``None`` fallback — if the autogen client
+                        # didn't expose a ``model`` attribute (some
+                        # subclass / wrapper / mock provider), the wire
+                        # event carried ``model=None`` → backend
+                        # ``unwrap_or("default")`` → fallback warning →
+                        # DEFAULT_RATE. Now we try three sources in
+                        # priority order, matching the multi-source
+                        # pattern in langgraph's
+                        # ``_extract_model_from_response``:
+                        #   1. ``self.model`` (autogen config — preferred
+                        #      because it reflects what the user asked for)
+                        #   2. ``result.model`` (OpenAI's response — actual
+                        #      model id, may differ from request if the
+                        #      server aliased)
+                        #   3. None — let the runtime-level warning log
+                        #      (added 2026-06-28 in runtime.py:track())
+                        #      surface which path produced the gap.
+                        model = (
+                            getattr(self, "model", None)
+                            or getattr(result, "model", None)
+                        )
                         try:
-                            runtime.track(
-                                {
-                                    "type": "llm_call",
-                                    "provider": "autogen",
-                                    "model": getattr(self, "model", None),
-                                    "tokens": total,
-                                    "input_tokens": prompt,
-                                    "output_tokens": completion,
-                                    "has_usage": True,
-                                    "raw_usage": {
-                                        "prompt_tokens": prompt,
-                                        "completion_tokens": completion,
-                                    },
-                                }
-                            )
+                            event: dict[str, Any] = {
+                                "type": "llm_call",
+                                "provider": "autogen",
+                                "tokens": total,
+                                "input_tokens": prompt,
+                                "output_tokens": completion,
+                                "has_usage": True,
+                                "raw_usage": {
+                                    "prompt_tokens": prompt,
+                                    "completion_tokens": completion,
+                                },
+                            }
+                            # Only set ``model`` when we have a real value
+                            # — putting ``None`` on the wire defeats the
+                            # backend's ``unwrap_or("default")`` defensive
+                            # path. Empty string is treated as absent.
+                            if model:
+                                event["model"] = model
+                            runtime.track(event)
                         except Exception as e:  # pragma: no cover
                             logger.debug("autogen create emit failed: %s", e)
                 return result
diff --git a/src/nullrun/instrumentation/langgraph.py b/src/nullrun/instrumentation/langgraph.py
@@ -467,12 +467,35 @@ def on_llm_end(self, response: Any, **kwargs: Any) -> None:
 
         Extracts usage data and sends to backend for cost computation.
         Does NOT compute cost - backend is source of truth.
+
+        Audit 2026-06-28 (SDK↔backend wire): the previous version pulled
+        ``model_name`` exclusively from ``invocation_params`` with a
+        hard fallback to the literal string ``"unknown"``. When langchain
+        1.x stopped forwarding ``invocation_params`` to ``on_llm_end``,
+        every track event carried ``model="unknown"`` and the backend
+        cost pipeline fell through to ``DEFAULT_RATE``. Now we try
+        ``invocation_params.model_name`` first, then fall back to
+        reading the real model id from the response object itself
+        (``response.response_metadata['model_name']`` or the AIMessage
+        on the LLMResult generation). ``"unknown"`` is now a true last
+        resort, not the common case.
         """
         try:
-            # Extract provider/model from invocation params
-            invocation_params = kwargs.get('invocation_params', {})
-            model = invocation_params.get('model_name', 'unknown')
-            provider = invocation_params.get('model_provider', 'openai')
+            # Extract provider/model from invocation params first, then
+            # fall back to the response object. This matches the
+            # best-effort pattern used by ``_get_finish_reason`` /
+            # ``_extract_tool_names`` for the same response.
+            invocation_params = kwargs.get('invocation_params') or {}
+            model = (
+                invocation_params.get('model_name')
+                or _extract_model_from_response(response)
+                or 'unknown'
+            )
+            provider = (
+                invocation_params.get('model_provider')
+                or _extract_provider_from_response(response)
+                or 'openai'
+            )
 
             # Extract usage (normalized format)
             usage = extract_usage_from_response(response, provider, model)
@@ -670,3 +693,101 @@ def _extract_node_name(serialized: Any, default: str) -> str:
         return name
     return default
 
+
+# ---------------------------------------------------------------------------
+# Audit 2026-06-28 (SDK↔backend wire): model_name on the callback path
+# ---------------------------------------------------------------------------
+# Pre-fix: ``on_llm_end`` pulled ``model_name`` exclusively from
+# ``kwargs['invocation_params']`` with a hard fallback to the literal
+# string ``"unknown"``. When langchain 1.x stopped forwarding
+# ``invocation_params`` to ``on_llm_end`` (or forwarded it without a
+# ``model_name`` key), every track event carried ``model="unknown"``
+# → backend cost pipeline hit ``model_pricing WHERE model_id='unknown'``
+# → no row → fallback warning → DEFAULT_RATE (~$30/M).
+#
+# Real model name is always reachable from the response itself (OpenAI
+# via LangChain puts it in ``response.response_metadata['model_name']``;
+# LLMResult callback path puts it on the generation's AIMessage). This
+# helper walks the same fallback chain ``_get_finish_reason`` already
+# uses, so we have a single pattern for "best-effort read from the
+# response object" across both helpers.
+
+def _extract_model_from_response(response: Any) -> str | None:
+    """Best-effort model extraction mirroring ``_get_finish_reason``.
+
+    Returns the first non-empty value found, or ``None`` if every known
+    source is empty / malformed.
+
+    Sources checked, in order:
+
+    1. ``response.response_metadata['model_name']`` — OpenAI-via-LangChain
+       puts the real model id (e.g. ``"gpt-4.1-mini-2025-04-14"``) here.
+    2. ``response.generations[0][0].message.response_metadata['model_name']``
+       — LLMResult callback path where the metadata lives on the AIMessage
+       rather than the LLMResult itself.
+    3. ``response.llm_output['model_name']`` — legacy LLMResult where the
+       chat-model wrapper hoisted the field onto the LLMResult dict.
+    4. ``response.model`` / ``response.model_name`` — direct attributes
+       on the response object (rare but seen in some custom wrappers).
+    """
+    # 1. response_metadata on the response.
+    resp_meta = getattr(response, "response_metadata", None)
+    if isinstance(resp_meta, dict):
+        val = resp_meta.get("model_name") or resp_meta.get("model")
+        if val:
+            return str(val)
+
+    # 2. LLMResult callback path — look on the generation's AIMessage.
+    gen_msg = _safe_get_gen_message(response)
+    if gen_msg is not None:
+        gm = getattr(gen_msg, "response_metadata", None)
+        if isinstance(gm, dict):
+            val = gm.get("model_name") or gm.get("model")
+            if val:
+                return str(val)
+        # Some wrappers put the model name directly on the AIMessage.
+        for attr in ("model_name", "model"):
+            v = getattr(gen_msg, attr, None)
+            if v:
+                return str(v)
+
+    # 3. llm_output dict (legacy LLMResult).
+    llm_out = getattr(response, "llm_output", None)
+    if isinstance(llm_out, dict):
+        val = llm_out.get("model_name") or llm_out.get("model")
+        if val:
+            return str(val)
+
+    # 4. Direct attribute on response.
+    for attr in ("model_name", "model"):
+        v = getattr(response, attr, None)
+        if v:
+            return str(v)
+
+    return None
+
+
+def _extract_provider_from_response(response: Any) -> str | None:
+    """Best-effort provider extraction mirroring ``_extract_model_from_response``.
+
+    Same fallback chain — ``model_provider`` is what langchain passes
+    in ``invocation_params`` and what we want to read from response
+    metadata when invocation_params is absent. Returns ``None`` if
+    nothing is found so the caller keeps the default ('openai').
+    """
+    resp_meta = getattr(response, "response_metadata", None)
+    if isinstance(resp_meta, dict):
+        val = resp_meta.get("model_provider") or resp_meta.get("provider")
+        if val:
+            return str(val)
+
+    gen_msg = _safe_get_gen_message(response)
+    if gen_msg is not None:
+        gm = getattr(gen_msg, "response_metadata", None)
+        if isinstance(gm, dict):
+            val = gm.get("model_provider") or gm.get("provider")
+            if val:
+                return str(val)
+
+    return None
+
diff --git a/src/nullrun/instrumentation/llama_index.py b/src/nullrun/instrumentation/llama_index.py
diff --git a/src/nullrun/runtime.py b/src/nullrun/runtime.py
