Skip to content

Add versioned eval-artifact conformance contract (#31) - #61

Open
greekera1000 wants to merge 1 commit into
evalops:mainfrom
greekera1000:feat/eval-artifacts-conformance-contract
Open

Add versioned eval-artifact conformance contract (#31)#61
greekera1000 wants to merge 1 commit into
evalops:mainfrom
greekera1000:feat/eval-artifacts-conformance-contract

Conversation

@greekera1000

Copy link
Copy Markdown

Summary

Orbit Agent's eval subsystem emits artifacts other tools consume — a JSONL
record per scenario (eval run), a run summary (eval report), and CSV/Markdown
tables (eval summary) — but their shapes are only implied by the code and
described in the README. A routine refactor could rename a field, flip a format
rule, or drop a CSV column, pass the tests, and silently break consumers.

This PR promotes those existing guarantees into a versioned conformance
contract
plus a deterministic, stdlib-only verifier that fails when the code
drifts. It extracts behavior already in evals.py / cli.py and the README — it
doesn't invent a new spec.

Closes #31.

Acceptance criteria (#31)

Criterion Where
Design note: workflow, correctness/threat model, research assumptions contracts/eval-artifacts-v1.md
Runnable verifier in CI or locally conformance/verify.py; make contract; pytest tests/test_contract_conformance.py
Evidence citing inputs, decisions, outputs Report.evidence() → JSON (inputs + SHA-256, per-rule decisions, verdict)
Negative/degraded case, observable tests/fixtures/conformance/ (missing field, format_ok lie, wrong type, bad CSV)
Docs link to EvalOps primitive or record standalone rationale design note → "EvalOps linkage / standalone rationale"

What's under contract

Pinned in conformance/contract.py (CONTRACT_VERSION = 1.0.0): the EvalRecord
JSONL schema, count consistency, the format rule (3–5 actions, exactly 3 risks,
non-empty advice) and its format_ok signal, the summarize_results keys, the
CSV/Markdown summary schemas, and the Typer CLI surface. Each requirement cites
its source function in the design note.

Design decisions worth reviewing

  • Stdlib-only. conformance/ never imports orbit_agent (which pulls in
    dspy), so it runs in pytest and as python -m conformance.verify with no
    model provider.
  • Fail-closed. A required rule the verifier can't confirm is a violation, not
    a silent pass.
  • Format rule verified behaviorally. _format_eval is extracted, executed in
    isolation, and probed at its boundaries — so an equivalent refactor still
    passes, and only a real behavior change (or an unevaluable function) fails.
  • CLI check inspects the Typer declaration, not just parameter names, so a
    renamed flag or an option⇄argument swap is caught.

Verify locally

python -m conformance.verify
pytest -q tests/test_contract_conformance.py

Expected: PASS: eval-artifacts v1.0.0 - 0 violation(s) and 14 passed. The
conformance tests need no dspy; the full repo suite still uses the dev deps.

Scope

Correctness/compatibility contract, not a security boundary. The CLI check reads
static Typer declarations, not runtime dispatch. Shipping standalone; changing an
artifact shape is expected to bump CONTRACT_VERSION and the note together.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Promote latent specs into a documented conformance contract for orbit agent (brutally honest high orbit startup)

1 participant