Add versioned eval-artifact conformance contract (#31) - #61
Open
greekera1000 wants to merge 1 commit into
Open
Conversation
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Orbit Agent's eval subsystem emits artifacts other tools consume — a JSONL
record per scenario (
eval run), a run summary (eval report), and CSV/Markdowntables (
eval summary) — but their shapes are only implied by the code anddescribed in the README. A routine refactor could rename a field, flip a format
rule, or drop a CSV column, pass the tests, and silently break consumers.
This PR promotes those existing guarantees into a versioned conformance
contract plus a deterministic, stdlib-only verifier that fails when the code
drifts. It extracts behavior already in
evals.py/cli.pyand the README — itdoesn't invent a new spec.
Closes #31.
Acceptance criteria (#31)
contracts/eval-artifacts-v1.mdconformance/verify.py;make contract;pytest tests/test_contract_conformance.pyReport.evidence()→ JSON (inputs + SHA-256, per-rule decisions, verdict)tests/fixtures/conformance/(missing field,format_oklie, wrong type, bad CSV)What's under contract
Pinned in
conformance/contract.py(CONTRACT_VERSION = 1.0.0): theEvalRecordJSONL schema, count consistency, the format rule (3–5 actions, exactly 3 risks,
non-empty advice) and its
format_oksignal, thesummarize_resultskeys, theCSV/Markdown summary schemas, and the Typer CLI surface. Each requirement cites
its source function in the design note.
Design decisions worth reviewing
conformance/never importsorbit_agent(which pulls indspy), so it runs inpytestand aspython -m conformance.verifywith nomodel provider.
a silent pass.
_format_evalis extracted, executed inisolation, and probed at its boundaries — so an equivalent refactor still
passes, and only a real behavior change (or an unevaluable function) fails.
renamed flag or an option⇄argument swap is caught.
Verify locally
Expected:
PASS: eval-artifacts v1.0.0 - 0 violation(s)and14 passed. Theconformance tests need no
dspy; the full repo suite still uses the dev deps.Scope
Correctness/compatibility contract, not a security boundary. The CLI check reads
static Typer declarations, not runtime dispatch. Shipping standalone; changing an
artifact shape is expected to bump
CONTRACT_VERSIONand the note together.