Skip to content

Generalize findings: each backend says what its evidence is, the host writes it up - #185

Open
ericeil wants to merge 5 commits into
masterfrom
eric/findings
Open

Generalize findings: each backend says what its evidence is, the host writes it up#185
ericeil wants to merge 5 commits into
masterfrom
eric/findings

Conversation

@ericeil

@ericeil ericeil commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

What this is

The report's Findings section existed, but the only way to fill it was an evidence fetcher that
meant "run the Certora Prover write-up". Returning a fetcher opted you into a heavy-model pass
whose prompt asserts the Certora Prover found a concrete counterexample. Foundry and the Rust
host returned None to opt out of code that could never have served them.

This generalizes that seam so any backend — including a Rust-authored one — can answer what did
this run find
in its own terms. The write-up loop stays in one place.

The seam

Formalizer.findings_policy(outcomes) -> FindingsPolicy | None. None means the backend produces
no findings and never starts the heavy model.

A FindingsPolicy is three fields, and only the first is a function:

Field Role
fetch_evidence Keyed by RuleRef(file, name), how the report already identifies a row. A check name alone does not: one deliverable can hold several components' checks. The only hook, because it is the only one that does I/O.
domain What this backend's evidence is. True of one backend and false of another. A Rust-authored backend ships this string as FindingsDeclaration.domain.
prompt A TypedTemplate[FindingsPromptParams]. The Prover supplies its own; every Rust-authored backend shares one host template. The loop binds the same fields either way.

Everything else is shared in report/findings.py: walking the BAD rows, resolving properties and
audit groups, collapsing rows that share one finding, the proof of concept, composing the
Finding. A failure there costs the findings, never the report.

Evidence has one shape

Every backend hands back the same RuleEvidence. Backends differ in what they can fill, not in
the shape. Absence always means "this run recorded nothing of that kind".

Field
label Which instance, where a rule can fail more than once
analysis Why the check failed, when the backend produces a root-cause account
counterexample What reproduces it
ran The run's own outcome, before any expected-to-fail mark is folded in
accounting What the run spent and covered
expected_failure_reason Why the author marked this check expected to fail, when they did
finding_key Opaque grouping key, when one conclusion covers several checks

One system prompt, one severity rule

A findings system prompt splits the same way for every backend: what this evidence is (and how
far it goes) is the backend's domain; how severity is reached and which sections come back is
the host's contract (autoprove_report_findings_system.j2).

The model rates impact and likelihood; severity_for maps the pair through a fixed matrix. No
backend picks a tier.

Report rows keep evidence, accounting, and expected-failure apart

The report Verdict / RuleVerdict carry three optional fields:

  • message — the counterexample or error (detail on the rust wire)
  • accounting — what the run spent (None for backends that do not report it)
  • expected_failureReproducedExpectedFailure or UnreproducedExpectedFailure (reason +,
    when unreproduced, the outcome the run actually reached)

HTML shows them as separate blocks. The findings write-up reads the same split off
RuleEvidence, so a proof of concept is never padded with accounting or declaration prose.

Rust-authored backends

In this repo, a Rust-authored backend is a wheel: compiled Rust that the Python host drives
through an FFI. Three wire additions, all deny_unknown_fields + required::present:

  • AppDescriptor.findings: Option<FindingsDeclaration> — opt-in. The domain string is what
    the evidence is. Default is None (no findings). The example app opts out.
  • Verdict.accounting — what the run spent and covered, kept apart from detail.
  • Verdict.finding_key: Option<FindingKey> — an opaque, run-global key for a conclusion the
    wheel could not attribute to one check. Rows stamped alike are written up once, against the
    union of their properties and groups. provenance.covers names every row the write-up answers
    for. Stamped and never inferred.

composer/rustapp/findings.py maps wire fields into RuleEvidence and leaves analysis unset.

Expected-to-fail checks still report as findings

A check marked expect_check_failure is a documented finding even if this run did not hit it.
The publish gate accepts such a check as clean without requiring a repro.

expected_failures rides RustFormalResult. reported_verdicts() folds the outcome to
BAD and leaves detail as the run reported it. expected_failure() builds the tagged union
for the report row. The console rollup reads the same outcome fold, so the terminal and
report.html cannot disagree about whether the run found something.

Docs

formalization-abstraction.md §4.6 covers the hook and the evidence table.
rust-applications.md §4.5 covers the wheel's side, and §5/§6 the expect_check_failure fold.

Validation

Cheap suite and pyright clean. The first commit stands alone (the suite passes there with no
rustapp participation).

Findings were synthesized inside the report phase, from evidence only the
prover produces. A `RuleEvidence` with a root-cause analysis and a CVL
counterexample is what a symbolic prover has; a fuzzing backend has a crashing
input and an account of what its run covered, and no analysis at all.

`Formalizer.findings_policy(outcomes)` replaces `findings_evidence`. It returns
a `FindingsPolicy` — or `None`, which is how a backend produces no findings and
never starts the heavy model.

Three things are the backend's, and only the first is a function:

- `fetch_evidence`, keyed by `RuleRef` — `(file, name)`, how the report already
  identifies a row. A check name alone does not: one deliverable can hold
  several components' checks, and two authors given the same property write the
  same name.
- `domain`, the claim about what this backend's evidence *is*. The host wraps it
  in the shared contract — how severity is reached, which sections come back —
  so `autoprove_report_findings_system.j2` is one template for every backend and
  the prover's half is a peer of any other's.
- `prompt`, a `TypedTemplate`, not a callable. Both backends' prompts take the
  same fields, so the backend owns the prose and `build_findings` owns the
  binding.

Everything else is shared: walking the BAD rows, resolving properties and
groups, grouping rows that share one finding, the proof of concept, the
concurrency, composing the `Finding`. Severity too — the model rates impact and
likelihood and `severity_for` maps the pair, so no backend picks a tier and the
one a reader sees is re-derivable from the write-up.

`RuleEvidence` grows the four fields a non-prover backend has (`ran`,
`accounting`, `declared`, `finding`) and stays one shared type in `collect.py`,
where it already lived. A backend fills what it has; absence always means "this
run recorded nothing of that kind", so a prompt can say what is missing instead
of guessing.
A Rust wheel now takes part in the shared findings seam. Three wire additions
carry it, all `deny_unknown_fields` + `required::present`, so a wheel that has
not been rebuilt fails at the seam rather than silently losing a field:

- `AppDescriptor.findings: Option<FindingsDeclaration>` — the domain half of the
  write-up system prompt. `None` declines, and the example app declines: reading
  a spec back is its whole checker, so a refuted rule says the spec did not say
  what was asked, nothing about the program. Declining is the default because a
  write-up asserts what its evidence is, and a host that guessed would be
  publishing prose nothing stands behind.
- `Verdict.accounting` — what the run spent and covered, kept apart from
  `detail`. They are separate claims: one is evidence about the program, the
  other about the run, and a proof of concept padded with accounting leaves a
  reader unable to see where the evidence ends. `fetch_verdicts` rejoins them
  into the report row's one `message`, evidence first.
- `Verdict.finding` — an opaque key for a conclusion the wheel could not
  attribute to one check. Rows stamped alike are written up once, against the
  set. Stamped and never inferred: fanned-out rows are otherwise
  indistinguishable from several checks that failed the same way, and those are
  two different facts about the program.

The author's `expect_check_failure` reasons now reach the result, and
`reported_verdicts()` folds them in on the way out: a declared check reports BAD
whatever the run said, with `NOT REPRODUCED` in the detail when the run did not
hit it. The gate accepts such a check as clean without requiring a repro, so
nothing else stood between a documented finding and a green report row.
`verdicts` stays verbatim — attribution remains the wheel's.

`composer/rustapp/findings.py` reads a run's results into `RuleEvidence` off
wire fields alone, leaving `analysis` unset: a wheel reports what its run found,
not a reading of why the check broke, and a reader must be able to tell those
apart.
`formalization-abstraction.md` gains §4.6 for `findings_policy`: what the shared
loop does, the `RuleEvidence` field table, and why the `FindingsPolicy` is a
record rather than a table of hooks — a Rust wheel ships one of its fields
across the FFI boundary as JSON, and anything that survives serialization was
never behaviour.

`rust-applications.md` §4.5 gains the wheel's side: the declaration, what the
host fills into `RuleEvidence` from wire fields, why `analysis` stays empty, why
severity is not the wheel's to opt out of, and why `accounting` and `detail` are
separate fields. §5/§6 record the `expect_check_failure` fold.
Collapsing on a finding stamp kept only the first rule's properties, groups,
and provenance, and the Rust write-up still led with that row as the failing
check. The host now binds the union, records the covered refs on
`provenance.covers`, and lists every name together so the model is not asked
to pick a subject the stamp says it cannot.

How far the evidence goes stays in `domain`: the shared system prompt no
longer closes with "rate only what a real actor could reach", which had
overwritten the Prover's "a counterexample is a confirmed break". Docs and
comments state the seam as a backend-neutral contract.
…ields

`finding` is `finding_key` (`FindingKey`). The report row carries `message`,
`accounting`, and `expected_failure` separately: reproduced vs unreproduced is
a tagged union, not prose stuffed into `detail`. HTML and the findings prompt
compose from those fields.

Collapse is keyed by `FindingKey | RuleRef`. Comments and docs state the seam
as a backend-neutral contract.
@ericeil
ericeil requested review from jtoman and naftali-g and a lite review from Copilot and removed request for Copilot August 21, 2026 22:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant