Generalize findings: each backend says what its evidence is, the host writes it up - #185
Open
ericeil wants to merge 5 commits into
Open
Generalize findings: each backend says what its evidence is, the host writes it up#185ericeil wants to merge 5 commits into
ericeil wants to merge 5 commits into
Conversation
Findings were synthesized inside the report phase, from evidence only the prover produces. A `RuleEvidence` with a root-cause analysis and a CVL counterexample is what a symbolic prover has; a fuzzing backend has a crashing input and an account of what its run covered, and no analysis at all. `Formalizer.findings_policy(outcomes)` replaces `findings_evidence`. It returns a `FindingsPolicy` — or `None`, which is how a backend produces no findings and never starts the heavy model. Three things are the backend's, and only the first is a function: - `fetch_evidence`, keyed by `RuleRef` — `(file, name)`, how the report already identifies a row. A check name alone does not: one deliverable can hold several components' checks, and two authors given the same property write the same name. - `domain`, the claim about what this backend's evidence *is*. The host wraps it in the shared contract — how severity is reached, which sections come back — so `autoprove_report_findings_system.j2` is one template for every backend and the prover's half is a peer of any other's. - `prompt`, a `TypedTemplate`, not a callable. Both backends' prompts take the same fields, so the backend owns the prose and `build_findings` owns the binding. Everything else is shared: walking the BAD rows, resolving properties and groups, grouping rows that share one finding, the proof of concept, the concurrency, composing the `Finding`. Severity too — the model rates impact and likelihood and `severity_for` maps the pair, so no backend picks a tier and the one a reader sees is re-derivable from the write-up. `RuleEvidence` grows the four fields a non-prover backend has (`ran`, `accounting`, `declared`, `finding`) and stays one shared type in `collect.py`, where it already lived. A backend fills what it has; absence always means "this run recorded nothing of that kind", so a prompt can say what is missing instead of guessing.
A Rust wheel now takes part in the shared findings seam. Three wire additions carry it, all `deny_unknown_fields` + `required::present`, so a wheel that has not been rebuilt fails at the seam rather than silently losing a field: - `AppDescriptor.findings: Option<FindingsDeclaration>` — the domain half of the write-up system prompt. `None` declines, and the example app declines: reading a spec back is its whole checker, so a refuted rule says the spec did not say what was asked, nothing about the program. Declining is the default because a write-up asserts what its evidence is, and a host that guessed would be publishing prose nothing stands behind. - `Verdict.accounting` — what the run spent and covered, kept apart from `detail`. They are separate claims: one is evidence about the program, the other about the run, and a proof of concept padded with accounting leaves a reader unable to see where the evidence ends. `fetch_verdicts` rejoins them into the report row's one `message`, evidence first. - `Verdict.finding` — an opaque key for a conclusion the wheel could not attribute to one check. Rows stamped alike are written up once, against the set. Stamped and never inferred: fanned-out rows are otherwise indistinguishable from several checks that failed the same way, and those are two different facts about the program. The author's `expect_check_failure` reasons now reach the result, and `reported_verdicts()` folds them in on the way out: a declared check reports BAD whatever the run said, with `NOT REPRODUCED` in the detail when the run did not hit it. The gate accepts such a check as clean without requiring a repro, so nothing else stood between a documented finding and a green report row. `verdicts` stays verbatim — attribution remains the wheel's. `composer/rustapp/findings.py` reads a run's results into `RuleEvidence` off wire fields alone, leaving `analysis` unset: a wheel reports what its run found, not a reading of why the check broke, and a reader must be able to tell those apart.
`formalization-abstraction.md` gains §4.6 for `findings_policy`: what the shared loop does, the `RuleEvidence` field table, and why the `FindingsPolicy` is a record rather than a table of hooks — a Rust wheel ships one of its fields across the FFI boundary as JSON, and anything that survives serialization was never behaviour. `rust-applications.md` §4.5 gains the wheel's side: the declaration, what the host fills into `RuleEvidence` from wire fields, why `analysis` stays empty, why severity is not the wheel's to opt out of, and why `accounting` and `detail` are separate fields. §5/§6 record the `expect_check_failure` fold.
Collapsing on a finding stamp kept only the first rule's properties, groups, and provenance, and the Rust write-up still led with that row as the failing check. The host now binds the union, records the covered refs on `provenance.covers`, and lists every name together so the model is not asked to pick a subject the stamp says it cannot. How far the evidence goes stays in `domain`: the shared system prompt no longer closes with "rate only what a real actor could reach", which had overwritten the Prover's "a counterexample is a confirmed break". Docs and comments state the seam as a backend-neutral contract.
…ields `finding` is `finding_key` (`FindingKey`). The report row carries `message`, `accounting`, and `expected_failure` separately: reproduced vs unreproduced is a tagged union, not prose stuffed into `detail`. HTML and the findings prompt compose from those fields. Collapse is keyed by `FindingKey | RuleRef`. Comments and docs state the seam as a backend-neutral contract.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
The report's Findings section existed, but the only way to fill it was an evidence fetcher that
meant "run the Certora Prover write-up". Returning a fetcher opted you into a heavy-model pass
whose prompt asserts the Certora Prover found a concrete counterexample. Foundry and the Rust
host returned
Noneto opt out of code that could never have served them.This generalizes that seam so any backend — including a Rust-authored one — can answer what did
this run find in its own terms. The write-up loop stays in one place.
The seam
Formalizer.findings_policy(outcomes) -> FindingsPolicy | None.Nonemeans the backend producesno findings and never starts the heavy model.
A
FindingsPolicyis three fields, and only the first is a function:fetch_evidenceRuleRef—(file, name), how the report already identifies a row. A check name alone does not: one deliverable can hold several components' checks. The only hook, because it is the only one that does I/O.domainFindingsDeclaration.domain.promptTypedTemplate[FindingsPromptParams]. The Prover supplies its own; every Rust-authored backend shares one host template. The loop binds the same fields either way.Everything else is shared in
report/findings.py: walking the BAD rows, resolving properties andaudit groups, collapsing rows that share one finding, the proof of concept, composing the
Finding. A failure there costs the findings, never the report.Evidence has one shape
Every backend hands back the same
RuleEvidence. Backends differ in what they can fill, not inthe shape. Absence always means "this run recorded nothing of that kind".
labelanalysiscounterexampleranaccountingexpected_failure_reasonfinding_keyOne system prompt, one severity rule
A findings system prompt splits the same way for every backend: what this evidence is (and how
far it goes) is the backend's
domain; how severity is reached and which sections come back isthe host's contract (
autoprove_report_findings_system.j2).The model rates impact and likelihood;
severity_formaps the pair through a fixed matrix. Nobackend picks a tier.
Report rows keep evidence, accounting, and expected-failure apart
The report
Verdict/RuleVerdictcarry three optional fields:message— the counterexample or error (detailon the rust wire)accounting— what the run spent (Nonefor backends that do not report it)expected_failure—ReproducedExpectedFailureorUnreproducedExpectedFailure(reason +,when unreproduced, the outcome the run actually reached)
HTML shows them as separate blocks. The findings write-up reads the same split off
RuleEvidence, so a proof of concept is never padded with accounting or declaration prose.Rust-authored backends
In this repo, a Rust-authored backend is a wheel: compiled Rust that the Python host drives
through an FFI. Three wire additions, all
deny_unknown_fields+required::present:AppDescriptor.findings: Option<FindingsDeclaration>— opt-in. The domain string is whatthe evidence is. Default is
None(no findings). The example app opts out.Verdict.accounting— what the run spent and covered, kept apart fromdetail.Verdict.finding_key: Option<FindingKey>— an opaque, run-global key for a conclusion thewheel could not attribute to one check. Rows stamped alike are written up once, against the
union of their properties and groups.
provenance.coversnames every row the write-up answersfor. Stamped and never inferred.
composer/rustapp/findings.pymaps wire fields intoRuleEvidenceand leavesanalysisunset.Expected-to-fail checks still report as findings
A check marked
expect_check_failureis a documented finding even if this run did not hit it.The publish gate accepts such a check as clean without requiring a repro.
expected_failuresridesRustFormalResult.reported_verdicts()folds the outcome toBADand leavesdetailas the run reported it.expected_failure()builds the tagged unionfor the report row. The console rollup reads the same outcome fold, so the terminal and
report.htmlcannot disagree about whether the run found something.Docs
formalization-abstraction.md§4.6 covers the hook and the evidence table.rust-applications.md§4.5 covers the wheel's side, and §5/§6 theexpect_check_failurefold.Validation
Cheap suite and
pyrightclean. The first commit stands alone (the suite passes there with norustapp participation).