Skip to content

Cross-repo evaluation harness #4788

Description

@JSONbored

Problem: Without a real benchmark, there's no way to know whether the "generalized" agent is actually general, or just quietly still tuned to gittensory's own repo shape.

Area: AMS / Miner / QA

Proposal: Assemble a benchmark set of real, diverse external repositories (different languages and stacks) and run the full loop against each, scoring success objectively (task completed, target repo's own CI passing, no gittensory-specific assumption triggered).

Deliverables:

  • A benchmark repo set and a repeatable evaluation run producing a pass/fail report per repo.

Acceptance criteria:

  • The loop succeeds against a majority of the benchmark set without any gittensory-specific configuration being required.
  • Failures are categorized (stack-detection gap vs. execution gap vs. something else) rather than left as an unexplained pass/fail.

Boundaries:


Part of #4778.

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:featureGittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.

    Projects

    Status
    Done

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions