-
-
Notifications
You must be signed in to change notification settings - Fork 70
Cross-repo evaluation harness #4788
Copy link
Copy link
Closed
Labels
gittensor:featureGittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.
Description
Metadata
Metadata
Assignees
Labels
gittensor:featureGittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.
Projects
StatusShow more project fields
Done
Problem: Without a real benchmark, there's no way to know whether the "generalized" agent is actually general, or just quietly still tuned to gittensory's own repo shape.
Area: AMS / Miner / QA
Proposal: Assemble a benchmark set of real, diverse external repositories (different languages and stacks) and run the full loop against each, scoring success objectively (task completed, target repo's own CI passing, no gittensory-specific assumption triggered).
Deliverables:
Acceptance criteria:
Boundaries:
Part of #4778.