Skip to content

Add vein estimator validation tools and benchmark record - #21

Open
Toti330 wants to merge 2 commits into
optimize/generation-pipelinefrom
optimize/planet-vein-estimator
Open

Add vein estimator validation tools and benchmark record#21
Toti330 wants to merge 2 commits into
optimize/generation-pipelinefrom
optimize/planet-vein-estimator

Conversation

@Toti330

@Toti330 Toti330 commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add a database-free exact vein exporter and deterministic candidate fixtures
  • add locked corpus validation, held-out protection, accuracy gates, scale fitting, and elapsed-overhead reporting
  • publish raw calibration and baseline artifacts plus a concise Markdown benchmark timeline

Scope

  • no production generation behavior changes
  • the recorded spacing estimator is rejected: it misses P95 and per-ore WAPE requirements
  • held-out data remains unopened and candidate performance remains unmeasured because calibration failed

Validation

  • PYTHONPATH=benchmarks python3 -m unittest benchmarks/test_evaluate_vein_accuracy.py
  • cargo test --release --locked in the pinned Rust 1.97.0 builder
  • JSON parsing and git diff --check
  • held-out exporter guard rejects unapproved access

Formatting Note

rustfmt --check reports existing formatting drift in shared algorithm modules and an existing section of generate_csv.rs; this PR does not reformat unrelated production code.

@Toti330

Toti330 commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator Author

The original goal was to increase seed-generation throughput without changing deterministic output, a current run has produced reproducible validation evidence, and that is a point where an unsuccessful approximation should be documented rather than promoted. The agent was not able to produce a vein estimator that met the predeclared accuracy targets, so production generation remains unchanged and the exact same as before.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant