benchmarks: reproducible factoring instance generator#7
Merged
Conversation
benchmarks/gen_factoring.py emits factoring_<n>x<n>.jsonl (10 balanced n-bit semiprimes per size, one CircuitSAT bundle per line via `pred create Factoring | pred reduce --to CircuitSAT`). Per-size seeding makes any subset regenerate byte-identically; line 0 of each size is a pinned N (historical fixtures / the 22x22 session instance) so prior results stay reproducible. The 11 MB of generated JSONL under benchmarks/data/ is gitignored — reproduce locally with `python3 benchmarks/gen_factoring.py 12 14 16 18 20 22`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nchmark set The 12/16/18/20/22 single-instance CircuitSAT fixtures are replaced by the 10-per-size benchmark set (benchmarks/gen_factoring.py; each set's line 0 pins the old N). None are referenced in tracked code. factoring_15.circuitsat.json stays — tests/factoring.rs include_str!'s it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a deterministic generator for the factoring benchmark set — 10 balanced n-bit semiprimes per size (12/14/16/18/20/22), one
pred-reduced CircuitSAT bundle per JSONL line.benchmarks/gen_factoring.py: per-size seeding (Random(SEED_BASE + n)), so any subset regenerates byte-identically regardless of invocation order. Line 0 of each size is a pinned N (the historical single-fixture N, or the 22x22 session instance8750074000153) so earlier results stay reproducible; the rest are random balanced semiprimes (both factors in[2^(n-1), 2^n)— the hard case).benchmarks/data/*.jsonlis regenerable, so only the generator is tracked. Reproduce withpython3 benchmarks/gen_factoring.py 12 14 16 18 20 22(needspredon PATH).Purpose: replaces the single-instance-per-size fixtures with a proper multi-instance set for statistically meaningful benchmarking (goal C / WS0). A JSONL-reading benchmark runner is a separate follow-up.
Only adds
benchmarks/gen_factoring.pyand a.gitignoreline — no solver code, no test changes.🤖 Generated with Claude Code