Skip to content

feat(campaign): discriminative scenario selection (E2)#321

Merged
drewstone merged 1 commit into
mainfrom
feat/discriminative-scenarios
Jul 7, 2026
Merged

feat(campaign): discriminative scenario selection (E2)#321
drewstone merged 1 commit into
mainfrom
feat/discriminative-scenarios

Conversation

@drewstone

Copy link
Copy Markdown
Contributor

Additive primitive for research claim E2: select holdout scenarios by DISCRIMINATION power instead of a random/balanced split.

Motivation — the OR benchmark is saturating

Run 7 measured ~75% tied holdout cells: most problems are solved optimally by the baseline AND every candidate, so those paired cells carry zero signal. A random/balanced holdout spends its budget on scenarios that cannot separate candidates, and a tie in a paired cell wastes it.

What this adds

src/campaign/scenario-selection.ts:

  • scoreDiscrimination(signals, opts?) — ranks scenarios by how well they separate candidates. discrimination = variance of the per-candidate composite scores (higher spread = more signal). Flags tied when variance < 1e-9 && meanScore >= saturationCeiling (default 0.999): every candidate scored the same at the ceiling, i.e. a saturated tie with no signal. Deterministic sort: discrimination desc, then meanScore asc (more headroom first), then scenarioId asc.
  • selectDiscriminative(signals, k, opts?) — returns the top-k scenario ids for a holdout, EXCLUDING saturated ties when enough non-tied scenarios exist. Falls back to the least-saturated ties only when non-tied < k. Throws if k < 1; returns all ids in discrimination order when signals.length <= k.

Deterministic by construction (no Date/Math.random). Exported from src/campaign/index.ts.

Tests

src/campaign/scenario-selection.test.ts (11 tests, all passing): high-variance ranks above low-variance; all-candidates-score-1.0 flagged tied; an all-equal-but-LOW scenario is NOT tied (headroom remains); custom saturationCeiling honored; ties excluded when enough non-tied exist; fallback to least-saturated ties when non-tied < k; determinism; k<1 throws; k>=n returns all in order.

tsc --noEmit clean, biome check clean.

Additive; tested. No changes to existing behavior.

@tangletools tangletools left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Auto-approved drewstone PR — f3ee4519

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

tangletools · auto-approval · reason: drewstone_author · 2026-07-07T17:25:23Z

@drewstone drewstone merged commit 42ea7ab into main Jul 7, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants