Skip to content

FE-1408: Run a trial's seeded simulations in parallel in the Petrinaut CLI, aggregated by mean - #9223

Draft
kube wants to merge 1 commit into
cf/fe-1411-petrinaut-cli-make-the-protocol-handler-async-safefrom
cf/fe-1408-petrinaut-cli-run-a-trials-seeded-simulations-in-parallel
Draft

FE-1408: Run a trial's seeded simulations in parallel in the Petrinaut CLI, aggregated by mean#9223
kube wants to merge 1 commit into
cf/fe-1411-petrinaut-cli-make-the-protocol-handler-async-safefrom
cf/fe-1408-petrinaut-cli-run-a-trials-seeded-simulations-in-parallel

Conversation

@kube

@kube kube commented Aug 16, 2026

Copy link
Copy Markdown
Collaborator

🌟 What is the purpose of this PR?

Optimization trials currently run one simulation with the fixed execution.seed, so a stochastic net is scored on a single replicate. This PR makes the Petrinaut CLI run a trial's execution.seedsPerTrial seeded simulations in parallel on a worker-thread pool and aggregate the per-seed objectives by mean, keeping the {objective} response shape so the Python service needs no logic changes. Simulator side of FE-1273; aggregation-method selection (FE-1277) and Optuna-side handling of per-seed values (FE-1281) are out of scope.

Stack: FE-1410 (contract) → FE-1411 (async-safe protocol) → this PR → arch-docs manual → FE-1270 (Python bindings) → FE-1412 (Python client seeded-trials support).

🔗 Related links

  • FE-1408 (internal) — this PR
  • FE-1273 (internal) — parent: M seeded runs per Optuna step
  • FE-1412 (internal) — top of stack: Python client + image support
  • FE-1277 (internal) — follow-up: user-selectable aggregation

🔍 What does this change?

@hashintel/petrinaut-cli only:

  • optimization.evaluate runs the trial's seeds and returns their mean objective plus a replicates: [{ seed, objective }] array; optimization.describe reports study.seedsPerTrial. Seeds derive once per study: replicate 0 keeps execution.seed (M=1 stays bit-identical to today, and one editor run still reproduces a trial replicate), replicate i uses the exported Monte Carlo deriveRunSeed(seed, i). Every trial reuses the same list → common random numbers across Optuna steps.
  • New worker-thread pool (dist/simulation-worker.js, a second vite entry): each worker compiles the model once at bootstrap, before the readiness line. Pool size is min(seedsPerTrial, cores − 1, 4), overridable via PETRINAUT_CLI_MAX_PARALLEL_SIMULATIONS; an effective size of 1 uses an in-process sequential runner (no worker threads at all — the default path, since seedsPerTrial defaults to 1).
  • Worker crashes fail the in-flight evaluate (the optimizer prunes the trial) and poison the pool with a descriptive error rather than hanging; stdin EOF disposes the pool so the process still exits 0.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

  • modifies a workspace but not a publishable library (@hashintel/petrinaut-cli is private; the @hashintel/petrinaut-core changeset landed with FE-1410)

📜 Does this require a change to the docs?

The changes in this PR:

  • require changes to docs which are made as part of this PR
    • OPTIMIZATION_INTEGRATION.md and the CLI README document the field, seed derivation, response shape, and pool controls. (The upcoming arch-docs PR folds these files into a Usage Manual.) The in-app user guide is untouched: the UI never sets seedsPerTrial yet, so in-app behaviour is unchanged until FE-1273's UI half lands.

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

  • affected the execution graph, and the turbo.json's have been updated to reflect this
    • petrinaut-cli's test:unit now depends on its own build, so the built-CLI smoke test always runs against a fresh bundle in CI.

⚠️ Known issues

  • A worker crash (e.g. OOM inside a worker) poisons the pool: later evaluates in that run fail with the crash message and get pruned. Replacing crashed workers would mask the underlying problem, so the run fails loudly.
  • Until FE-1412 lands (top of this stack), the deployed optimizer image cannot run manifests with seedsPerTrial > 1: its Node permission flags lack --allow-worker and its 240 s response timeout is unscaled. Such runs fail loudly at evaluate/bootstrap; nothing sends such manifests today.

🐾 Next steps

  • FE-1412: Python client + image support (top of stack).
  • FE-1270: extract the Python bindings package the client changes land in.
  • FE-1277 / FE-1281 / FE-1273 UI half: aggregation selection, Optuna-side per-seed handling, drawer controls.

🛡 What tests cover this?

  • optimization.test.ts — mean aggregation, per-seed replicates, identical seed list across trials (CRN), non-finite replicate rejection, deriveTrialSeeds range/stability/uniqueness.
  • simulation-pool.test.ts — pool scheduling and ordering, per-run failure vs. worker crash, readiness failure, dispose.
  • transports.test.ts — end-to-end multi-seed evaluate over stdio, seedsPerTrial in describe.
  • built-cli.test.ts — spawns the real bundled CLI with two worker threads and checks the full describe/evaluate exchange.

❓ How to test this?

  1. turbo run test:unit --filter @hashintel/petrinaut-cli
  2. Manual: add "seedsPerTrial": 4 to execution in libs/@hashintel/petrinaut-cli/examples/supply-chain-profit-optimization.json, yarn workspace @hashintel/petrinaut-cli build, then node libs/@hashintel/petrinaut-cli/dist/cli.js serve --optimization <manifest> --stdio and send an optimization.evaluate request.
  3. Confirm the response carries the mean objective plus four replicates, and that repeating the request returns identical values.

🤖 Generated with Claude Code

@kube kube self-assigned this Aug 16, 2026
@vercel

vercel Bot commented Aug 16, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
hash Ready Ready Preview Aug 16, 2026 5:19pm
petrinaut Ready Ready Preview Aug 16, 2026 5:19pm
1 Skipped Deployment
Project Deployment Actions Updated (UTC)
hashdotdesign-tokens Ignored Ignored Preview Aug 16, 2026 5:19pm

@github-actions github-actions Bot added area/infra Relates to version control, CI, CD or IaC (area) area/libs Relates to first-party libraries/crates/packages (area) type/eng > frontend Owned by the @frontend team area/tests New or updated tests area/apps labels Aug 16, 2026
…t CLI

Optimization trials now run execution.seedsPerTrial seeded simulations on
a worker-thread pool (dist/simulation-worker.js, pool capped by
PETRINAUT_CLI_MAX_PARALLEL_SIMULATIONS or min(cores-1, 4), in-process
sequential fallback at 1), aggregated into the trial objective by mean.
Seeds derive deterministically from execution.seed - replicate 0 keeps
the base seed, later replicates reuse the Monte Carlo derivation - and
every trial reuses the same list, so Optuna compares configurations
under common random numbers. The evaluate response stays
{objective}-compatible and reports per-seed replicates; describe reports
seedsPerTrial. The deployed optimizer image needs FE-1412 (--allow-worker
and its bootstrap/timeout handling) before manifests with
seedsPerTrial > 1 can run there.
@kube
kube force-pushed the cf/fe-1408-petrinaut-cli-run-a-trials-seeded-simulations-in-parallel branch from 797689e to ec38102 Compare August 16, 2026 17:10
@github-actions github-actions Bot removed area/tests New or updated tests area/apps labels Aug 16, 2026
@kube
kube changed the base branch from main to cf/fe-1411-petrinaut-cli-make-the-protocol-handler-async-safe August 16, 2026 17:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/infra Relates to version control, CI, CD or IaC (area) area/libs Relates to first-party libraries/crates/packages (area) type/eng > frontend Owned by the @frontend team

Development

Successfully merging this pull request may close these issues.

1 participant