Skip to content

Run, observe, and analyze Slurm benchmarks #877

Description

@nabinchha

Priority Level

High

Task Summary

Implement deterministic benchmark expansion, batch execution, fresh-process observation, and analysis for the optional Slurm integration.

Technical Details & Implementation Plan

  • Expand strict benchmark intent into deterministic child run configurations and immutable benchmark records.
  • Submit child runs through the public Slurm execution service and return without resident monitoring.
  • Persist benchmark-to-run identity so later processes can observe and analyze the same children.
  • Refresh child state through normalized scheduler and persisted-state evidence.
  • Compute stable aggregate analysis from validated child results while preserving failed, incomplete, and missing-run classifications.
  • Expose equivalent Python service and CLI operations for benchmark run and analyze workflows.

Acceptance criteria

  • Equivalent benchmark input expands to the same ordered child runs and digests.
  • Benchmark submission returns after scheduling children and does not require a resident controller.
  • Analysis works from a fresh process and never guesses success from incomplete evidence.
  • Missing, failed, partial, stale, and scheduler-inconsistent child runs remain explicit in results.
  • Local/fake tests cover expansion, submission, refresh, mixed outcomes, and deterministic analysis.
  • Real-cluster acceptance validates at least one multi-run benchmark workflow before release.

Out of scope

  • Interactive dashboards or resident monitoring.
  • New benchmark algorithms unrelated to Slurm execution.
  • Generic scheduler or platform adapters.

Investigation / Context

This is the benchmark implementation lane in #850. #865 and #872 explicitly exclude benchmark implementation, while #870 treats benchmark analysis as a final acceptance scenario.

Agent Plan / Findings

Reuse the same immutable planning, execution, and state contracts as ordinary runs; benchmark records should add hierarchy and analysis intent rather than a second control plane.

Dependencies

Depends on shared benchmark records in #873, public services in #874, deterministic planning in #875, the client worker in #876, fake infrastructure in #872, runtime in #868, and persistent state in #869.

Metadata

Metadata

Assignees

Labels

taskInternal development task

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions