Skip to content

Compile Slurm configuration into deterministic execution plans #875

Description

@nabinchha

Priority Level

High

Task Summary

Resolve strict authored Slurm input and cluster profiles into an immutable, deterministic execution plan before any scheduler submission occurs.

Technical Details & Implementation Plan

  • Load and validate authored run configuration and the selected cluster profile.
  • Resolve Data Designer invocation inputs, deployments, images, resources, mounts, shard ranges, client placement, and output intent into package-owned plan records.
  • Keep authored semantics separate from resolved scheduler/runtime facts.
  • Produce stable plan serialization and digests suitable for pinning in run, shard, and attempt records.
  • Validate cross-field resource, placement, path, and deployment constraints before submission.
  • Provide pure planning entry points that require no Slurm commands or runtime services.

Acceptance criteria

  • The same authored input, profile, and referenced artifacts produce identical plan bytes and digests.
  • Invalid deployment, placement, resource, mount, shard, and destination combinations fail before submission.
  • Plans contain all inputs required by serving, image, launcher/runtime, client-worker, state, and benchmark lanes.
  • Plans contain no credentials, mutable source references, or unresolved environment-dependent defaults.
  • Unit and golden tests cover single-node, multi-node, multi-deployment, sharded, and invalid configurations.

Out of scope

  • Slurm submission or command parsing.
  • Starting model services or the Data Designer client.
  • State persistence, retry, collection, or benchmark execution.

Investigation / Context

This is the configuration-resolution and deterministic-plan lane in #850. #866 and #868 consume planner-supplied placement and resources but do not own authored configuration or plan compilation.

Agent Plan / Findings

Keep planning pure and deterministic so every side effecting lane consumes a reviewed immutable plan rather than re-resolving authored input.

Dependencies

Depends on the public contract in #851, the optional package foundations in #852 and #853, and the shared records in #873.

Metadata

Metadata

Assignees

Labels

taskInternal development task

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions