perf(scheduler): improve utilization and scale

[eric-tramel]

📋 Summary

Makes async task admission work-conserving when only one scheduling group is runnable, bounds fair-queue priority state by active groups, compacts checkpointed completion state, and removes repeated synchronous metadata checkpointing from long generations.

The admission policy still tracks borrow debt and yields the next released slot to a waiting peer. This removes speculative idle capacity without adding a tuning knob or scheduler abstraction.

This is the continuously updated best-case branch for telemetry-guided scheduler optimization. It remains a draft while the benchmark matrix expands.

Optimization is gated on five simultaneous outcomes, not a single timing number: (1) inference idle time/utilization, (2) end-to-end throughput, (3) production LoC/scheduler complexity, (4) automatic operation with no new user tuning, and (5) smooth scale from a few records to millions. The retained E2/E3/E4/E6 changes are net +16 nonblank production LoC and add no file, dependency, public API, feedback controller, or user knob. Small/medium real-inference results and 1M-record scheduler/checkpoint results are reported separately; the 1M result is not an inference-throughput claim.

🔗 Related Issue

No linked issue; this follows the async-scheduling telemetry work in #800 and #802.

🔄 Changes

• Let a solo bounded-borrow group use all available scheduler resource slots.
• Preserve peer handoff through existing borrow-debt accounting after the next incumbent completion.
• Replace the task-history heap with priority state for currently active scheduling groups only.
• Compact CompletionTracker row-group state after checkpointing and after every worker for that group has finished.
• Write resume metadata after the first durable row group and refresh it at completion; resume already scans parquet files for progress.
• Add focused policy, bounded-queue-state, checkpoint, compaction, metadata-cadence, and concurrent failed/slow fan-out coverage.

📈 Performance

Deterministic capacity-8 heavy-root/late-peer simulation on macOS arm64, 768 tasks:

Metric	origin/main	This PR	Delta
Wall time	36.70 s	34.00 s	-7.36%
Model-wait slot utilization	91.55%	98.82%	+7.27 pp
Idle slot-seconds	24.80 s	3.20 s	-87.1%
Generation-column idle	24.05 s	21.35 s	-11.2%
Peer p95 wait	11.95 s	12.40 s	+3.8%

The peer p95 remains 62.9% lower than the strict work-conserving policy. A neutral two-group control remains unchanged at 6.40 s and 100% utilization.

A paired 2,172-task sweep across fan-out, fan-in, bottleneck, long-prefill, and mixed-duration DAGs produced two wall-time wins, three ties, and no regressions. Mean wall time improved 0.91% and mean utilization improved 0.90 percentage points; all caps held and every run ended with zero leases. The explicit tradeoff is non-preemptible late-peer latency: a peer arriving behind a 400 ms task started up to 350 ms later.

Paired real-scheduler telemetry runs used one warmup and five measured iterations per arm while changing only the reserve fraction:

In-process scheduler workload	origin/main policy	This PR	Delta
Slow→fast, 32 records	71.18 ms / 449.6 records/s	65.85 ms / 486.0 records/s	wall -7.49%, throughput +8.09%
Slow→fast, 512 records	798.02 ms / 641.6 records/s	783.90 ms / 653.1 records/s	wall -1.77%, throughput +1.80%
Request pressure, 32 records	182.74 ms	186.31 ms	+1.96%, one noisy outlier
Request pressure, 256 records	914.91 ms	913.85 ms	-0.12%, neutral

For the 32-record mixed-speed DAG, llm_wait utilization rose 76.30% → 86.11% (+9.81 pp), idle capacity fell 45.7%, starved idle fell 34.6%, and every candidate iteration beat every baseline iteration. Request wait while holding a scheduler lease was unchanged in the pressure cases. All 40 measured iterations passed task counts, caps, overlap, and zero task/request lease/waiter gates.

The same policy comparison also ran through the real async OpenAI-compatible HTTP client against a loopback inference server with 80 ms/5 ms mixed model latency. Each workload used one warmup and five measured iterations per arm with scheduler JSONL enabled:

HTTP workload	origin/main policy	This PR	Delta
32 records, endpoint capacity 8	0.930 s / 34.4 records/s	0.889 s / 36.0 records/s	wall -4.38%, throughput +4.60%
256 records, endpoint capacity 8	4.044 s / 63.3 records/s	4.026 s / 63.6 records/s	wall -0.45%, throughput +0.45%
64 records, endpoint capacity 4	1.950 s / 32.8 records/s	1.967 s / 32.5 records/s	wall +0.86%, throughput -0.86% (neutral/noisy)

Scheduler model-slot utilization improved by 9.36, 3.95, and 0.52 percentage points respectively. The paired 95% wall-time CI for the small case was [-8.04%, -0.67%], while scaled [-4.00%, +3.17%] and endpoint-queued [-0.55%, +2.29%] crossed zero and are treated as neutral. Across 30 measured runs and 7,040 measured HTTP requests, every dataset and request count was correct, endpoint capacity was never exceeded, and scheduler leases, request leases/waiters, and endpoint activity all drained to zero.

A separate public DataDesigner.create run scraped the live Prometheus OTel endpoint every 10 ms during 96 real HTTP model requests. data_designer_model_request_active peaked at 8 and returned to 0; the lifecycle counters moved from one started/zero completed job at peak to one started/one completed job at the end. The final metrics also reported 48 generated records, 96 model-request durations, and one create operation, matching the workload exactly.

Completion-state scale check (four generated columns):

Workload	origin/main retained	This PR retained	Delta
1M records, 1,000 × 1,000-row groups	217.180 MiB	0.059 MiB	-99.97%
100k records, 100k × 1-row groups	151.899 MiB	8.045 MiB	-94.70%

The deliberately pathological 1M-groups-at-one-row-each case retains 70.511 MiB because terminal completion keeps one compact marker per checkpointed group. Normal buffer sizes make that marker cost negligible.

Scheduler-only scale validation used the real row-group plan, tracker, scheduler, admission controller, fair queue, buffer lifecycle, and validating in-memory checkpoints. Its native-async generators are no-ops, so these numbers measure scheduler/checkpoint scaling—not inference or Parquet throughput:

Records	Before	This PR	Improvement
1k	0.240 s	0.042 s	5.69x
10k	22.794 s / 10,010 stale heap entries	0.462 s / 0 retained priority entries	49.34x
100k	unfinished at 369.6 s and ~99% CPU	4.935 s	>74.9x to cutoff
1M	impractical on the quadratic path	49.085 s	completed

The 1M run validated 1,001,000 tasks and 1,000 checkpoints, sustained 20,373 scheduler records/s, added 8.36 MiB peak RSS, retained no additional RSS or queue priority entries, respected the 1,024-task/three-row-group caps, and drained every task, worker, lease, queue, and buffer. A pathological 100k-record/buffer-size-1 control also completed 200k tasks and 100k checkpoints in 27.725 s with no cap violation or leak.

The next draft gate is a reusable corpus of sufficiently large multi-model linear, diamond, fan-out/fan-in, bottleneck, shared-endpoint, independent-endpoint, heterogeneous-cap, and fast/slow-skew DAGs. Its first 1,024-record/5,120-request S2 calibration exposed an invalid compressed profile: the nominal slow cap-8 endpoint reached only 15.1% full-job service utilization, so local Python/HTTP overhead dominated. That run is excluded from policy evidence. Standard cells now require at least 1,000 records, 10,000 model requests, 256 bottleneck waves, eight seconds wall time, at least 192 retained waves/five seconds after trimming the first and last 32 waves, <=5% planned/actual service dilation, and idle-host/correctness/cap/drain gates. Automatic scheduler-free controls passed at 98.64% for S/cap4, 98.58% for S/cap8/one model, 98.31% for S/cap8/two models, and 98.62% for S/cap2. The initial 45-75 ms shared-medium profile failed its dilation gate; the predeclared 1.25x adjustment to 60-90 ms passed four controls at 90.7-97.0% utilization. Policy-cell utilization remains an outcome so baseline scheduler bubbles are not filtered out. The initial scenario results above remain useful evidence, but broader scheduler conclusions will wait for paired, statistically converged corpus runs.

The first decision-grade corpus cell is now complete. S2 exercised four model endpoints in a five-node fast/medium/slow fork, synthesis fan-in, and fast post-judge. Eight ABBA/BAAB pairs processed 2,000 records, 10,000 model requests, and 500 bottleneck waves per arm (160,000 scored requests total). Baseline/candidate geomean wall time was 113.485/85.979 s: paired wall -24.238%, median -24.478%, with a within-cell six-look Bonferroni 99.17% CI of [-25.346%, -23.114%]. Mean slow-endpoint utilization rose 68.607% -> 90.538% (+21.931 pp). Every scored run passed standard evidence, output/hash, capacity, dilation, host-load, and drain gates. Same-trace full-event replays added 6.17% baseline / 7.81% candidate wall time and remain unscored. This is a meaningful S2 within-cell win; Holm correction and any corpus-wide claim wait for the remaining primary cells.

S5 is the first falsification-oriented control: a three-stage generation chain releases two early and three late judge columns onto two canonical model resources sharing one physical endpoint cap. Eight pairs processed 1,250 records, 10,000 requests, and 781 shared-judge waves per arm. Baseline/candidate geomean wall was 122.846/123.052 s: paired wall +0.168%, median -0.038%, with Bonferroni 99.17% CI [-0.445%, +0.784%] and utilization -0.011 pp, establishing practical equivalence within the +/-2% margin. All 160,000 scored requests passed. Pair 8 transparently resumed after a runner-session loss from the frozen harness snapshot with fresh symmetric warmups; both arms remained standard-class. Matched diagnostics found no hidden judge tradeoff: early-judge ready-to-worker p95 changed at most +0.19%, late judges stayed within -0.13% to +0.10%, and full capture added 2.49%/2.28% wall time. Corpus-wide Holm still awaits the remaining primary cells.

Before the remaining matrix, corpus harness v3 was hardened with exact source/product/runtime/profile identity per process segment, whole-pair restart after an interrupted partial arm, monotonic cumulative looks, compact per-arm ready-to-worker p95, adjacent same-trace compact/full observer pairs, and exact complete-nine-cell family analysis. A real interruption demonstrated partial-arm discard and rerun, but the subsequent n=12 S6b artifact still fails closed because its interrupted extension segment remains marked running; that artifact is rejected below. Twenty-five focused tests plus Ruff/compile/self-tests pass.

To avoid spending most benchmark time in synthetic sleep, the rapid family now fixes model service at 0.1x while retaining 1,024-2,500 records, 10,000-11,264 requests, and 500-1,250 bottleneck waves per arm; it measures engine/client/HTTP scheduling overhead and makes no inference-utilization claim. Independent controls established 60-90 ms as the shortest common utilization profile at caps 2/4/8 (95.5-97.5% control utilization, 1.043-1.049 dilation), so calibrated utilization runs are limited to four mechanistic sentinels. Historical S2/S5 remain valid within-cell evidence, but they use the old 120-180 ms v1 profile and are excluded from the v3 Holm family.

The first v3 S6b shared-resource rapid-overhead look was valid but inconclusive at eight pairs: paired wall -1.950%, descriptive 95% CI [-6.87%, +3.23%], and predeclared two-look 97.5% CI [-7.83%, +4.31%]. Its predeclared extension completed 12 pairs but is rejected as evidence: pair 11 exceeded the frozen host-load gate at 0.809/CPU (limit 0.80), with baseline/candidate wall 53.474/38.531 s, and the interrupted process segment remained marked running. Excluding that pair descriptively leaves -1.470% with a sequential CI crossing zero. The artifact is retained unchanged as failed evidence; it will not be pruned or resumed. A clean S6b run will be created after the novel topology cells.

The metadata-cadence optimization is now implemented in this PR. Resume metadata is written after the first durable row group and refreshed at completion; intermediate progress already comes from scanning parquet files. This is one closure-local flag (net +4 nonblank production LoC), with no new API, knob, dependency, scheduler state, or abstraction. At the default 1,000-row buffer size, the isolated 1M-record metadata loop fell from 8.705 s to 0.336 s (-96.1%). Across decision-grade multi-model cells, S2 wall/throughput/utilization changed -1.630% / +1.657% / +1.289 pp, S5 -0.054% / +0.054% / +0.032 pp, and HG -0.222% / +0.222% / +0.260 pp. S2 is beneficial/non-inferior within +2%; S5 and HG establish practical equivalence. Incremental metadata blocking fell 99.50-99.85%. The residual compatibility risk is an undocumented external poller of live metadata.json.file_paths; no in-repo or documented callback contract depends on per-row-group freshness, and OTel progress plus on_batch_complete remain live signals.

The same profiling found that opt-in flush-per-event scheduler JSONL added 29.8% mean / 44.9% median wall time in a 256-record case, so throughput arms collect scheduler events in memory and persist them after each run rather than treating JSONL as a transparent observer.

The E4 corpus overlay now has hard-crash/resume coverage for seven filesystem states and a provenance-locked paired harness. Its first identical-arm S2/buffer-64 control at the 60-90 ms profile failed closed: pair 1 differed by +0.379%, but three pair-2 retries classified both arms ineligible because workload service dilation was 1.061-1.081 (>1.05), despite exact output/metadata, host load 0.205-0.326/CPU, and an endpoint-only control at 1.044. The failed artifact is retained unchanged; the evidence threshold was not relaxed.

That 120-180 ms A/A control has now passed four pairs without a retry. Identical-arm B/A wall deltas were +0.516%, +0.048%, -1.922%, and -0.235%; every arm had exact content and normalized metadata, host load 0.225-0.395/CPU, and workload dilation 1.024-1.043. The independent control reached 98.60% utilization at 1.045 dilation. This immutable artifact is the prerequisite for the scored S2/buffer-8 metadata-cadence comparison.

A predecessor S2/buffer-8 artifact retained three valid descriptive pairs before a pair-4 host-load failure stopped the run. Valid-pair first+final wall deltas are +0.094%, +0.360%, and -0.595%: geomean wall -0.048% / throughput +0.048%, descriptive 95% CI [-1.267%, +1.186%]. Utilization deltas are +0.292/-0.442/+0.826 pp and idle-slot deltas -1.013/+1.470/-2.621 s, so both are neutral/mixed. Incremental metadata blocking fell 99.85-99.88% (~0.66-0.70 s to ~1 ms). The candidate is not promoted from the subphase improvement alone. Because this artifact froze fresh fixed-n runs only; no --resume-results support, its n=3 prefix remains descriptive failed-run evidence and will not be resumed or combined. That artifact remains excluded and is superseded by the decision-grade n=8 result below.

The default-size E4 crash/resume probe also passed all seven planned filesystem states at 128 records, buffer 32, and a non-aligned 65→130-record extension: missing/stale first metadata, an out-of-order durable hole, old-target and stale-progress extension states, a dropped first group including the all-dropped DatasetBuilder path, and callback behavior. Exact resumed request IDs, parquet paths, normalized metadata, and provenance all passed. This closes the planned functional resume surface; timing-dependent concurrent crash holes remain out of scope. The S2/S5/HG non-regression gates now also pass.

Two future-resumable v2 A/A setup artifacts are retained as failed calibration evidence. The first stopped before measured pairs because its saturation control ran at host load 1.267/CPU, exposing a missing pre-control quiescence gate. The second passed the new 0.488/CPU pre-control gate, a 98.60% utilization / 1.022 dilation control, and a symmetric first pair at -1.35%, but was intentionally interrupted during pair 2 after review found missing artifact-wide single-writer locking and overly strict predecessor-segment validation. Both harness defects are now fixed and independently re-reviewed. Neither artifact is counted or resumed; the next E4 A/A must be a fresh immutable run.

That fresh post-review v2 A/A prerequisite now passes four pairs with no retry or discarded attempt. Identical-arm B/A wall deltas were +1.638%, +0.817%, -1.758%, and -0.088% (mean +0.152%). Every arm passed exact output/normalized metadata, cap/drain, and standard-evidence gates; host load was 0.304–0.450/CPU and service dilation 1.024–1.042. Its independent control passed at 98.69% utilization, 1.043 dilation, and 0.284 load/CPU. The immutable artifact (033e6300…, identity 7a440a8e…) is the sole prerequisite for the fresh S2 n=8 E4 comparison.

The fresh E4 S2/buffer-8 n=8 comparison then completed without retry or discard. First+final produced geomean wall -1.630%, throughput +1.657%, mean central utilization +1.289 pp, and lower idle capacity in 7/8 pairs; incremental metadata time fell 99.64–99.85%. The fixed-n 95% wall interval is [-2.877%, -0.366%] and the conservative 99.17% interval is [-3.541%, +0.320%]. Exact output/normalized metadata, cap/drain, host, dilation, and standard-evidence gates all passed. This establishes beneficial/non-inferior S2 behavior within the +2% margin, not a proven ≥2% meaningful win, so E4 advances to S5 rather than rerunning an independent non-poolable S2 n=30.

The fresh E4 S5-core/buffer-8 n=8 shared-endpoint/fairness cell also completed with no retry or discard and established practical equivalence: geomean wall -0.054%, throughput +0.054%, mean central utilization +0.032 pp, and mean idle capacity slightly improved. The 95% interval is [-0.290%, +0.183%] and the conservative 99.17% interval [-0.417%, +0.310%]. Incremental metadata time fell 99.55–99.79%; exact output/metadata, cap/drain, host, dilation, and standard-evidence gates all passed. The final fairness audit found pooled early ready-to-worker p95 -1.20% and pooled late +1.40%, split evenly between better/worse observations; physical late-model queue wait was +0.014% geomean with every pair within [-0.395%, +0.435%]. One late column shifted upward while two improved, indicating redistribution rather than model-level starvation. S5 therefore advances E4 to the hourglass cell.

The fresh E4 HG-inverse/buffer-8 n=8 five-wide → slow-join → five-wide cell completed with no retry or discard and established practical equivalence: geomean wall -0.222%, throughput +0.222%, mean central utilization +0.260 pp, and mean idle capacity improved 0.503 slot-seconds. The 95% interval is [-0.803%, +0.363%] and the conservative 99.17% interval [-1.114%, +0.679%]. Metadata blocking fell 99.50–99.71%; exact output/metadata, cap/drain, host, dilation, and standard-evidence gates all passed. This clears the final E4 topology gate.

A separate request-prefetch hypothesis remains scratch-only. Independent review rejected the first broad endpoint-gap analyzer before product work. The corrected fail-closed diagnostic now gates only same-clock request_lease_released → request_lease_acquired gaps with in-flight below limit, zero request waiters, an exhausted canonical scheduler resource, and compatible queued demand; endpoint idle is reported separately and cannot trigger H9. Offline analysis of old-v1 full streams found conservative qualifying-hole lower bounds of 6.989% / 20.309 ms p95 for S2 reserve 0 and 1.973% / 10.427 ms for S5 reserve 0, but these remain directional because the analyzer postdates the frozen runs. No request-cap product change is justified until a fresh current-harness baseline confirms the signal. Fixed W+1 headroom would then require at least 50% hole reduction, a ≥2% target wall-time win, and corpus/rate/cooldown non-regression.

🧪 Testing

• Latest-head GitHub CI in progress (12 passed, 36 pending, 2 skipped; previous head passed 58/58 with 2 skipped)
• Unit tests added/updated
• E2E mock HTTP inference endpoint benchmark exercised

Validated the admission change with:

uv run pytest -q \
  packages/data-designer-engine/tests/engine/dataset_builders/scheduling/test_task_policies.py \
  packages/data-designer-engine/tests/engine/dataset_builders/scheduling/test_task_admission.py

Result: 24 passed.

The scheduling, scheduler, async-builder, and row-group-buffer suites pass 235 tests; the full DatasetBuilder file passes 85 tests; and the full engine suite passes 2,214 tests. Coverage includes bounded queue state, exact first+final metadata cadence, and a concurrent fan-out regression where one branch drops the row while a slow sibling remains in flight. make check-all and git diff --check pass.

The deterministic benchmark and five-topology paired sweep also enforce completed task counts, capacity limits, queue drain, and zero final leases.

✅ Checklist

• Follows commit message conventions
• Commits are signed off (DCO)
• Architecture docs updated (not applicable; no public API or architecture change)