feat(ci): run independent jobs in parallel#380
Merged
Conversation
Jobs previously ran one at a time in topological order. They now execute concurrently: each job starts as soon as every job in its `needs` has finished, so independent jobs (and independent branches of the graph) run in parallel, bounded by a per-run semaphore (maxParallelJobsPerRun). To make this safe, each job instance now owns a private log buffer and line counter (jobAccumulator) instead of sharing mutable maps; the shared run state (results, per-job status, the aggregate log, the overall conclusion) is merged under a mutex only around the quick bookkeeping, never during step execution. `needs` skipping and job-level `if:` are still honored, and matrix instances still run sequentially within their job. Verified with the race detector (no data races) and live: three independent `sleep 2` jobs plus a `gather` job that needs all three complete in ~4s wall-clock instead of the ~6-8s a sequential run would take, with `gather` correctly running only after a/b/c finish.
🍊 Orange Codens レビュー指摘事項はありません。✨ head: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Jobs previously ran one at a time in topological order. They now execute concurrently: each job starts as soon as every job in its
needshas finished, so independent jobs run in parallel, bounded by a per-run semaphore.Safety
Each job instance now owns a private log buffer + line counter (
jobAccumulator) instead of sharing mutable maps; shared run state (results, per-job status, aggregate log, conclusion) is merged under a mutex only around quick bookkeeping — never during step execution.needsskipping and job-levelif:are still honored; matrix instances still run sequentially within their job.Verified
go test -race— no data races; new test proves two independent jobs run concurrently (each blocks on a barrier until the other starts, failing if sequential).sleep 2jobs + agatherjob needing all three complete in ~4s wall-clock (vs ~6-8s sequential),gatherruns only after a/b/c finish.