feat: event log and history server (Spark History Server equivalent)#1925
Open
andygrove wants to merge 13 commits into
Open
feat: event log and history server (Spark History Server equivalent)#1925andygrove wants to merge 13 commits into
andygrove wants to merge 13 commits into
Conversation
Move the scheduler REST API's JobResponse/TaskSummary/TaskStatus/Percentiles/ QueryStageSummary/QueryStagesResponse types onto the ballista-history crate's shared DTOs, and extract the graph-to-DTO construction logic that previously lived inline in the handlers into pub(crate) builder functions in a new api::dto_build module. Handlers become thin wrappers over these builders, which a later event-log writer will also call. Behavior is unchanged; the existing handler unit tests move to dto_build.rs alongside the moved helper functions they exercise, plus one new plan-string assertion test.
Emit HistoryEvents from the QueryStageScheduler event loop:
JobSubmitted -> JobStart, TaskUpdating -> TaskEnd (one per finished
task), and JobFinished/JobRunningFailed -> JobEnd. Event builders live
in a new scheduler_server::event_log module and reuse the same
dto_build DTOs the REST API serves, so a job's JobEnd event matches
its live GET /api/job/{id} response. The EventLogWriter is constructed
from SchedulerConfig::event_log_dir and threaded into
QueryStageScheduler; emission is best-effort and a no-op when
event_log_dir is unset.
… missing jobs
- HistoryStore::load now skips unreadable/corrupt .eventlog files with a
warning instead of failing the whole load, so one bad log can't hide
every other completed job.
- Job endpoints (/api/job/{id}, /stages, /config, /dot) now return 404
for unknown job ids instead of 200 with a null body, matching the
live scheduler's behavior.
- Add /api/state with a static payload matching the shape the TUI
deserializes at startup.
- Strengthen router tests: assert stage-response body contents, cover
the corrupt-eventlog skip path, and cover the 404 behavior.
…e scheduler Add an end-to-end DTO parity test that builds live JobResponse/ QueryStagesResponse DTOs, writes the same JobEnd event through the real EventLogWriter, reloads it via HistoryStore::load, and asserts the serialized JSON matches exactly -- proving the history server would serve the same data a running scheduler produced.
EventLogWriter::append enqueues via non-blocking try_send, which drops the event when the channel is saturated. Using it for the terminal JobEnd event meant a completed job's log could end up with JobStart and TaskEnds but no JobEnd, making read_completed_job return None and the job silently disappear from the history server. Add append_final, which awaits channel capacity instead of dropping, and finish_job, which flushes and then closes the per-job file handle (also fixing fd accumulation). Wire both into the JobFinished and JobRunningFailed on_receive arms in place of append + flush_job.
…-api All ballista_history usage in the scheduler is behind the rest-api feature (DTO builders, event-log wiring, history module). Gating the dependency on rest-api keeps it out of the graph for consumers that build the scheduler with default-features = false (e.g. pyballista), and out of non-rest-api builds.
serde_json is only used in production by the rest-api-gated history module (all other uses are in tests, covered by the dev-dependency). Gating it keeps it out of the dependency graph for default-features = false consumers such as pyballista, so python/Cargo.lock stays in sync.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #1923.
Rationale for this change
Ballista has a live TUI that shows jobs, stages, tasks, executors, and metrics by
reading the scheduler's REST API — but that state is ephemeral. Completed jobs are
cleaned up after
finished_job_state_clean_up_interval_seconds, and everything isgone when the scheduler restarts. There is no way to inspect a job after the fact,
the way Spark's History Server lets you.
This PR adds a Spark-History-Server equivalent: the scheduler can record a durable
per-job event log during execution, and a standalone history server replays those
logs and serves the same
/api/*responses the live scheduler does — so theexisting TUI browses completed jobs unchanged, with no scheduler running.
What changes are included in this PR?
New
ballista-historycrate (a leaf crate; depends only onserde,serde_json,tokio):HistoryEvent):JobStart,StageStart/StageEnd,TaskEnd(the incremental timeline), and a terminalJobEndthat embeds the fully-built REST DTOs.JobResponse,QueryStagesResponse, …), shared sothe live scheduler and the history server serialize byte-identical JSON.
EventLogWriter(appends never block the scheduler hot path)and a reader that folds a completed log into the DTO bundle the history server
serves.
Scheduler:
reusable
dto_buildbuilders (behavior-preserving — existing API output andtests are unchanged).
event_log_dirconfig option (off by default). When set, anEventLogWriteris tee'd off the internalQueryStageSchedulerEventbus inQueryStageScheduler::on_receive, mappingJobSubmitted→JobStart,TaskUpdating→TaskEnd, andJobFinished/JobRunningFailed→JobEnd. Whendisabled there is no channel, no task, no file, and no per-event work beyond one
Optioncheck.JobEndis enqueued with a blocking send and flushed before theloop proceeds, so a completed job's record cannot be dropped under load; its file
handle is then closed.
New
ballista-history-serverbinary:ballista-history-server --event-log-dir <dir> --bind-host <h> --bind-port <p>loads completed logs and serves an axum router over the same
/api/*paths fromthe stored DTOs. Corrupt/partial logs are skipped rather than failing startup;
missing jobs return 404 like the live scheduler.
Scope for this first cut: local-filesystem storage and completed jobs only.
Incremental
TaskEndtimeline events are captured but not yet surfaced in a UI.Verification: the DTO builders are shared by both the live handler and the writer,
and an end-to-end test asserts the history server serves byte-identical JSON to
the live scheduler for the same job.
Are there any user-facing changes?
Yes, all additive and opt-in:
--event-log-dir <dir>(default: disabled).ballista-history-serverbinary.--host/--port) to browsecompleted jobs with no live scheduler.
No breaking changes to public APIs; live REST responses are unchanged.