Skip to content

feat(storage,entity): path identity + speculation path→build mapping store#331

Open
behinddwalls wants to merge 1 commit into
preetam/ext/speculation-prioritizerfrom
preetam/int/storage-build-lookup
Open

feat(storage,entity): path identity + speculation path→build mapping store#331
behinddwalls wants to merge 1 commit into
preetam/ext/speculation-prioritizerfrom
preetam/int/storage-build-lookup

Conversation

@behinddwalls

@behinddwalls behinddwalls commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

Summary

Why?

Integrating speculation needs the controllers to answer "which build belongs to this path" in both directions, without ever looking a row up by non-key attribute — the storage contract stays get/put-by-key so any KV backend can satisfy it. The build system mints its own build identifiers, and those stay the build store's primary key: the runner's ID is the natural name for the build row. What's missing is a durable link between a tree path and its build that exists independent of any in-flight message, hung on the path identity (SpeculationPathInfo.ID) introduced at the bottom of this stack.

What?

entity.SpeculationPathBuild is the new mapping entity ({PathID, BuildID, BatchID, Version, CreatedAt}, named after its speculation_path_build table) with a SpeculationPathBuildStore (Create/Get keyed by PathID): the forward path→build lookup, written only by the build controller, at most one build per path with ErrAlreadyExists making the existing row the truth on races. BatchID makes the row self-describing without parsing PathID's format; Version follows the repo's optimistic-locking convention (write-once today, reserved for future conditional re-pointing flows). entity.Build keeps the runner-minted ID as its primary key and gains SpeculationPathID as a plain column — the reverse build→path lookup; the previously embedded SpeculationPath value is dropped as a redundant denormalized copy of what the tree already stores. SpeculationPath.Equal (order-sensitive Base + Head) provides structural path identity for the few controller spots where only structure can identify a path (deduplicating enumerator output, carrying entries over across re-enumeration). entity.QueueID with ToBytes/QueueIDFromBytes mirrors the BatchID payload pattern for queue-scoped stages.

MySQL gains the speculation_path_build table and the build table swaps speculation_path/runner_id for speculation_path_id; schema files are picked up automatically (the schema dir is globbed by both Bazel and the testutil ApplySchema helper).

Test Plan

bazel test //submitqueue/entity/... //submitqueue/extension/storage/... //submitqueue/orchestrator/... and the storage integration suite exercises Create/Get round-trips and the duplicate-create race.

Stack

  1. feat(entity): give speculation paths an assigned identity #337
  2. feat(speculation): add enumerator + dependency-limit extensions #315
  3. feat(speculation): add path scorer extension #316
  4. feat(speculation): add selector + selection-limit extensions #317
  5. feat(speculation): add prioritizer + prioritization-limit extensions #320
  6. @ feat(storage,entity): path identity + speculation path→build mapping store #331
  7. feat(speculation): add parity default impls for the speculation seams #332
  8. feat(speculation): add probability path scorer #333

## Summary

### Why?

Integrating speculation needs the controllers to answer "which build belongs to this path" in both directions, without ever looking a row up by non-key attribute — the storage contract stays get/put-by-key so any KV backend can satisfy it. The build system mints its own build identifiers, and those stay the build store's primary key: the runner's ID is the natural name for the build row. What's missing is a durable link between a tree path and its build that exists independent of any in-flight message, hung on the path identity (SpeculationPathInfo.ID) introduced at the bottom of this stack.

### What?

`entity.SpeculationPathBuild` is the new mapping entity ({PathID, BuildID, BatchID, Version, CreatedAt}, named after its `speculation_path_build` table) with a `SpeculationPathBuildStore` (Create/Get keyed by PathID): the forward path→build lookup, written only by the build controller, at most one build per path with ErrAlreadyExists making the existing row the truth on races. BatchID makes the row self-describing without parsing PathID's format; Version follows the repo's optimistic-locking convention (write-once today, reserved for future conditional re-pointing flows). `entity.Build` keeps the runner-minted `ID` as its primary key and gains `SpeculationPathID` as a plain column — the reverse build→path lookup; the previously embedded `SpeculationPath` value is dropped as a redundant denormalized copy of what the tree already stores. `SpeculationPath.Equal` (order-sensitive Base + Head) provides structural path identity for the few controller spots where only structure can identify a path (deduplicating enumerator output, carrying entries over across re-enumeration). `entity.QueueID` with ToBytes/QueueIDFromBytes mirrors the BatchID payload pattern for queue-scoped stages.

MySQL gains the `speculation_path_build` table and the build table swaps `speculation_path`/`runner_id` for `speculation_path_id`; schema files are picked up automatically (the schema dir is globbed by both Bazel and the testutil ApplySchema helper).

## Test Plan

✅ `bazel test //submitqueue/entity/... //submitqueue/extension/storage/... //submitqueue/orchestrator/...` and the storage integration suite exercises Create/Get round-trips and the duplicate-create race.
@behinddwalls behinddwalls force-pushed the preetam/ext/speculation-prioritizer branch from 0426ccb to 55d0b08 Compare July 11, 2026 05:03
@behinddwalls behinddwalls force-pushed the preetam/int/storage-build-lookup branch from bbc4956 to e0c6857 Compare July 11, 2026 05:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant