Skip to content

Compaction conversion - #711

Draft
jshook wants to merge 4 commits into
integration-robustnessfrom
compaction-conversion
Draft

Compaction conversion#711
jshook wants to merge 4 commits into
integration-robustnessfrom
compaction-conversion

Conversation

@jshook

@jshook jshook commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

This PR is for illustrative purposes only. It should not be merged, so I'm setting it as Draft. We'll delete it when we're done with the review of the approach.

Purpose

Convert the general execution paths onto the embedding seams, and document the compactor conversion as an illustration of how the upstream integration robustness APIs might be used.

Stacked PR. This builds on the embedding-seams PR (integration-robustness) and targets that branch, not main. Please review/merge it after the base PR.

Why

The base PR added the embedding seams with no consumers. This PR shows how jvector's operations adopt them. Two categories:

  • The general build/quantize paths are converted as real code.
  • The compactor's own conversion is captured as a design note, not code: in the current lineage the compactor's seam-wiring is inseparable from the compaction-algorithm work (a single ~1,100-line diff), so this PR documents the target the clean compaction work should hit rather than carrying that algorithm here.

Real code

  • GraphIndexBuilderParallelExecutor. Build and finalize route through ParallelExecutor (a host pool, or caller-runs so no work escapes to the common pool). Existing ForkJoinPool constructors are preserved — they now wrap ParallelExecutor.forkJoin(pool) — so current callers are unaffected.
  • Quantization (PQ/NVQ/BQ) → ParallelExecutor. encode/refine/compute accept a ParallelExecutor; the existing ForkJoinPool entry points remain as delegating overloads. Includes QuantizationCallerRunsTest, which verifies caller-runs output matches the pool-backed path.
  • EmbeddedExecutionContext. A single carrier for the execution resources an embedder supplies — a compute ParallelExecutor plus merge/io executors — so the pool is passed once and "no work escapes to the common pool" lives in one place. of(pool) bounds everything to a host pool; callerRuns() runs inline on the calling thread (the memtable-flush case). Build and PQ/NVQ train/refine/encode are wired through it.

Documented — docs/compaction-seam-conversion.md

before/after for each compactor calling convention, using the real integration-lineage signatures:

  • execution — an Executor constructor + taskWindowSize bound, defaulting to the shared pool only when none is supplied;
  • progress + throttlesetProgressLimiter(...), with onProgress (up) / acquire (down) at phase boundaries doubling as cancellation checkpoints;
  • outputcompact(CompactionDestination) with the Target open/commit/close lifecycle and host-owned footer via SeekableSink.over(...);
  • PQ retrain through the compute executor (closing the historical retrain leak);
  • parallel writer via OnDiskParallelGraphIndexWriter.Builder.withExecutor(io);
  • compactor-local memory safety — drain-on-unwind and truncate-reused-outputs — that lands with the conversion (the reader-side guard is already in the base PR).

Note on EmbeddedExecutionContext scope

Its compactor/writer/retrain factory methods (newCompactor, newParallelWriter, retrainPQ) are intentionally omitted here — they call converted compactor APIs that don't exist yet — and are shown in the design note instead. The merge/io executor roles are still carried so the note can reference mergeExecutor() / ioExecutor() directly; those methods return as thin wiring once the compactor is converted.

Testing

jvector-base compiles at --release 11; QuantizationCallerRunsTest added and compiles.

Stats: 9 files, +934/−168.

Commits

  • graph: convert GraphIndexBuilder to the ParallelExecutor seam
  • quantization: accept ParallelExecutor for PQ/NVQ/BQ encode
  • graph: EmbeddedExecutionContext as the single execution carrier
  • docs: illustrate the compactor's calling-convention conversion

jshook added 4 commits August 12, 2026 17:43
Route build and finalize through ParallelExecutor instead of a directly
held ForkJoinPool, so an embedding host can supply its own pool or run
caller-runs (no work escaping to ForkJoinPool.commonPool()).

Existing ForkJoinPool constructors are preserved; they now wrap
ParallelExecutor.forkJoin(pool), so current callers are unaffected. The
internal parallel-for sites move from executor.submit(...) to the
ParallelExecutor.forEachInt / forEach entry points.
Let PQ, NVQ, and BQ encode/refine on a caller-supplied ParallelExecutor
so quantization participates in the host's execution budget instead of
the common pool.

Additive: the existing ForkJoinPool entry points are kept as delegating
overloads, so no current call path changes.
Carry the execution resources an embedder passes to jvector in one place
-- a compute ParallelExecutor plus merge/io executors -- so the pool is
supplied once and "no work escapes to the common pool" holds in a single
spot. of(pool) bounds everything to a host pool; callerRuns() runs inline
on the calling thread (the memtable-flush case).

Wired here to the seams that exist on this branch: graph build
(newBuilder) and PQ/NVQ train/refine/encode route through the compute
executor. The compaction and parallel-writer wiring depends on the
compactor's calling-convention conversion, which is illustrated in prose
in doc/compaction-seam-conversion.md; the merge/io roles are carried so
the note can reference them directly.
Add docs/compaction-seam-conversion.md: how OnDiskGraphIndexCompactor's
calling conventions map onto the integration-robustness seams -- execution
(Executor / EmbeddedExecutionContext), progress + throttle (ProgressLimiter),
output (CompactionDestination / SeekableSink), PQ retrain and the parallel
writer via the carried executors, plus the compactor-local memory-safety
guards (drain-on-unwind, truncate-reused-outputs) that land with it.

In the current lineage the compactor's seam-wiring is inseparable from the
compaction-algorithm work, so it is described here rather than carried as
code; this note is the target the clean compaction work should hit.
@github-actions

Copy link
Copy Markdown
Contributor

Before you submit for review:

  • Does your PR follow guidelines from CONTRIBUTIONS.md?
  • Did you summarize what this PR does clearly and concisely?
  • Did you include performance data for changes which may be performance impacting?
  • Did you include useful docs for any user-facing changes or features?
  • Did you include useful javadocs for developer oriented changes, explaining new concepts or key changes?
  • Did you rebase your branch onto the latest main for regression testing and PR submission?
  • Did you trigger regression testing via Run Bench Main and review results?
  • Did you adhere to the code formatting guidelines (TBD)
  • Did you group your changes for easy review, providing meaningful descriptions for each commit?
  • Did you ensure that all files contain the correct copyright header?
  • Did you add documentation for this feature to the release notes directory?

If you did not complete any of these, then please explain below.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant