MySQL Parallel Snapshot#37642
Draft
peterdukelarsen wants to merge 3 commits into
Draft
Conversation
Partition each table's primary-key range across timely workers so they read disjoint PK ranges of the initial snapshot concurrently, reducing initial-load time for large tables with a single-column integer primary key. A snapshot leader establishes the consistent point and broadcasts SnapshotInfo to all workers over a timely feedback loop; each worker then reads its range under a CONSISTENT SNAPSHOT transaction. Tables without a suitable PK fall back to single-worker-per-table mode. Also adds a MySqlInitialLoadMultiWorker feature benchmark. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Generalize PK-range snapshot partitioning beyond single-column integer primary keys to also cover non-integer (char/varchar/text) and composite PKs, so large tables keyed by strings (e.g. ULIDs) or multi-column keys also load their initial snapshot in parallel across workers instead of falling back to a single worker. Single-column integer PKs keep the cheap MIN/MAX split. Other supported PKs sample partition boundaries by keyset pagination over the primary-key index (one forward pass), reading each boundary key back already rendered as a SQL literal (QUOTE() for text, CAST(.. AS CHAR) for numeric) so range predicates compare under each column's own collation, matching the ORDER BY used to pick them. Composite keys use row-value comparisons. Every boundary is found with a strict `pk > prev`, so boundaries strictly increase and the resulting half-open ranges are disjoint and cover the entire key domain: each row is read by exactly one worker regardless of data distribution. Boundary computation runs before LOCK TABLES so the O(rows) sampling scan for non-integer PKs does not extend the table-lock window; correctness does not depend on boundaries matching the exact snapshot point. Also fix a snapshot duplication bug: when a partitioned table has fewer partitions than the cluster has workers, the worker responsible for the table could fall through to the `responsible_for` path and read the whole table in addition to the range-owning workers, duplicating every row. A read plan now distinguishes range-owning workers, the single responsible worker of an unpartitioned table, and a responsible worker that owns no range (which reads nothing but still emits the table's rewind request). Adds char and composite PK correctness cases to mysql-cdc.td and a test/mysql-pk-benchmark composition that demonstrates the speedup under a bandwidth-limited MySQL link (the regime where parallel range reads help). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Description
What does this PR actually do? Focus on the approach and any non-obvious
decisions. The diff shows the code --- use this space to explain what the
diff can't tell a reviewer.
Verification
How do you know this change is correct? Describe new or existing automated
tests, or manual steps you took.