Update columnar, timely, and differential dependencies#36804
Open
antiguru wants to merge 6 commits into
Open
Conversation
Bump the dataflow dependency family to their latest releases:
* columnar 0.12.1 -> 0.13.0
* timely 0.29.0 -> 0.30.0
* differential-dataflow 0.23.0 -> 0.24.0
* differential-dogs3 0.23.0 -> 0.24.0
These releases carry several breaking changes that ripple through the
batcher/arrange machinery and the timely communication layer:
* columnar 0.13 reworks `AsBytes` to require `SLICE_COUNT` and
`get_byte_slice`, and removes the `chain` helper. Updated the manual
`AsBytes` impls for `Overflows`, `Timestamps`, and `Rows`.
* differential 0.24 removes the `Batcher::Input`/`push_container` API and
the `merge_batcher::container` compatibility layer (`InternalMerge`,
`InternalMerger`). The `Batcher` now consumes already-chunked input via
`PushInto<Output>`, `seal` returns `(chain, description)`, and chunking
moves into the operator via a separate `ContainerBuilder`/chunker.
`MergeBatcher` collapses to a single `Merger` type parameter.
- Reimplemented `ColInternalMerger` as a chunk-list `Merger` over
`ColumnationStack` chunks.
- Migrated `ColumnMergeBatcher` to the new `Batcher` trait.
- Introduced an `ArrangeChunker<C>` trait that maps a (batcher, input
container) pair to its chunker, so `mz_arrange`/`consolidate_pact`
call sites keep working without threading an extra chunker type
parameter everywhere.
- Updated `consolidate_pact`, `consolidate_and_pack`, the
`MergeBatcherWrapper`, the upsert v2 source stash, and the
interchange test to drive the chunker explicitly.
- `TraceBox.trace` is now accessed via `trace()`, and
`ShutdownButton::press_on_drop` is replaced by a local press-on-drop
guard.
* timely 0.30 adds a spill-policy parameter to the process allocators and
bundles refill/spill/log hooks into `Hooks` for
`initialize_networking_from_sockets`; `BytesRefill` now yields
`Send` buffers, so `LgallocHandle` gains an `unsafe impl Send`.
https://claude.ai/code/session_01CrNqefrHBNbfJetgjRHsmH
antiguru
commented
May 30, 2026
antiguru
commented
May 30, 2026
antiguru
commented
May 30, 2026
* Fix the broken intra-doc link to `ColumnChunker` in the `columnar::merge_batcher` module docs (it now lives in the operator-level chunker, not this module), and reword the doc to match the differential 0.24 contract. * Document on `ColumnMergeBatcher::push_into` / `seal` where sorting, consolidation, and staging-buffer draining now happen (the operator's chunker), answering the review questions. * Leave the timely 0.30 communication spill policy unset with a `TODO(CLU-99)` to wire up the zero-copy pager. https://claude.ai/code/session_01CrNqefrHBNbfJetgjRHsmH
Thread the chunker through the arrange/consolidate wrappers as an explicit `Chu` type parameter (mirroring differential's `arrange_core`) instead of recovering it from the batcher via the `ArrangeChunker` associated-type trait. * Remove the `ArrangeChunker` trait and its impls from `columnar`. * Add a `Chu` parameter to `mz_arrange`, `mz_arrange_core`, `consolidate_pact`, and `consolidate_and_pack`, bounded exactly as `arrange_core` requires (`ContainerBuilder<Container = Ba::Output> + for<'a> PushInto<&'a mut C>`). * `consolidate_named`/`consolidate_named_if` stay single-parameter: they only ever consolidate `Vec` input, so they pass `ColumnationChunker` explicitly. * Update call sites to name the chunker: `ColumnationChunker<_>` for `Vec` input, `batcher::Chunker<_>` for `Column` input, and `batcher::ColumnChunker<_>` for the paged batcher. https://claude.ai/code/session_01CrNqefrHBNbfJetgjRHsmH
* Remove the now-redundant `timely::container::ContainerBuilder` import in `consolidate_pact`; the explicit `Chu: ContainerBuilder` bound brings the trait methods into scope. * Wrap the `mz_arrange` turbofish call sites that exceeded the 100-column limit onto multiple lines (rustfmt does not break method-call turbofish itself). https://claude.ai/code/session_01CrNqefrHBNbfJetgjRHsmH
Member
Author
|
Marking as ready for review, but needs another round of reviews, and nightly tests. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Bump the dataflow dependency family to their latest releases:
columnartimelydifferential-dataflowdifferential-dogs3These releases carry breaking changes that ripple through the batcher/arrange machinery and the timely communication layer.
What changed
columnar 0.13 —
AsBytesnow requiresSLICE_COUNTandget_byte_slice, and thechainhelper was removed. Updated the manualAsBytesimpls forOverflows,Timestamps, andRows(theRowsimpl now relies on the defaultas_bytesbuilt fromget_byte_slice).differential 0.24 — removes the
Batcher::Input/push_containerAPI and themerge_batcher::containercompatibility layer (InternalMerge,InternalMerger). The batcher now consumes already-chunked input viaPushInto<Output>,sealreturns(chain, description), and chunking moves into the operator via a separateContainerBuilder/chunker.MergeBatchercollapses to a singleMergertype parameter.ColInternalMergeras a chunk-listMergeroverColumnationStackchunks (the oldInternalMerge-based impl is gone).ColumnMergeBatcherto the newBatchertrait.ArrangeChunker<C>trait (a subtrait ofBatcher) that maps a (batcher, input-container) pair to its chunker. This letsmz_arrange/consolidate_pactrecover the chunker from the batcher type, so the ~30 arrange call sites keep working without threading an extra chunker type parameter everywhere.consolidate_pact,consolidate_and_pack,MergeBatcherWrapper(temporal bucketing), the upsert-v2 source stash, and an interchange test to drive the chunker explicitly.TraceBox.traceis now read viatrace();ShutdownButton::press_on_dropwas removed, replaced by a small local press-on-drop guard.timely 0.30 — adds a spill-policy parameter to the process allocators and bundles refill/spill/log hooks into
Hooksforinitialize_networking_from_sockets.BytesRefillnow yieldsSendbuffers, soLgallocHandlegains anunsafe impl Send(it exclusively owns its allocation).Testing
cargo check --workspace --all-targetsis clean (libs, tests, benches, examples).mz-fivetran-destinationbuild script, which fetches a CA bundle over the network (TLS failure due to container clock skew) — not affected by this change.🤖 Draft PR — opened for review of the dependency migration. CI will exercise the full test suite.
https://claude.ai/code/session_01CrNqefrHBNbfJetgjRHsmH
Generated by Claude Code