Correct dogs3 worst-case optimal tools - #821
Merged
frankmcsherry merged 5 commits intoAug 7, 2026
Merged
Conversation
The three shared `lookup_map`, which matched arranged updates at times less-or-equal in the *partial* order, and emitted at the update's own time. That is a correct lookup but not a correct delta join. Pairs at incomparable times never met, and a chained stage compared against the previous stage's time rather than the original. They now sit on half_join, which compares in a caller-supplied total order and carries the original time beside a payload time advanced by lattice join. `propose` and `validate` are wrappers over the safe entry point. `count` uses the unsafe one, to sum a scalar and annotate the prefix without joining times. `lookup_map` and `propose_distinct` are deleted; the latter proposed presence without maintaining an indicator to test it against. Differences are pinned to `isize` and the `Multiply` bounds dropped. BiGJoin (Ammar et al., VLDB 2018) is stated over sets: its extension indices are set-valued and its intersection step is an existence test. Above arity two an extension index is a projection, whose multiplicity counts completions rather than the record's own annotation, so carrying annotations would need indicator projections and a contributes-once rule. That is InsideOut, a different algorithm with different indices. The set expectation is documented rather than enforced, so callers whose data are already distinct pay nothing. `count` routes a prefix with a zero count rather than dropping it, because the sum spans an interval and zero there does not mean empty at every time within it. It sums absolute values, since magnitude carries the extension count and polarity only churn. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`wcoj_triangle.rs` cannot see the payload time do any work. For a totally ordered time the total order used by `comparison` coincides with the lattice order, so every match satisfies `lub(t2, payload) == payload` and all matches collapse onto one output time. This runs the same triangle query over `Product<usize, usize>` with unordered input, against an exhaustive-search oracle, at every input time and every pairwise join of them. It adds a constructed case where an extension is born at (0,0) and retracted at (0,2), both preceding a delta at (1,0) in the total order while (0,2) stays incomparable to (1,0) in the lattice. The extension is therefore live at (1,0) while its updates sum to zero, and a `count` that gates on a non-zero sum drops the prefix and loses the triangle. Randomized generation does not reach that configuration: `count` is per key, so every extension of a key must be born and die inside the window, which twelve seeds never produced. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`count`, `propose` and `validate` take `strict: bool` in place of a comparison closure. Only two comparisons were ever valid, and which one follows from the query: a relation looking up in a later one is strict, and cannot see updates concurrent with the delta it is responding to; looking up in an earlier one is non-strict, and can. A relation never looks up in itself. The closure existed for monomorphization rather than for generality, so the operators branch on `strict` once while building the dataflow and instantiate `half_join` twice, leaving nothing to test at each timestamp. `half_join` keeps its closure: it is the externally relied-on surface, and it knows nothing about which relations are being joined. That subsumes `AltNeu`, whose `neu` flag was a stand-in for the same choice. A delta enters at `alt`, so an arrangement left at `alt` is visible at the delta's own time and one delayed to `neu` is not; with strictness stated directly the tag carries no information. The four tagged indexes in the worst-case-optimal example and its tests collapse to one per orientation, and the nested scope goes with them. `delta_query.rs` compared two delta implementations, agreeing only after concatenating all three fragments. It now compares the concatenated fragments against a conventional three-way join, which shares no machinery with them and so is an independent answer rather than a second opinion from the same method. That also removes the last use of `Arranged::enter_at` in the repository. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both modules are gone from the worst-case-optimal path, which now states strictness at each lookup rather than encoding it in the timestamp. `calculus` cannot outlive `altneu`. `Differentiate` is defined by emitting a change at `alt` and retracting it at `neu`, and `Integrate` recovers the collection by filtering on `!t.neu`, so the tag is its representation rather than an implementation detail. This removes two public modules from the crate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The dogs3 worst-case optimal operators are ancient, and have various bugs to fix. At least:
import()traces, which if used in iterative contexts can fail to terminate (switching from timely's holistic progress to in-band).half_join's total order use, to ensure each tuple of updates match exactly once.I expect to retarget them atop
half_join, which may result in some API churn. The set should evolve further towards a. chunks, and b. tactics, but doing this clean-up first to minimize the surface that needs to churn.Updated: the
count/propose/validateoperators are now built overhalf_join, and should be more correct than before, with no known incorrectness (though .. not certain correctness either; tests are limited). Their shape has changed to matchhalf_joinwrt inputs, and may change again if its input shapes change (e.g. to be chunk-aligned). TheAltNeuwrapper type has vanished, and with it thecalculus.rsstuff (differentiation and integration).Most of the LOC churn is removal. About ~400 lines of tests added, leaving a general net deletion of system code.