Skip to content

Fix DecoupledLookback cross-block coherence (completes #91)#98

Open
shreyas-omkar wants to merge 4 commits into
JuliaGPU:mainfrom
shreyas-omkar:sh/dl-coherence-fix
Open

Fix DecoupledLookback cross-block coherence (completes #91)#98
shreyas-omkar wants to merge 4 commits into
JuliaGPU:mainfrom
shreyas-omkar:sh/dl-coherence-fix

Conversation

@shreyas-omkar

@shreyas-omkar shreyas-omkar commented Jul 15, 2026

Copy link
Copy Markdown
Member

Completes #91 — a working version of the DecoupledLookback() scan.

Background

#91 corrected DecoupledLookback()'s exclusive multi-block carries by publishing each block's inclusive aggregate in a dedicated array. That fixed the algorithm, but the cross-block publish/consume protocol was still not memory-coherent, so it failed ~40% of the time on smaller GPUs (e.g. A100 MIG 1g.5gb, sm_80) for both inclusive and exclusive scans.

Two problems remained:

  1. Coherence — the aggregate was read with an ordinary (L1-cacheable) load while its companion flag was read atomically, so a consumer could observe a fresh flag == ACC_FLAG_A yet read a stale aggregate.
  2. Ordering — the producer's aggregate store was not ordered before its flag store (nor the consumer's flag load before its aggregate load) at device scope.

Fix

  • Coherence: the aggregate is now read/written with a relaxed (monotonic), L1-bypassing atomic, reinterpreted through a same-width unsigned integer so float element types are supported (compile-time fallback to a plain access for other widths).
  • Ordering: a device-scope memory fence now separates the aggregate store from the flag store (producer) and the flag load from the aggregate load (consumer).

There is no vendor-agnostic device-scope fence in the ecosystem today (KA only has work-group barrier/@synchronize; UnsafeAtomics.fence(acq_rel) does not lower on NVPTX; CUDA.threadfence is CUDA-only). So _decoupled_fence() composes the already-available primitives, kept self-contained here:

  • generic definition = UnsafeAtomics.fence(acq_rel) (lowers on OpenCL/SPIR-V, Metal, oneAPI, AMDGPU),
  • overridden to the native CUDA.threadfence() (membar.gl) via a small CUDA package extension, since NVPTX does not select scoped atomic fences.

No new hard dependency: CUDA is a weakdep behind an extension.

Also: aggregates is now always its own contiguous array (never a temp view) so pointer(aggregates, i) is valid inside the kernel; the non-uniform exclusive-scan test is re-extended to cover DecoupledLookback().

Validation

  • CUDA (RTX 5080): 4400/4400 passed (exclusive + inclusive, random sizes/block-sizes, incl. Float32/Float64) — 0 failures.

Performance / when to use it

DecoupledLookback() is now correct, but it is not the fastest scan on CUDA — launches are cheap there, so the standalone ScanPrefixes() (which stays the default on every backend) wins, and the gap grows with n:

n (Int32, excl.) DecoupledLookback ScanPrefixes DL / SP
10 k 0.091 ms 0.097 ms 0.94×
100 k 0.120 ms 0.103 ms 1.17×
1 M 0.292 ms 0.111 ms 2.62×
4 M 0.910 ms 0.171 ms 5.33×
16 M 3.15 ms 0.31 ms 10.1×

So this PR does not change the default (ScanPrefixes() everywhere). Its value is making DecoupledLookback() correct so it can be opted into on dispatch-bound backends (e.g. Metal), where collapsing kernel launches is the win — the motivation for the single-pass/onesweep direction.

maleadt and others added 4 commits July 15, 2026 16:54
…kback

Re-enable DecoupledLookback() coverage in the non-uniform exclusive accumulate
test (excluded on the main branch). This branch carries the work-in-progress
fix for DecoupledLookback's exclusive carries, so it must exercise that path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…need

The decoupled-lookback aggregate publish/consume protocol is not memory-
coherent across blocks (stale L1 aggregate reads + missing device-scope
ordering), causing ~40% failures on smaller GPUs. The proper fix needs a
native device threadfence; `UnsafeAtomics.fence` and acquire/release atomics
do not lower on recent NVPTX toolchains (LLVM 18, sm_80) — only `monotonic`
does. Document this inline as the path forward.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The exclusive-carry fix published each block's inclusive aggregate in a
dedicated array, but the publish/consume protocol was not memory-coherent
across blocks: a consumer could observe a fresh flag == ACC_FLAG_A yet read a
stale L1 copy of the aggregate, and the producer's aggregate/flag stores were
not ordered device-scope. This caused ~40% failures on smaller GPUs.

Fix both problems:
  - Coherence: read/write the aggregate with a relaxed (monotonic), L1-bypassing
    atomic, reinterpreted through a same-width unsigned integer so float element
    types are supported (compile-time fallback to a plain access otherwise).
  - Ordering: introduce a device-scope `_decoupled_fence()` separating the
    aggregate store from the flag store (producer) and the flag load from the
    aggregate load (consumer). The generic definition is an acquire-release
    atomic fence (lowers on OpenCL/SPIR-V, Metal, oneAPI, AMDGPU); CUDA overrides
    it with `CUDA.threadfence()` (membar.gl) via a new package extension, since
    the NVPTX backend does not select scoped atomic fences.

`aggregates` is now always its own contiguous array (never a temp view) so
`pointer(aggregates, i)` is valid inside the kernel.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants