fix nightly failures - #1027
Open
simeonschaub wants to merge 8 commits into
Open
Conversation
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
AMDGPU.jl Benchmarks
Details
| Benchmark suite | Current: b480bac | Previous: 16f5974 | Ratio |
|---|---|---|---|
amdgpu/synchronization/context/device |
770 ns |
755 ns |
1.02 |
amdgpu/synchronization/stream/blocking |
322.25 ns |
317.5 ns |
1.01 |
amdgpu/synchronization/stream/nonblocking |
457.5 ns |
432.25 ns |
1.06 |
array/accumulate/Float32/1d |
95024.5 ns |
97092.25 ns |
0.98 |
array/accumulate/Float32/dims=1 |
287978.5 ns |
299718.5 ns |
0.96 |
array/accumulate/Float32/dims=1L |
122437 ns |
121446.75 ns |
1.01 |
array/accumulate/Float32/dims=2 |
127116.75 ns |
130896.75 ns |
0.97 |
array/accumulate/Float32/dims=2L |
2809645.75 ns |
2806321.5 ns |
1.00 |
array/accumulate/Int64/1d |
111314.5 ns |
113217 ns |
0.98 |
array/accumulate/Int64/dims=1 |
267181 ns |
269376.25 ns |
0.99 |
array/accumulate/Int64/dims=1L |
140499.25 ns |
144501.75 ns |
0.97 |
array/accumulate/Int64/dims=2 |
130144.5 ns |
127879.25 ns |
1.02 |
array/accumulate/Int64/dims=2L |
2946915 ns |
2948585.75 ns |
1.00 |
array/broadcast |
83419.75 ns |
64054.75 ns |
1.30 |
array/construct |
3062.5 ns |
3220 ns |
0.95 |
array/copy |
40252.25 ns |
39815 ns |
1.01 |
array/copyto!/cpu_to_gpu |
94114.5 ns |
94539.5 ns |
1.00 |
array/copyto!/gpu_to_cpu |
94577 ns |
95039.5 ns |
1.00 |
array/copyto!/gpu_to_gpu |
43074.75 ns |
42115 ns |
1.02 |
array/iteration/findall/bool |
198294 ns |
198036.5 ns |
1.00 |
array/iteration/findall/int |
205316.25 ns |
207411.5 ns |
0.99 |
array/iteration/findfirst/bool |
164796.75 ns |
167311.75 ns |
0.98 |
array/iteration/findfirst/int |
153806.5 ns |
155156.75 ns |
0.99 |
array/iteration/findmin/1d |
179379.25 ns |
179444.25 ns |
1.00 |
array/iteration/findmin/2d |
161676.5 ns |
161089.25 ns |
1.00 |
array/iteration/logical |
314726 ns |
317088.25 ns |
0.99 |
array/iteration/scalar |
344310.75 ns |
343653.25 ns |
1.00 |
array/permutedims/2d |
82659.5 ns |
82694.75 ns |
1.00 |
array/permutedims/3d |
83422.25 ns |
82232.25 ns |
1.01 |
array/permutedims/4d |
85262 ns |
84369.75 ns |
1.01 |
array/random/rand/Float32 |
52722.25 ns |
53677.5 ns |
0.98 |
array/random/rand/Int64 |
67107.25 ns |
66274.75 ns |
1.01 |
array/random/rand!/Float32 |
49102.25 ns |
49389.75 ns |
0.99 |
array/random/rand!/Int64 |
67384.75 ns |
59654.75 ns |
1.13 |
array/random/randn/Float32 |
94677 ns |
93112.25 ns |
1.02 |
array/random/randn!/Float32 |
72604.75 ns |
69269.75 ns |
1.05 |
array/reductions/mapreduce/Float32/1d |
141886.75 ns |
141249.25 ns |
1.00 |
array/reductions/mapreduce/Float32/dims=1 |
104702 ns |
105462 ns |
0.99 |
array/reductions/mapreduce/Float32/dims=1L |
853138.25 ns |
862533.5 ns |
0.99 |
array/reductions/mapreduce/Float32/dims=2 |
106369.5 ns |
108059.5 ns |
0.98 |
array/reductions/mapreduce/Float32/dims=2L |
155276.75 ns |
155446.75 ns |
1.00 |
array/reductions/mapreduce/Int64/1d |
145059.25 ns |
140932 ns |
1.03 |
array/reductions/mapreduce/Int64/dims=1 |
105286.75 ns |
104764.5 ns |
1.00 |
array/reductions/mapreduce/Int64/dims=1L |
853575.75 ns |
860696 ns |
0.99 |
array/reductions/mapreduce/Int64/dims=2 |
107987 ns |
107757 ns |
1.00 |
array/reductions/mapreduce/Int64/dims=2L |
160981.5 ns |
163496.75 ns |
0.98 |
array/reductions/reduce/Float32/1d |
141539.25 ns |
145184.25 ns |
0.97 |
array/reductions/reduce/Float32/dims=1 |
104734.5 ns |
105042 ns |
1.00 |
array/reductions/reduce/Float32/dims=1L |
859465.75 ns |
856898.25 ns |
1.00 |
array/reductions/reduce/Float32/dims=2 |
106242 ns |
106659.5 ns |
1.00 |
array/reductions/reduce/Float32/dims=2L |
160001.5 ns |
161129.25 ns |
0.99 |
array/reductions/reduce/Int64/1d |
141344.25 ns |
141302 ns |
1.00 |
array/reductions/reduce/Int64/dims=1 |
105494.5 ns |
105144.5 ns |
1.00 |
array/reductions/reduce/Int64/dims=1L |
859710.5 ns |
851721 ns |
1.01 |
array/reductions/reduce/Int64/dims=2 |
107861.75 ns |
107839.5 ns |
1.00 |
array/reductions/reduce/Int64/dims=2L |
163634.25 ns |
162989.25 ns |
1.00 |
array/reverse/1d |
48377.25 ns |
53687.25 ns |
0.90 |
array/reverse/1dL |
64074.75 ns |
78502.25 ns |
0.82 |
array/reverse/1dL_inplace |
59129.75 ns |
62952.25 ns |
0.94 |
array/reverse/1d_inplace |
43724.75 ns |
45130 ns |
0.97 |
array/reverse/2d |
52957.25 ns |
55499.75 ns |
0.95 |
array/reverse/2dL |
95872 ns |
111822 ns |
0.86 |
array/reverse/2dL_inplace |
68434.75 ns |
74057 ns |
0.92 |
array/reverse/2d_inplace |
46774.75 ns |
49312.25 ns |
0.95 |
array/sorting/1d |
378600.75 ns |
359238.75 ns |
1.05 |
integration/byval/reference |
41829 ns |
41860 ns |
1.00 |
integration/byval/slices=1 |
43220 ns |
42700 ns |
1.01 |
integration/byval/slices=2 |
132229 ns |
133939 ns |
0.99 |
integration/byval/slices=3 |
239969 ns |
239880 ns |
1.00 |
integration/volumerhs |
4923665 ns |
4916200 ns |
1.00 |
kernel/indexing |
46967.25 ns |
44594.75 ns |
1.05 |
kernel/indexing_checked |
67632.25 ns |
47782.25 ns |
1.42 |
kernel/launch |
1770 ns |
1785 ns |
0.99 |
kernel/rand |
107742 ns |
110472 ns |
0.98 |
latency/import |
2248298275 ns |
2242598207 ns |
1.00 |
latency/precompile |
52725426780 ns |
52102215931 ns |
1.01 |
latency/ttfp |
3079590314 ns |
3070234127 ns |
1.00 |
This comment was automatically generated by workflow using github-action-benchmark.
Member
|
Nightly is allow to soft fail, which still happens here. Is something else missing? |
Member
Author
|
Yes, seems like the fix just surfaced another issue |
Member
Author
|
Should be addressed by JuliaLang/julia#62807 |
… top of `noinline`
Recent Julia nightlies (observed on 1.14.0-DEV.2948, LLVM 21) emit the
LLVM `noinline` attribute on outlined throw functions such as
`julia__throw_boundserror_indices`. `finish_module!` force-inlines
exception-related functions by adding `alwaysinline` to anything
matching `__throw_`/`signal_exception`/`report_exception`/`malloc`, and
LLVM's verifier rejects a function carrying both attributes:
Attributes 'noinline and alwaysinline' are incompatible!
ptr @julia__throw_boundserror_indices_0
LLVM ERROR: Broken module found, compilation aborted!
The verifier only runs when GPUCompiler's `should_verify()` is true
(debug builds, `-g2`, or `CI=true`), which is why this aborted
precompilation and every kernel compile on Buildkite while remaining
invisible locally.
Since force-inlining these functions is the deliberate intent here,
strip `noinline` before adding `alwaysinline`, mirroring what
GPUCompiler does in `inline_unreachable_control_flow!`. Also drop the
vestigial `unsafe_fp_atomics ||` condition in the loop: its body only
ever acted on `do_inline` matches, so it had no effect.
Verified on 1.14.0-DEV.2964 with `CI=true`: compiling a GCN kernel
containing a bounds-error throw path reproduces the exact CI abort
before this change and succeeds after, including the full package
precompilation with the `precompile.jl` workload under the verifier.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ia 1.14 JuliaLang/LinearAlgebra.jl#1671 (in Julia 1.14.0-DEV.1837) moved the BLAS fast path for matrix-vector products from `generic_matvecmul!` to the char-based `mul!`, whose `StridedVector{<:BlasFloat}` method captures `ROCArray` (a `DenseArray`) and hands device pointers to CPU BLAS, causing a `ReadOnlyMemoryError`. Overload the char-based `mul!` — the now-documented extension point for custom storage types — to forward to the existing `generic_matvecmul!` implementation. Assisted-by: Claude Code (Fable 5)
On Julia 1.14, the bounds-check error path for 3+-dimensional indexing (`_throw_boundserror_indices`) is specialized as a vararg-widened MethodInstance, whose invoke edge currently gets no CodeInstance under GPUCompiler's cache and is emitted as an unsupported dynamic call. The kernel indices are in bounds by construction (guarded by the thread-count check, and the reversal maps indices within the same range), so elide the bounds checks entirely. Assisted-by: Claude Code (Fable 5)
LLVM 21 extended `GEPToVectorIndex` to walk nested GEP chains, but computes the vector index with an unsigned division of the constant byte offset. A negative intermediate offset — e.g. the `gep i32 %tuple, %j` + `gep i8, -4` julia emits for 1-based dynamic tuple indexing — yields an out-of-bounds index, which constant-folds to poison after unrolling and silently deletes the accesses. This made GPUArrays' batched `rand!` write nothing to arrays whose length is a multiple of the values-per-call batch size. LLVM 20 rejects such chains (unaffected) and LLVM 22 divides signedly (fixed), so disable the pass only for in-process LLVM 21: with the external back-end, llc (LLVM 22+) re-runs promote-alloca in its codegen pipeline, so the optimization is not lost. Assisted-by: Claude Code (Fable 5)
The LinearAlgebra bump that moved the BLAS fast path to the char-based `mul!` is julia commit b77eb3bc91 = 1.14.0-DEV.2991; the previously used 1837 was an undercount from naming the commit in a shallow clone. Assisted-by: Claude Code (Fable 5)
simeonschaub
force-pushed
the
sds/fix_nightly
branch
from
August 18, 2026 15:53
f6b8a40 to
5f6fdab
Compare
NFC, but avoids another level of indentation and makes the diff smaller
araujoms
pushed a commit
to JuliaLang/LinearAlgebra.jl
that referenced
this pull request
Aug 18, 2026
These were removed in #1668, but SparseArrays still relies on these methods in `generic_matmatmul!` since it calls `uppercase`. Found tracking down regressions in AMDGPU.jl on nightly ref JuliaGPU/AMDGPU.jl#1027
Member
Author
|
It looks like the nightly timeout might need to be increased? (Nightly is still expected to fail until JuliaLang/julia#62823 is merged) |
Member
|
Yes, good catch. One could bump it to 90 as for the other tests. |
This reverts commit ba599b5. Nightly switched to LLVM 22 and 1.13 is LLVM 20
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Previously, the
noinlineattribute emitted by Julia for some error paths collided with us settingalwaysinline. Make thealwaysinlinetake precedence here