dasLLAMA: size-tiered allocation alignment, tune-rail hardening, M1/zen2 re-baseline#3423
Open
borisbat wants to merge 19 commits into
Open
dasLLAMA: size-tiered allocation alignment, tune-rail hardening, M1/zen2 re-baseline#3423borisbat wants to merge 19 commits into
borisbat wants to merge 19 commits into
Conversation
…e sessions establish_baselines grows an --ngl flag (Metal-enabled clean build, separate baseline_llm_<box>_metal.tsv via a suffix param on the baseline I/O) — first M1 Max full-offload sweep committed: ~4.4x prefill / ~2.7-3.0x decode over the best CPU rows at 4-8B. gen_profile sessions move to KVDtype.f16, matching the create_session facade default and llama-bench's F16 KV reference config; small-tier re-profile on master lands within noise (+-4.5%) as expected at pp512/tg128 depths. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…x fold addressing The generator folds emitted bare FMul+FMul+FAdd — hand-built [llvm_code] IR never passes das-level lowering, so mad()'s fusion never applied and no contract flags licensed the backend to fuse: every family paid 3 FP ops per accumulator where lcpp's kernel pays 2. fold_fma emits the same llvm.fmuladd das's mad() resolves to (fmul+fadd fallback when unresolvable). Census: busd512 tile 24mul+12add -> 12mul+12fmadd; amx steady loop 128mul+64add -> 64mul+64fmadd. M1 kernel-iso: +2.9-3.9% on nrsplit2 tile rows, box winner kstep4_nrsplit2_mr8_gkstep4 +2.0%. amx fold addressing moves from precomputed pointer arrays (64 y rows + 32 xs rows, live across the k-loop -> spilled, ~140 movq reloads per k-iteration = 28% of the steady loop) to running pointers off three bases, lcpp's base + m*ldc form. Steady loop 494 -> 378 insns, movq 141 -> 89; now inside the lcpp mmq.cpp census window. Gates: x64 emission 97/97, arm64 emission green, native grid GEN TUNE TEST OK, tests/dasLLAMA 319/319 -jit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ench rig The SPR-session opener: attributes prefill deltas to the GEMM kernels alone (no attention, no KV, no model). bench_gemm_iso grows the fleet geometry table (smol135m/qwen06b/gemma1b/ llama1b/qwen4b/llama8b + the original tinyllama rows) selected via DASLLAMA_BENCH_MODEL, NTOK override via DASLLAMA_BENCH_NTOK, reps from a fixed MAC budget. The lcpp half is test-backend-ops perf -o MUL_MAT with the same shapes: harness/backend_ops_shapes.patch adds the fleet cases + a GGML_BENCH_THREADS pin (upstream hardcodes hardware_concurrency). harness/pure_gemm_ladder.sh runs both legs at one thread count and emits a merged TSV (GFLOPS/TFLOPS -> GMAC/s normalized). gemm_1core_probe comment updated (examples/kernel-bench no longer exists upstream; the ladder is the cross-engine instrument). M1 shakeout @T8/512tok: lcpp CPU plateaus ~460-500 GMAC/s across fleet shapes; das arm64-gen 610-700. SPR runs the same instrument against the AMX build. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
>=2KB blocks get cache-line (64B) alignment; >=2MB blocks get 2MB alignment + MADV_HUGEPAGE on Linux; Windows stays uniform 64B (_aligned_msize must receive the alloc-time alignment, so it cannot be size-tiered there). The old uniform 16B alignment line-split every 512-bit vector load and AMX tileloadd row on 75% of big-array allocations (~260M split-loads/s measured on SPR AMX GEMM), and made per-launch allocation luck flip kernels between fast and slow modes: das-amx qwen4b panels ran 4x bimodal across process launches. Silicon-validated on Xeon 8488C: bimodality gone (12 runs, +-3% spread), every shape at or above the old fast mode, cls +20%. THP A/B (enabled=never, same alignment): the hugepage tier itself is +1-6% - the alignment tiers carry the win. Intel SDM ch.20 mandates 64B for tile data. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- run_gemm_iso.das: direct-run wrapper. dastest compiles tests with policies.aot_module=true which trips tune_aot_gate() and stamps reference bodies for every [tune] kernel - the dastest rail silently benches the slow tier (82 vs 1600 GMAC/s on SPR). The M1 ladder rows in d553e5fa5 need re-measuring through this wrapper. - bench_gemm_iso: pin_kernel_backend instead of select_kernel_backend for the repack half - pin runs the available() witness, which on amx stamps is the per-process XTILEDATA enable; bare select SIGILLs the first tile op. - DASLLAMA_BENCH_SKIP_ROWMAJOR skips the row-major half for pin-lattice sweeps. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… tail Hinting the tail page backs it with a full 2MB of RSS (a 2.05MB block balloons to ~4MB resident). Round the madvise length down to a 2MB multiple: the streamed bulk keeps hugepages, the tail stays 4K, physical waste ~0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… rework backend_ops_shapes.patch now also adds GGML_PERF_WEIGHT_BUFT to eval_perf: quantized leaf tensors are placed into the matching CPU extra buffer type, so test-backend-ops can actually measure llama.cpp's AMX mmq path (stock tool - and upstream - never exercises it; weights sit in the default CPU buffer and mmq requires the AMX buft). Verified on SPR: +51% on llama8b q at bs512, with per-case engagement notes on stderr and a supports_op rollback for gated shapes. pure_gemm_ladder.sh: das leg runs through run_gemm_iso.das (direct rail - the dastest rail is tune-inert, see bench commit); lcpp leg becomes two rungs (plain + lcpp-amx via the env; env stays UNSET for the plain rung - an empty value strstr-matches any buft). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
aot_module=true on every test compile (both the run path in suite.das and the serializer path) fed tune_aot_gate(), so [tune] kernels always compiled their reference bodies — the suite never exercised the tuned tier production runs use. C++-side the flag only affects module-shaped main files (ast_parse.cpp), which test programs are not, so plain interp/jit test compiles are otherwise unchanged. Known follow-up: tests/dasLLAMA/test_batch_decode.das q8 parity subtests assumed reference-tier bit-parity between the batch GEMM and GEMV kernels and now fail at tol 1e-6 under tuned kernels; fix lands separately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
select_kernel_backend activated any registered backend without invoking available() — for amx the witness doubles as the runtime enable (XTILEDATA arch_prctl), so selecting it cold SIGILLed on the first tile op. Run the witness like pin_kernel_backend and select_matmul_backend_for_load already do; warn and keep the current backend when it reports unavailable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Declined perms all run the same reference body — timing each at best-of-ROUNDS was pure duplicate measurement (a foreign-ISA box burned ~4 minutes measuring one body 36 times) AND handed the winner-picker N noise-lottery tickets for a single body. Keep one declined representative (the explicit 'reference' row when present), erase the rest via the per-variant witness. M1 tune: ~4min -> ~60s; the e2e confirm gate still guards a noise-crowned near-tie (validated live: challenger 889 vs champ 895 tok/s -> rejected, champ kept). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…venance
gen_profile / gen_asr_profile carry [tune_policy(missing = "auto")] (the
server's zero-config rail): an untuned box runs the scope tuner and re-execs
with winners stamped, and log_tune_status reports the stamp source at startup.
The platform block records provenance ('tune': 'fname=suffix (manifest|
fallback|reference)') via gather_platform, so fallback-tier numbers can never
masquerade as tuned ones — a deleted dasllama.tune.json silently cost ~5%
prefill across a whole published M1 sweep before this.
harness/backend_probe.das: tiny active-backend/witness diagnostic (which
backend is live post-load, is the gen family emitted).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…smoke The q8 bit-exact/1e-6 batch-vs-sequential contracts in test_batch_decode and test_prefill held only while the batch tile and the GEMV compiled the same fold order — true for reference bodies and for the per-ISA fallback stamps, broken by design once a tune manifest stamps e.g. a kstep4 tile next to a gkstep2 GEMV. The strict subtests now pin the portable backend (its batch and GEMV kernels share dot_q8q8, so same-dot parity holds under any stamp) and clear the pin on exit; new test_batch_decode_tuned covers the active backend at greedy- token exactness plus a 1e-3 numeric budget. Green with and without a manifest. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
M1 (tuned manifest, allocator fix, F16 KV, one process per model — an in-process 16-model sweep understates late models via heap accumulation; E2B measured 22% low): das wins prefill 16/16 vs llama.cpp (1.00-1.69x; E2B 1.68x / E4B 1.69x), trails emission 0.85-0.99x (memory-bound decode, the named next arc). Platform blocks now carry the tune stamp provenance. zen2 (allocator fix, F16 KV, 16 threads; gen manifest absent -> fallback maddubs stamps, i.e. conservative): prefill +4-19% and emission flat-to-+12% vs the prior stored profile on all 16 models; ASR noise-equal. Tuned zen2 resweep is a follow-up when the box frees up. Scoreboards regenerated from the JSONs by gen_results.das. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR hardens dasLLAMA’s tuning/profiling pipeline and updates performance methodology/results, alongside a core allocator alignment change intended to improve large-SIMD/GEMM behavior across platforms.
Changes:
- Introduces size-tiered allocation alignment in
das_aligned_alloc16(64B for ≥2KB; 2MB + Linux hugepage hint for ≥2MB; Windows fixed 64B). - Fixes tune-rail correctness (AOT gating in
dastest, backend explicit selection now runs availability witness) and adds tune provenance logging into profile JSON. - Updates/extends dasLLAMA tests and harnesses (portable pin for strict parity tests, adds a tuned-tolerance batch decode test, adds pure-GEMM ladder tooling and shape patch for llama.cpp).
Reviewed changes
Copilot reviewed 26 out of 26 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/dasLLAMA/test_prefill.das | Pins portable backend for strict numeric parity in prefill test; clears pin afterward. |
| tests/dasLLAMA/test_batch_decode.das | Pins portable backend for strict parity subtests; adds tuned-tolerance batch test. |
| modules/dasLLAMA/tune_for_this_box.md | Documents new tune/profiling behavior and provenance. |
| modules/dasLLAMA/performance/results_llm.md | Regenerated LLM results tables with new measurements/metadata. |
| modules/dasLLAMA/performance/profile_llm_zen2.json | Updated zen2 LLM profile numbers + metadata. |
| modules/dasLLAMA/performance/profile_llm_m1.json | Updated M1 LLM profile numbers + adds tune provenance field. |
| modules/dasLLAMA/performance/profile_common.das | Adds optional tune field and baseline path suffix support. |
| modules/dasLLAMA/performance/profile_asr_zen2.json | Updated zen2 ASR profile numbers + metadata. |
| modules/dasLLAMA/performance/profile_asr_m1.json | Updated M1 ASR profile numbers + adds tune provenance field and librispeech stats. |
| modules/dasLLAMA/performance/gen_profile.das | Adds [tune_policy], logs tune status, and uses F16 KV for measurement sessions. |
| modules/dasLLAMA/performance/gen_asr_profile.das | Adds [tune_policy] and logs tune status before profiling. |
| modules/dasLLAMA/performance/establish_baselines.das | Adds --ngl axis and writes per-axis baseline TSVs (e.g. _metal). |
| modules/dasLLAMA/performance/baseline_llm_m1_metal.tsv | New baseline TSV for Metal offload axis. |
| modules/dasLLAMA/harness/pure_gemm_ladder.sh | New script to compare das vs llama.cpp GEMM throughput on matched shapes. |
| modules/dasLLAMA/harness/gen_tune_probe.das | Dedupes declined-permutation benchmarking to avoid redundant work/noise bias. |
| modules/dasLLAMA/harness/gemm_1core_probe.das | Updates harness commentary to align with new ladder tooling. |
| modules/dasLLAMA/harness/backend_probe.das | New diagnostic to print active backend/witness status around a model load. |
| modules/dasLLAMA/harness/backend_ops_shapes.patch | Patch for llama.cpp test-backend-ops to add shapes + buffer controls for ladder parity. |
| modules/dasLLAMA/dasllama/dasllama_math.das | select_kernel_backend now runs availability witness (avoids unsafe activation). |
| modules/dasLLAMA/dasllama/dasllama_gemm_gen.das | Adds fmuladd folding and AMX fold addressing tweaks for generator output. |
| modules/dasLLAMA/benchmarks/matmul/run_gemm_iso.das | New direct-run wrapper to avoid tune-gate issues in dastest. |
| modules/dasLLAMA/benchmarks/matmul/bench_gemm_iso.das | Extends bench to multiple model shape sets and NTOK override; uses repack tier pin. |
| modules/dasLLAMA/benchmarks/asr/results.md | Regenerated ASR results tables. |
| include/daScript/misc/platform.h | Implements size-tiered alignment + Linux hugepage hint; Windows alignment updated. |
| dastest/suite.das | Fixes aot_module to track ctx.use_aot (tune no longer inert under tests). |
| dastest/dastest.das | Same aot_module fix for serialization path. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
bench_gemm_iso clears its backend pin on exit — with dastest no longer tune-inert, dastest-driven bench runs are meaningful and a standing global pin would leak into later files in the same process. pure_gemm_ladder.sh gets pipefail so a daslang failure can't hide behind a successful awk at the end of a pipe (a measurement harness must not emit partial TSV silently). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…module
aot_module means 'module-shaped compile' (the -aot tool, dastest's every test
compile, the AST serializer's module-symbol marking) — borrowing it as the tune
gate made dastest tune-inert, and flipping it per-run broke AST serialization
(AstSerializer::patch() read past end of stream on --deser; extended_checks red
on all three OSes).
New CodeOfPolicies.tune_frozen ('never apply per-box [tune] stamps'), set
exactly where cross-box artifact soundness demands it: the -aot driver and
dastest's serialize path. tune_aot_gate() becomes
tune_frozen || (aot && !jit_enabled): an AOT-consuming run freezes tune only
when the JIT is off — a stamp changes the semantic hash so a stamped function
cannot AOT-link, but under -jit the JIT supersedes AOT bodies and tuned stamps
stay live. dastest's run path returns to its original aot_module = true.
AST serializer version 98 -> 99 (CodeOfPolicies stream grows the field).
Verified: --ser/--deser over tests/ 10713/10713 both passes; tests/dasLLAMA
321/321 under -jit with stamps live (the tune line now appears under plain
dastest — the original goal, without the borrowed flag).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rialize AstSerializer::getVersion() existed as a bump-discipline constant with no consumer — a stale stream misparsed field-by-field until a patch() throw escaped rtti_ast_serializer_deserialize_program uncaught (the SIGABRT extended_checks showed on all three OSes this PR). Program::serialize now writes/checks the version at the head of every program record (a desynced stream keeps failing cleanly on each record), and deserialize_program wraps the read in the same dasException catch serializeScript already uses, so same-version truncation/corruption lands on the failToCompile path too. Verified: --ser/--deser over tests/ 10713/10713; corrupted version word -> clean 'stream version X does not match serializer version 99'; corrupted structural length word -> caught 'ast serializer read overflow', clean per-program failure report. No aborts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rogram) The cache path (trySerializeProgramModule) checks only mtime+filename before handing the stream to serializeProgram — a cache written by a different serializer version misparsed from the first layout difference. Same gate as Program::serialize; on mismatch ser.failed is set and the caller's existing failed-check falls back to a full parse. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… clamps thread env env -u GGML_PERF_WEIGHT_BUFT on the plain rung — an exported var would silently turn the baseline into the AMX rung (and empty is NOT unset: an empty value strstr-matches every buft). GGML_BENCH_THREADS in backend_ops_shapes.patch now falls back to hardware_concurrency on unset/zero/garbage instead of casting atoi straight to size_t. Patch re-verified to apply clean on upstream ebd048f. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…obe-verified) The zen2 profiles were generated before the platform.tune field existed; the value is backfilled from the box's own backend_probe output (x64-gen active, no manifest -> 'q8q8_tile_gen=dot_maddubs_width256_mr8_kstep2 (fallback)'), measurement data untouched. Makes the fallback-tier caveat auditable in the artifact itself; the tuned zen2 resweep stays the documented follow-up. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 34 out of 34 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
modules/dasLLAMA/dasllama/dasllama_math.das:986
pin_kernel_backendalready invokes the backend’savailablewitness before activation, andselect_kernel_backendnow invokes the same witness as well. Becausepin_kernel_backendends by callingselect_kernel_backend(name), pinning a backend will run the witness twice (which is redundant and can be undesirable if the witness has side effects like AMX enablement). Consider activating directly after the successful witness check inpin_kernel_backendto avoid the double-invoke.
activate(g_kernel_backends[name])
}
//! Pin the active backend to `name` (forces it; suppresses auto-select on later registrations). For A/B
//! benchmarking on one binary — e.g. pin "arm64-sdot" to measure the row-major baseline against the gen tier.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Four themes, one arc — the SPR-rematch fallout plus the M1/zen2 re-baseline it forced:
1. Size-tiered allocation alignment (
das_aligned_alloc16)platform.h: blocks ≥2KB now get cache-line (64B) alignment; blocks ≥2MB get 2MB alignment +MADV_HUGEPAGEon Linux (interior-only — hinting the partial tail would balloon RSS). The old uniform 16B left 75% of big SIMD panels line-split on every 512-bit/tile load (~260M split-loads/s measured on SPR AMX) and made per-launch heap luck flip kernel benches 4x bimodal. Windows stays uniform 64B (_aligned_msizemust be called with the exact alloc-time alignment, so it cannot be size-tiered). Affects every das SIMD workload; zen2 gained +4-19% prefill from this alone.2. Tune-rail hardening
aot_module = trueon every test compile fedtune_aot_gate(), so[tune]kernels always compiled their reference bodies — the suite never exercised the tuned tier production runs use (82 vs 1600 GMAC/s on SPR fixtures).tune_frozenpolicy —[tune]no longer piggybacksaot_module:aot_modulemeans "module-shaped compile" (the-aottool, dastest's test compiles, the AST serializer's module-symbol marking); borrowing it as the tune gate is what made dastest inert, and flipping it per-run broke AST serialization. NewCodeOfPolicies.tune_frozen("never apply per-box stamps") is set exactly where cross-box artifact soundness demands: the-aotdriver and dastest's serialize path.tune_aot_gate()=tune_frozen || (aot && !jit_enabled)— an AOT-consuming run freezes tune only when the JIT is off (a stamp changes the semantic hash so a stamped function cannot AOT-link; under-jitthe JIT supersedes AOT bodies and tuned stamps stay live).AstSerializer::getVersion()existed with no consumer — a stale stream misparsed field-by-field into an uncaughtpatch()throw (SIGABRT). Both entry points (Program::serialize,serializeProgram/module cache) now write/check the version and fail cleanly;deserialize_programcatchesdasExceptiononto thefailToCompilepath. Serializer version 98 → 99 (tune_frozenin the CodeOfPolicies stream).select_kernel_backendskipped the availability witness — for amx the witness doubles as the runtime enable (XTILEDATAarch_prctl), so explicit selection could SIGILL. It now runs the witness like pin and load-time auto-select already did.gen_tune_probebenched every declined perm at best-of-N — 36 identical reference bodies measured separately (~4 min of duplicate work on arm64) that also handed the winner-picker 36 noise-lottery tickets. Declined perms bench once via the per-variant witness; M1 tune ~60s; the e2e confirm gate validated live (rejected a near-tied challenger 889 vs 895 tok/s).gen_profile/gen_asr_profilecarry[tune_policy(missing = "auto")], and the platform block records stamp provenance ("tune": "fname=suffix (manifest|fallback|reference)"). This gap cost a day: a deleteddasllama.tune.jsonsilently put a whole M1 sweep on fallback stamps, ~5% low, with nothing in the output saying so.3. M1 + zen2 re-baseline (new numbers, new methodology)
4. Harness / rig additions
harness/backend_ops_shapes.patch: llama.cpptest-backend-opsnever engages AMX (mmq needs weights in the AMX extra buft; the stock tool uses the default CPU buffer) —GGML_PERF_WEIGHT_BUFTin eval_perf fixes it (+51% verified). All prior stock-tool "lcpp AMX" per-op numbers anywhere were actually non-AMX.harness/backend_probe.das: active-backend/witness diagnostic (which backend is live, is the gen family emitted) — earned its keep twice this arc.benchmarks/matmul/ladder rig: shape-matched das vs llama.cpp MUL_MAT bench (run_gemm_isodirect wrapper dodges the dastest tune-gate),--nglMetal-offload baseline axis in the establisher.llvm.fmuladd) + running-pointer amx fold addressing.tests/dasLLAMA/test_batch_decode.das+test_prefill.das: the strict bit-exact/1e-6 batch-vs-sequential parity subtests assumed reference-tier kernels (the batch GEMM and the GEMV were literally the same dot loop). Under tuned kernels those tiers legitimately fold in different orders (a kstep4 tile vs a gkstep2 GEMV), so the strict q8 subtests now pin theportablebackend (its batch and GEMV kernels sharedot_q8q8— same-dot parity holds under any stamp), and a newtest_batch_decode_tunedsubtest covers the active backend at token-exactness + a 1e-3 numeric budget. Both files were green on the untuned CI configuration before AND after; the pin makes them manifest-proof.Verification
tests/dasLLAMAfull suite: 321/321 under-jitat final tree state (tuned manifest present); also green with no manifest (the CI configuration).--ser/--deserover the fulltests/tree: 10713/10713 both passes, plus corruption negatives — flipped version word → clean "stream version X does not match serializer version 99"; flipped structural length word → caught "ast serializer read overflow", clean per-program failure. No aborts.--full: 17/17 gates green (format, lint, cpp-syntax full sweep, dasgen, ci-das, 6 doc gates + pdflatex, tests-cpp, interp, JIT, AOT, sequence smoke).🤖 Generated with Claude Code