New -O4 tree-level optimization passes, x86 peepholes, and supporting fixes#13
Open
joaopauloschuler wants to merge 73 commits into
Open
New -O4 tree-level optimization passes, x86 peepholes, and supporting fixes#13joaopauloschuler wants to merge 73 commits into
joaopauloschuler wants to merge 73 commits into
Conversation
Introduce a node-level loop-invariant code motion pass over tfornode and
twhilerepeatnode bodies. It hoists side-effect-free, exception-free,
loop-invariant numeric/pointer subexpressions (constants and non-aliased
local/parameter reads combined with +, - and *) into the loop preheader as
write-once temps, so they evaluate once instead of on every iteration --
orthogonal to strength reduction, which rewrites the index recurrence rather
than lifting invariants out of it.
Soundness policy (a wrong hoist is a miscompile, so this is deliberately
conservative):
* invariance is proven via the DFA def-set of the whole loop node, which for
a for-loop includes the counter, so anything assigned in the loop -- the
counter included -- disqualifies the expression;
* only pure, non-trapping trees are hoisted (no calls, pointer derefs, array
indexing, div/mod, range/overflow-checked arithmetic, volatile/threadvars,
property/field access, or managed-type operations such as string "+"),
which makes preheader evaluation safe even for a zero-trip loop;
* aliasing is punted on: any addr-taken variable is treated as non-invariant
and no pointer/field/index read is ever hoisted, so a store through a
pointer in the loop cannot invalidate a hoist.
Gated by the new cs_opt_loopmotion switch (-OoLICM / {$OPTIMIZATION LICM}),
enabled by default at -O4 via genericlevel4optimizerswitches. Runs in psub
inside the DFA block, after induction-variable strength reduction, skipping
procedures that contain labels (as strength reduction does).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
testtool cases that fail if LICM miscompiles: an invariant multiply in a nested loop, a while-loop invariant, an invariant float subexpression, a zero-trip loop (hoist must stay harmless), a var modified in the loop (not invariant), pointer-aliased writes to an addr-taken var (not invariant), a div-by-a-zero-variable in a zero-trip loop (a trap must never be speculated into the preheader), and an invariant string "+" concat (a managed op must never be hoisted as arithmetic). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Port gcc's -funswitch-loops to FPC as a node-level pass over tfornode and
twhilerepeatnode bodies. When a conditional inside a loop tests a loop-invariant
expression, the test is evaluated once into a preheader temp and the loop is
cloned into a then-variant and an else-variant, so each cloned body is
branch-free for that test:
for i:=... do temp:=<cond>;
if <cond> then --> if temp then
A for i:=... do A { branch-free }
else else
B; for i:=... do B; { branch-free }
This is the neural-api kernel shape: an element loop that branches on a per-call
invariant flag (use-bias / apply-activation / stride / padding), paying a
compare+branch per element and blocking a uniform, vectorizable body.
Soundness policy (a wrong unswitch is a miscompile, so this is deliberately
conservative and reuses the LICM analysis):
* the condition must be pure, non-trapping and loop-invariant per the DFA
def-set of the whole loop node -- proven with the LICM helper factored out
as licm_is_pure_invariant, extended only with relational (=,<>,<,<=,>,>=)
and boolean-not wrappers, which likewise cannot trap or have side effects;
calls, derefs, div/mod, checked arithmetic and addr-taken/volatile/threadvar
reads are all rejected, so nothing that could trap is ever speculated;
* because the condition is pure and non-trapping, evaluating it once in the
preheader is safe even for a zero-trip loop;
* only one if is unswitched per loop (no cascading) and only when the body
fits a node-weight budget (node_count_weighted <= 50), so cloning cannot
blow up code size;
* nested loops are unswitched innermost-first (postorder), matching LICM;
* procedures containing labels are skipped at the call site, as strength
reduction and LICM do, so goto never crosses a clone boundary; break/exit/
continue inside a branch stay correct because the two clones are mutually
exclusive and each is its own loop.
The target if is marked by pointing its condition at the (unique) preheader
temp; after node_getcopy each clone carries an if testing that temp, which lets
each clone be specialised to its then/else branch unambiguously.
Gated by the new cs_opt_loopunswitch switch (-OoLOOPUNSWITCH /
{$OPTIMIZATION LOOPUNSWITCH}), enabled by default at -O4 via
genericlevel4optimizerswitches. Runs in psub inside the DFA block, immediately
before LICM with a DFA refresh in between, so the branch-free clones it produces
are exposed to invariant hoisting.
Tests under unleashed/tests/testfiles/optunswitch cover the condition-true and
condition-false paths, a zero-trip loop (preheader eval must stay safe), a
condition that would trap if speculated (must not unswitch), a non-invariant
condition and an addr-taken/aliased condition (must not unswitch), nested loops
(inner unswitched), break inside a branch, a while-loop, a relational condition,
and a body over the size cap (must not unswitch). Runtime trip counts plus
NOAUTOINLINE keep the loops from being fully unrolled so unswitching is the
transform actually exercised.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Recognize the canonical scalar clear-lowest-set-bit population-count loop
while x <> 0 do begin inc(c); x := x and (x - 1) end;
and rewrite it to
inc(c, PopCnt(x)); x := 0;
so the backend emits POPCNT under the CPUID feature gate (e.g. -CpCOREAVX2)
and the fpc_popcnt_* RTL routine otherwise, replacing a multi-instruction
scalar loop with a single instruction in the hot paths of bit-heavy code
(core.bloom / core.hyperloglog / core.roaring / core.bits).
Pass OptimizeBitIdiom in optloop.pas, driven from the psub.pas DFA block
next to OptimizeLICM / OptimizeLoopUnswitch, gated by the new switch
cs_opt_bitidiom (-OoBITIDIOM) which genericlevel4optimizerswitches enables
at -O4. Switch plumbing mirrors the two loop passes: globtype.pas enum +
OptimizerSwitchStr, x86_64/cpuinfo.pas supported_optimizerswitches, and
ppudump.pp.
Recognized shapes (a false positive is a miscompile, so matching is strict):
* the two body statements in either order;
* the counter as inc(c) or as c := c + 1 (inc() is lowered to the latter by
firstpass before this pass runs);
* the unsigned while x > 0 guard (rejected for signed x);
* x and c distinct simple, non-aliased, non-volatile, non-threadvar
local vars or value params, ordinal, exactly 32 or 64 bits;
* the counter step exactly +1; the body exactly the two statements;
* only bit-preserving same-size ordinal reinterpret typeconvs are looked
through, so a narrowing while Word(x) <> 0 is never misread;
* signed x is reinterpreted to the same-width unsigned type before PopCnt,
so the count matches the loop for negative/high-bit values.
The -O2 while -> "if cond then do..while cond" simplification turns the loop
into a test-at-end form by the time this pass sees it; for that form the
rewrite is wrapped in a zero-trip if <cond> then ... guard so x=0 stays a
no-op (PopCnt(0)=0). repeat..until is rejected: it runs the body once for
x=0, which PopCnt would miscount.
Tests under unleashed/tests/testfiles/optbitidiom/: positive rewrites (dword,
qword, reversed body, c:=c+1, x>0, signed 32/64) and must-not-fire negatives
(step +2, extra statement, x shr 1 bit-length, repeat..until), verified by
behaviour and by assembly (popcnt / fpc_popcnt present vs absent).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New optimizer switch cs_opt_rangecheckelim / -OoRANGEELIM (in
genericlevel4optimizerswitches, so on by default at -O4). Pass
OptimizeRangeElim in optloop.pas, driven from the DFA block in psub.pas
after OptimizeBitIdiom and skipped for procedures containing labels.
When range checking is on (-Cr), the per-access array bounds check that
-Cr injects in front of a[i] inside a counted for-loop is removed when
the loop counter i is provably always a valid index. The suppression
reuses FPC's own per-node range-check switch: for a dynamic array the
check (fpc_dynarray_rangecheck) is gated on the vecn's localswitches, and
for a static array it lives in the index->subrange typeconv child, so the
pass clears cs_check_range on both the vecn and its index child -- exactly
the primitive tvecnode.gen_array_rangecheck already uses in nmem.pas.
Two deliberately conservative patterns qualify (a wrongly removed check is
a memory-safety miscompile):
A. static array[Lo..Hi] indexed by the counter whose *constant* loop
bounds lie inside [Lo..Hi];
B. dynamic array a (a simple non-aliased local/value-param loadn, not
address-taken, DFA-proven not reassigned in the body) indexed by the
counter when the loop bound is literally high(a) or length(a)-1 and
the other bound is a constant >= 0.
The counter must be a simple non-aliased local/value-param variable not
written in the loop body (DFA def-set). Everything else keeps its check:
offset indices (a[i+1]), a manually computed index, a different array, a
plain length(a) off-by-one bound, a global counter, or an array
SetLength'd smaller inside the loop. Zero-trip safe (high(nil) = -1).
Plumbing mirrors the OoBITIDIOM pass: globtype.pas (enum + OptimizerSwitchStr
'RANGEELIM' + genericlevel4), x86_64/cpuinfo.pas supported_optimizerswitches,
ppudump.pp. Tests under unleashed/tests/testfiles/optvrp/ cover in-bounds
static/dynamic/backward/length-1 loops (check-free, correct output),
zero-trip nil, and five must-still-raise cases (i+1, off-by-one, SetLength
shrink, manual index, runtime-variable bound). Assembly diff confirms the
qualifying inner loop drops all fpc_rangeerror/fpc_dynarray_rangecheck;
a hot c[i]:=a[i]*b[i]+c[i] loop built -Cr -O4 is ~2.9x faster with the
pass on, byte-identical output.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rewrites the canonical single-precision element-wise loop for i := lo to hi do a[i] := b[i] <op> c[i] (op in + - *) over simple non-aliased dynamic arrays of single into a 128-bit SSE (or AVX-encoded, when available) packed main loop processing 4 lanes per iteration plus a scalar remainder loop. Loop control stays as ordinary nodes; only the 4-lane body is a dedicated backend node (tvectoropnode, x86 override emitting movups + addps/subps/mulps). The recognizer is strict so a miscompile cannot slip through: signed non-aliased local counter proven unmodified by DFA, exactly one plain assignment of the recognized shape, same index on every access (so element-wise semantics make it alias-safe without runtime guards), range/overflow checking disables the transform, and the FP result is bit-identical to scalar (no reassociation, no fast-math gate needed). Opt-in via -O4 -OoVECTORIZE for now; promoting it into the -O4 default set is a follow-up once it has soaked. Tests: unleashed/tests/testfiles/optvect/ — verified bit-exact against scalar recomputation for all tail residues, aliasing (a:=b shared block), negatives/special values (NaN/Inf/-0.0), -Cr fallback, and the AVX encoding; full 804-test suite green (3 pre-existing environment failures unrelated to this change). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed `x mod c` with c not a power of two previously fell through to a hardware idiv. Add a magic-number lowering in tx86moddivnode.pass_generate_code (x86/nx86mat.pas): reuse the existing signed-division magic sequence to compute the quotient q := x div c, then derive the remainder as x - q*c, avoiding idiv. Gated identically to the sibling const paths (right is ordconstn, signed, sizes 4 and 8) and placed after the power-of-two signed-mod branch so that fast path is unaffected. Bit-exactness verified against hardware/reference semantics for negative dividends (truncation toward zero), Low(LongInt)/Low(Int64), divisors c and -c (3, 7, 10, 1000003, and Low(Int64) mod -1 which traps idiv but the magic/pow2 path returns 0), across longint and int64. New tests under unleashed/tests/testfiles/optmodconst/. Full regression suite green (only the known pre-existing failures remain). Assembly check: signed `mod 10`/`mod 1000003` emit imul+shifts, no idiv. Micro-benchmark (-O3, tight x mod 10 loop, 3000 * 2M iters): magic 15.46 s vs idiv 25.54 s (~1.65x faster), identical results. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Extend OptimizeBitIdiom with two scalar bit-scan loop shapes, lowered to
the Bsf/Bsr intrinsics (TZCNT/BSF, LZCNT/BSR under the CPU feature gate):
* count-trailing-zeros: if x<>0 then while (x and 1)=0 do
begin inc(c); x:=x shr 1 end
-> inc(c, Bsf(x)); x := x shr Bsf(x)
The scalar loop is infinite for x=0, so this is only sound when a
dominating nonzero test proves x<>0. The pass therefore anchors on the
enclosing if x<>0 (or unsigned x>0), keeps that guard verbatim and only
rewrites its body; loops without such a structural proof are left alone.
* highest-set-bit / floor-log2: while x>1 do begin inc(c); x:=x shr 1 end
-> if x<>0 then begin inc(c, Bsr(x)); x := 1 end
This loop is total (zero-trip for x in 0 and 1); the emitted if x<>0 only
avoids the undefined Bsr(0) and reproduces the x=0 no-op. Unsigned only.
Both shapes are recognized in the raw test-at-begin while form and in the
-O2 do-while-in-if form (shared bitidiom_scan_loop_parts helper, which
verifies the do-while entry guard equals the loop condition). Restricted to
32/64-bit ordinal locals/value-params, exact +1 counter step, exactly the
two-statement body -- the same strict policy as the population-count matcher
it reuses. 8/16-bit operands are integer-promoted and fall through to the
scalar loop unchanged.
Tests under unleashed/tests/testfiles/optbitscan/ verify bit-exact identity
with a scalar reference across all single-bit values, dense sweeps, High/Low
and the guarded x=0 no-op for 8/16/32/64-bit (and signed 32-bit tzcnt), and
a must-not-fire set whose results would differ if the pass misfired.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…er offsets
Induction-variable strength reduction previously only lowered a bare
x := i*stride in a for-loop to an additive accumulator; the same multiply
buried in an array index or pointer offset (g[i*7], p[i*stride],
p[ch*size + row*width + col]) still emitted a per-iteration imul, because
FPC wraps the counter load in an implicit value-preserving conversion for
index/offset arithmetic, so taddnode(n).left was a typeconvn, not the bare
loadn the MUL case matched.
dostrengthreductiontest's MUL case now recognises the counter through a
chain of value-preserving integer conversions (is_reducible_loopvar /
is_value_preserving_int_conv: strict integer widening, not signed->unsigned).
Soundness gates mirror the existing pass and are deliberately conservative:
* the multiply is skipped when cs_check_overflow/cs_check_range is active
on the muln node (a checked multiply may trap and must run every
iteration); this also preserves the historic behaviour that -Cr/-Co
disabled the reduction (previously by accident, via the check-emitting
typeconvs breaking the loadn match);
* a conversion carrying a range/overflow check is never seen through;
* only plain reads of the counter (no nf_write/nf_modify) qualify;
* the multiplier must be loop-invariant (unchanged: is_loop_invariant).
Nested conv-like offsets reduce level-by-level: each loop's own
counter*invariant term (ch*size, row*width) is reduced when that loop is
processed; the innermost +col add stays. Range checks on the index are
unaffected because the reduction is gated off under -Cr and, when it does
fire, feeds the vecn the identical index value via the accumulator temp.
New tests under unleashed/tests/testfiles/optstrred/ verify bit-exact
results vs a naive recompute for constant/runtime strides, the multi-term
flat offset in a nested loop, backward loops, boundary trip counts (0/1/many),
a non-matching variant-stride case, and -Cr correctness. Full regression
suite green (minus the known pre-existing failures). On a 32x64x64
conv-like flat-offset nest the inner loop drops from 6 imuls to 0 and runs
~35% faster (1.9s vs 3.0s).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Report, per hot for-loop, whether -O4 -OoVECTORIZE autovectorized it and, when it did not, the concrete reason the recognizer bailed. This is a maintainer feedback signal so loops that fall back to scalar code no longer have to be found by reading generated assembly. It measures/reports only: codegen is unchanged (verified by a byte-identical .s diff with/without the patch). Mechanism: - Two new codegenerator notes in msg/errore.msg: cg_n_loop_vectorized (06065) and cg_n_loop_not_vectorized (06066, "$1" = reason). Regenerated msgidx.inc and msgtxt.inc via utils/msg2inc (both are .gitignore'd build artifacts, so they are rebuilt from errore.msg at compile time and not committed). - Instrumented tvectorizecontext.processloop: the recognizer body moved into a nested vectorize_reason function whose every early exit now carries a human-readable reason string (descending loop, non-unit step, aliasing, non-single element type, multiple statements in body, checked code -Cr/-Co, non-counter/offset index, unsupported operator, counter modified in body, ...). On failure it emits 06066 at the for-loop position; on success 06065. - vect_single_dynarray_elem is reimplemented on top of a new vect_elem_reason helper so per-operand array-shape failures (destination/first/second source) get precise reasons. Zero-cost and gating: OptimizeVectorize only runs when cs_opt_vectorize is set (psub.pas), so normal builds never build reason strings or emit anything. The notes follow standard FPC note gating -- shown at default/-vn verbosity, suppressed with -vn-. Verified manually: - vectorizable single a[i]:=b[i]+c[i] in a proc prints "Loop autovectorized (SSE, 4-wide packed single)" at the loop line; - double elements -> "array element type is not single-precision float"; two-statement body -> "loop body is empty or has multiple statements"; b[i+1] index -> "array index is not a plain variable read"; descending -> "descending (downto) loop"; -Cr -> "range/overflow checking is enabled (-Cr/-Co)"; - without -OoVECTORIZE nothing new is printed; - .s output for a sample is byte-identical with and without the patch. Two run+compile tests added under unleashed/tests/testfiles/optvect/ exercising the diagnostic active (-vn) on vectorized and several non-vectorized loops, asserting bit-exact scalar-equivalent results. Full suite green (814 pass; only the documented pre-existing failures remain). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Extend the conservative SSE/AVX autovectorizer (-O4 -OoVECTORIZE) beyond the
array-array a[i]:=b[i] op c[i] shape to two more single-precision element-wise
kernels neural-api's TNNetVolume uses constantly:
* scalar broadcast: a[i] := b[i] op s and a[i] := s op b[i]
where s is a loop-invariant single (a simple non-aliased local/value-param
the single-statement body never writes, or a constant literal). The scalar
is splatted ONCE before the vector loop (movss/vmovss + shufps/vshufps into
a 16-byte memory slot, hoisted by a new vok_broadcast tvectoropnode kind),
then a packed add/sub/mul per iteration. Non-commutative subtraction keeps
s - b[i] per lane via the scalarleft flag.
* plain copy: a[i] := b[i] -> a packed movups load+store pair with a scalar
tail.
tvectoropnode grows a `kind` (vok_arr_arr/arr_scalar/copy/broadcast) and a
`scalarleft` field; the x86 pass_generate_code switches on kind. Soundness gates
mirror the array-array shape exactly: dynamic arrays of single, plain-counter
index on every access, single-precision arithmetic only (is_single guard on the
RHS and the scalar operand, so a double/int scalar that would widen the scalar
loop is rejected), -Cr/-Co disables, and the invariant scalar must be a simple
non-global, non-address-taken, non-volatile local/param or a constant. FP
results stay bit-identical (same per-lane op and order, the broadcast puts the
identical bit pattern of s in every lane; NaN/Inf/-0.0 propagate). The
-OoVECTORIZE diagnostic (06065/06066) reason strings are extended for the new
near-misses.
optdfa: register vectoropn in the CheckAndWarn uninitialized-variable walk
(MaybeSearchIn) -- reachable now that a live scalar is read by the body node.
New optvect tests: scalar broadcast across all ops incl. non-commutative and
constant operands (trip counts 0..17, 100, 1000), plain copy, special values
(NaN/Inf/-0.0, bit-compared), self-alias, and global/address-taken rejection.
Full suite green (minus known pre-existing failures). Micro-benchmark of a
memory-bandwidth-bound b[i]*s kernel: ~1.13x.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The stock -Oodeadstore pass only removes a redundant store whose LHS is a
whole-variable loadn (redundancy decided from DFA whole-variable liveness,
which is too coarse to reason about an individual field or element). Add a
second, memory-location-precise straight-line pass that kills a store to a
record field or a constant-index static-array element when a LATER store
overwrites the exact same location before anything can observe it.
The extended pass keeps a per-run "pending" set of stores to compile-time
known slots (base variable + constant field/element path) and removes an
earlier store only when a later store to the structurally-equal location
(tnode.isequal: same base sym, same field vs, same constant index) is reached
with no intervening observer. It is deliberately over-conservative:
* base must be a non-address-taken local / value-or-const parameter /
non-different-scope static var of the current routine (same predicate as
the stock pass), so two distinct bases can never overlap;
* the location path may only contain record-field subscripts and constant
ordinal indexes into genuine non-packed non-special static arrays -- any
pointer-backed access (p^, dynamic/open array) is rejected because it
could alias a different base variable (e.g. f(x,x));
* managed/ref-counted slot types are skipped (finalization side effects);
* the RHS must be free of exception/range/overflow side effects and contain
no call, pointer deref or asm node;
* any read of the base variable, any call, deref, asm, compound assignment
or control-flow construct between two stores keeps the earlier store.
The function-result variable is intentionally not covered: it is passed by
reference and can alias a var/const parameter (a := f(a)), which base-only
reasoning cannot see -- the stock whole-var pass excludes it for the same
reason.
Level wiring: left opt-in under -Oodeadstore (not promoted into -O4). A forced
-O4/-Oodeadstore sweep showed the pre-existing stock pass already miscompiles
several mode-unleashed constructs (statement-expressions, inline vars,
function references) that were never exercised because the switch is in no -O
level; promoting it would regress those. A -O3 -Oodeadstore sweep over the
full testfiles corpus is byte-for-byte identical in pass/fail with the
extension on vs off, i.e. this change introduces no new failures.
Adds unleashed/tests/testfiles/dead_store_ext/ covering record-field and
array-element removal, the read/call/address-taken/managed safety matrix, and
soundness under -Cr -Co.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add CallFrameRet2Jmp, a -O4 post-peephole that converts a sibling tail call
made from a routine that owns a stack frame into a jump. When the caller has
its own frame, the teardown sits between the call and the ret
call X
L: leaq N(%rsp),%rsp ; stack release
popq %rbx ; optional callee-saved restores
ret
so the existing CallRet2Jmp / Lea/PushCallRet variants no longer match. The
new pass hoists a copy of the teardown above the call and rewrites the call to
a jump, leaving the original teardown+ret in place (label L may be the merge
point of other, non-call, exit paths such as base cases):
leaq N(%rsp),%rsp
popq %rbx
jmp X
Covered subset (chosen for provable peephole-level soundness):
* teardown = a stack release (leaq N(%rsp),%rsp or addq $N,%rsp, N>0)
and/or pops of callee-saved integer registers only. Callee-saved regs
are never argument registers, so hoisting the pops cannot clobber an
outgoing argument of X, and the teardown restores rsp exactly to routine
entry, so the jump target is entered with ABI-mandated alignment
regardless of target_info.stackalign.
* only labels/directives may sit between the call and the teardown, so the
callee's result is returned unchanged (a genuine tail call). A result
routed through a callee-saved register (mov %rax,%rbx after the call) is
a separate result-forwarding problem and is deliberately not handled.
Soundness gates (CurrentProcAllowsSiblingTailFrameReuse): reject routines with
any address-taken local/parameter (a pointer into the released frame could be
an argument), routines whose frame is captured by a nested routine
(parentfpstruct assigned), assembler routines, implicit-finally / exception
frames (pi_uses_exceptions, pi_needs_implicit_finally, pi_has_implicit_finally)
and dynamic stack allocation (pi_has_stack_allocs).
Tests (unleashed/tests/testfiles/optsibcall): deep void and callee-saved-pop
mutual recursion that overflow the stack at -O2 but run in O(1) stack at -O4;
negative behavioural cases (pointer-to-local, try/finally, managed local) that
must keep the call and stay correct. Full suite green (minus known
pre-existing failures). Isolated micro-benchmark (same -O4, peephole off vs
on) on a deep mutual tail-call chain: ~22.9 s -> ~11.8 s (~1.9x).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ports gcc's -fthread-jumps / VRP-based jump threading to the FPC node
tree: when a dominating branch (or a value-range fact) already decides a
predicate, a nested if that re-tests it folds straight to the taken arm,
deleting the redundant compare/branch and the dead arm. Unlike
if-conversion or GVN-PRE this restructures control flow itself.
Two fact kinds, both strictly gated (a wrong-way implication is a
miscompile):
* comparison facts V <op> c (=,<>,<,<=,>,>=; c ordinal const; V a
simple non-aliased, non-address-taken, non-volatile, non-threadvar
local/value-param). A nested comparison on the same V and same
signedness is decided by integer implication over the unbounded
number line (jt_all_satisfy): S(fact) subset of Q -> true, disjoint
-> false, else undecided. Ignoring the type's domain bounds can only
lose folds, never create an unsound one.
* identical-predicate facts: any side-effect-free, call-free,
memory-free condition over constants, simple-var loads and pure
operators folds a tnode.isequal re-test.
A fact is asserted for a branch only when its variable(s) are provably
unassigned in that whole branch (jt_writes_var), so it stays invariant
throughout -- a re-test anywhere in the region, even inside a nested
loop, is decidable. Address-not-taken rules out by-ref/pointer writes,
so an ordinary call on the path cannot touch the variable and does not
block the fold. Facts inherited from an enclosing dominating if are
carried down unchanged, letting chained if/elsif ladders fold later
re-tests of earlier decisions.
Folding removes code (no duplication), so there is no code-growth budget
to manage. Wired as cs_opt_jumpthread / 'JUMPTHREAD', added to
genericlevel4optimizerswitches (-O4) and x86_64 supported switches; the
psub call refreshes DFA since the CFG changes; skips procedures with
labels.
Tests (unleashed/tests/testfiles/jumpthread): jt_exhaustive cross-checks
432 signed + 432 unsigned outer/inner comparison-pair combinations over
the full x domain against an unfoldable runtime-op reference (catches
any wrong-way implication); jt_gates proves an intervening assignment,
an address-taken store, a global and a threadvar all block the fold
while a call between does not; jt_evidence and jt_ladder use
%CHECKBIN_LACKS to show the dead arm's string literal disappears.
Full suite green (minus the 4 known pre-existing failures), including a
forced-on sweep across the whole corpus. Chained-decision microbench
~25-30% faster with threading on.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ies (-Oodeadstore) The dead-store switch (-Oodeadstore) gates a call to opttree.normalize() in psub, which only runs at -O3+ (inside the cs_opt_nodedfa block). normalize's searchblock walked a statement's whole expression subtree with a single fixed hoist target (the enclosing statement), but foreachnodestatic descends through nested statement lists too -- so a statement/if/match-expression block buried in a loop or if body was hoisted to BEFORE the loop. The block-expression was then evaluated once with a stale counter and its temp reused every iteration, so e.g. `for n... var s := if n=0 then 'zero' else 'other'` produced 'other' for n=0. This mis-hoist caused ~a dozen mode-unleashed miscompiles at -O3 -Oodeadstore (statement_expr/*, match, inline_vars inferred-array-string, array_size_shortcut). Fixes: - searchblock now STOPS at a nested statementn (does not hoist its block-expressions past its own enclosing statement); searchstatements grows a second phase that recurses into nested statement lists (loop/if/case/try bodies) and normalizes each with its own correct insertion point. - The "move the whole block out of the expression" branch replaced a statement slot with a raw block node and re-firstpassed a managed init/cleanup temp; for interface / function-reference / some generic-intrinsic block-expressions this dereferenced freed/mis-typed nodes and crashed the compiler (funcref_*, type_intrinsic_generic_arg_13). None of the constructs this pass must normalize need that branch (they yield a temp-ref or constant), so it now declares the proc not-normalizable (normalize returns false, DSE is skipped for it, the block-expression is left exactly as with the switch off) instead of crashing. - guard the post-phase-1 sibling walk against the slot no longer being a statementn. Result: at -O3 -Oodeadstore the suite has the same failure set as -O3 alone (816/22; the remainder are pre-existing -O3 loop bugs), so -Oodeadstore adds no failures and is now promotable. Adds regression tests under unleashed/tests/testfiles/dead_store_normalize/ (all fail on the old compiler). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…itives + case exhaustiveness)
A stage-2 self-compile at -O4 (our stage-1 compiler recompiling the compiler
with OPT=-O4) failed under -Sew (warnings-as-errors) on a series of
warning-as-errors, none caused by the new optimizer passes -- they are DFA
uninitialized-variable false positives and case-exhaustiveness warnings in
fork-added (and a couple of stock) routines that had simply never been
self-compiled at -O2+/-O4 (the standard build bootstraps with system FPC 3.2.2,
which does not run DFA / emit these). Fixes, all behaviour-preserving:
- defutil.walk_field_path_bits: accumulate the running bit offset in a plain
local instead of directly in the "out" parameter (bit_total). Reading an out
param from inside an inlined nested procedure tripped a DFA "not initialized"
false positive; the local is written back to bit_total on success.
- optloop (vectorize / jumpthread / bitidiom passes):
* initialize the recognizer-result locals (counter, ctype, avec/bvec/cvec,
scalarnode, scalarleft, vecop, vshape) that the nested vectorize_reason fills
-- per-proc DFA cannot see a nested routine define parent locals;
* default-initialize the managed jt_pure_cond out-record before the call;
* add explicit "else ;" to the bitidiom node-type case and the jt_relkind /
jt_all_satisfy comparison/kind cases (case-exhaustiveness warning).
- pexpr / pstatmnt indexed-label parsing: initialize the label index (arrlabidx /
labidx) that is only assigned on the ordinal-constant path but always passed to
get_or_create_indexed_labelsym (ignored there for string labels).
- dbgdwarf flexible-array-member DWARF: initialize count_size and delta, which
are set and used only under correlated assigned(flexcount) guards that DFA
cannot relate.
With these, `make -C compiler ppcx64 PP=<stage1> OPT=-O4` compiles and links the
whole compiler cleanly, so an -O4 self-compile can serve as a self-host smoke
test. Full unleashed suite unchanged (835/7, only the documented pre-existing
failures).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…uild for tests (task 273) Root-causes the three pre-existing bare-compiler unleashed failures (none are optimizer-related, none are /etc/fpc.cfg artifacts -- behaviour is identical with and without the installed config): - inline_vars_inferred_double_01 (FIXED): `var pi := 3.14` inferred the default best real type (Extended, SizeOf 10) instead of Double, so SizeOf(pi) <> SizeOf(Double). The array-literal inference path already normalizes float literals to Double (s64floattype) -- the scalar inline-var path did not. Promote a plain real-constant initializer of the default extended type to Double in inline_var_statement (guarded: not an explicit cast, only s80real/sc80real, only a realconstn), matching the array path and the documented intent. - multi_var_init_double_01 (documented, not a compiler bug): `x: Double = 3.14; if x <> 3.14`. The untyped real constant 3.14 is Extended, and comparing a Double variable against it is done at Extended precision, where Double(3.14) and Extended(3.14) differ (3.14 is inexact). The sibling multi_var_init_typed_const passes only because it uses 3.5, which is exact in both. This is stock FPC real-constant precision semantics; the test would need Double(3.14) or a tolerance. Not fixable without changing float comparison semantics globally. - inline_vars_inferred_array_string_widestring_01 (documented, not feasibly fixed): `[w, 'two', 'three']` with w: WideString. The array-constructor typecheck unifies the elements to their common string type (UnicodeString, since WideString loses to UnicodeString) and retypes the first element node in place *before* unleashed_infer_array_literal reads its "first element wins" kind, so the original WideString is unrecoverable from the post-typecheck tree (there is no typeconv wrapper to look through). The sibling unicode test passes only because UnicodeString is itself the unification winner. Fixing it would require capturing element types before the constructor unifies -- invasive and regression-prone for one niche case. Also documents in unleashed/tests/README.md the minimal fork-compiler build step for the `Rtti` unit (packages/rtl-objpas/src/inc/rtti.pp); with it on the unit path all three composable_records_rtti_flatten_* tests compile and pass (they were only failing on a missing unit path, not a compiler bug). Full suite 836/6 (was 835/7): inline_vars_inferred_double_01 now passes; the remaining failures are the two documented-above tests, the runner-artifact incfile test, and the three Rtti tests when run without the unit path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ports gcc's -ftree-loop-distribute-patterns to FPC: recognizes a counted,
ascending, unit-stride for-loop whose *entire* body is one store walking a
contiguous array region and lowers the whole loop to the RTL block primitive
the C runtime already tunes per target, instead of a per-element scalar store:
for i:=lo to hi do a[i]:=0; -> FillChar(a[lo], n*sz, 0)
for i:=lo to hi do a[i]:=v; -> FillChar/FillWord/FillDWord/FillQWord (sz 1/2/4/8)
for i:=lo to hi do a[i]:=b[i]; -> Move(b[lo], a[lo], n*sz)
n = hi-lo+1, guarded by if lo<=hi (a for-loop is a no-op when lo>hi and the
byte count must not come from a negative difference); the count is widened to
SizeInt so n*sz cannot wrap.
Shapes recognized:
* zero-fill of any unmanaged element (ordinal/float/pointer/enum/record) via
FillChar over the byte span (constant 0/nil/+0.0; -0.0 is rejected);
* non-zero fill of an ordinal/pointer element of size 1/2/4/8 with a constant
or loop-invariant value, reinterpreted to the matching unsigned width so the
identical bytes the scalar store would write are emitted;
* element-wise copy between two dynamic arrays -> Move. Two dynamic-array
references can only fully alias (share the block at offset 0, after a:=b) or
be disjoint -- a shifted overlap is impossible -- so Move (memmove-safe)
reproduces the forward copy in both cases with no runtime guard.
The destination base may be any side-effect-free, counter-independent l-value
(e.g. Self.FData), so field-backed dynamic arrays like neural-api's
TNNetVolume.FData lower too, not just plain variables.
Shapes declined (compile exactly as before): downto / non-unit step, multi-
statement bodies, non-counter (offset) indices, managed element types, non-zero
float fills (a float->int cast would round, not reinterpret), element-type-
changing copies (double[i]:=single[i]), copies whose source/dest are static
arrays or pointers (a shifted overlap cannot be excluded), -Cr/-Co checked code,
inline candidates, and procs with labels. The counter is required signed 32/64-
bit, simple and non-aliased, and proven unmodified in the body by DFA; the
lowered block leaves the counter at hi (the value an executed ascending for-loop
leaves) inside the lo<=hi guard, matching scalar semantics.
New switch cs_opt_loopdistpat (-OoLOOPDISTPAT), node-level pass in optloop.pas
(postorder, so nested loops lower first), call site in psub before strength
reduction and the for->while lowering, added to genericlevel4optimizerswitches
so plain -O4 enables it. Per-loop diagnostics 06067/06068 (-vn) mirror the
VECTORIZE notes.
Benchmark (fantastica/nn-bench-style volume clear+copy over `array of single`,
best-of results, identical `acc` checksum on/off proving equal output):
N (singles) clear -O4 off -> on copy -O4 off -> on
64 185 ms -> 18 ms ~10x 150 ms -> 26 ms ~5.8x
4096 595 ms -> 46 ms ~13x 529 ms -> 41 ms ~13x
65536 806 ms -> 87 ms ~9x 645 ms -> 79 ms ~8x
1048576 460 ms -> 80 ms ~5.7x 492 ms ->253 ms ~1.9x
Large copies converge toward memory-bandwidth bound; clears and cache-resident
sizes see the biggest wins from the hand-tuned RTL primitives replacing FPC's
scalar element store loop.
Tests: unleashed/tests/testfiles/optdistpat/ -- fill (zero + every unmanaged
width), non-zero FillChar/Word/DWord/QWord, copy (ordinal/float/record, self-
copy, shared-block a:=b), static arrays incl. partial-range and declined static
copy, field-backed TNNetVolume-shaped clear+copy, post-loop counter value
(executed hi and non-executed unchanged), the -O4 default set, and a -vn
negatives file asserting each decline path stays correct. Full unleashed suite
836->844 pass / 6 fail (the 6 are the documented pre-existing failures), stage-2
-O4 self-compile clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Port of gcc's -fpeel-loops (full-peel case) to the FPC node tree. When a
counted for-loop has a small compile-time-constant trip count, replace it
with that many straight-line copies of the body, folding the induction
variable to its per-iteration constant and deleting the counter, the loop
compare and the back-branch. Each peeled copy is then exposed to ordinary
constant folding/propagation (a[i] -> a[k] with k constant).
This is the deliberate -O4-only complement to the stock unroller
(unroll_loop in tfornode.pass_1): that heuristic already fully unrolls a
constant-count loop when the whole loop fits its unroll budget, but declines
the medium-body / low-trip kernels (3x3 convolution taps, per-channel Depth
1/3/4 loops) that sit inside hot outer loops where the loop control rivals
the body. LOOPPEEL picks those up: trip count <= 8, per-body weighted-node
cap 40, trip*body growth cap 160.
Soundness gates (a wrong peel is a miscompile):
* unit step, both bounds ordinal constants (exact trip count known);
* counter a simple non-aliased, non-address-taken, non-volatile local /
value param of ordinal type, DFA-proven unmodified in the body;
* body free of break/continue/goto/label/exit/raise;
* not TP/Mac mode, not an inline candidate, not -Os.
A statically empty loop (trip <= 0) is declined so the ordinary for-loop --
which leaves its counter unmodified when it never runs -- is what executes;
when the loop does run, the counter is left at its last taken value (the "to"
bound), matching normal for-loop post-conditions and keeping DFA consistent.
Wiring: cs_opt_looppeel switch (globtype.pas + ppudump table), added to
genericlevel4optimizerswitches; OptimizeLoopPeel in optloop.pas run in
psub.pas right after LOOPDISTPAT (still on for-nodes, before the for->while
lowering and strength reduction); notes cg_n_loop_peeled (06069) /
cg_n_loop_not_peeled (06070).
Tests (unleashed/tests/testfiles/optpeel, 6 files): ascending/descending
correctness across trip counts; counter-after value (last index when run,
untouched when statically empty); boundary trips (1, 8, 9-declined,
over-budget-declined); negatives that must decline (break, continue,
non-unit step, global counter) and stay correct; -O4-default firing; nested
peelable loops. Full suite 850 pass / 6 fail (the 6 known pre-existing);
-O4 stage-2 self-compile stays clean. Peeled kernel verified branch-free in
the emitted assembly (no back-branch or counter); representative hot-kernel
microbench ~3-7% faster (noisy).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… -O4
Port of gcc's -fsplit-loops to the FPC node tree. When the whole body of a
counted for-loop is a single conditional whose predicate compares the
induction variable against a loop-invariant bound m (if i<m then A else B and
its <=, >, >= / swapped-operand relatives), the truth value flips exactly once
as i sweeps the range, so the iteration space splits cleanly at the crossover:
for i := lo to hi do --> c := clamp(crossover, lo, hi+1);
if i < m then A else B for i := lo to c-1 do A; {branch-free}
for i := c to hi do B; {branch-free}
Both sub-loop bodies are branch-free: the per-iteration compare is gone, the
interior loop (the bulk of iterations -- e.g. the non-border rows of a padded
convolution) becomes a uniform kernel a later pass can vectorize, and only the
short border loop keeps the boundary work. Because the split runs before the
autovectorizer in psub, the exposed interior loop is offered to it directly.
Distinct from loop unswitching (whose condition is invariant in *both* operands
and never changes across iterations): here the IV side crosses the bound once.
Soundness gates (a wrong split is a miscompile):
* ascending unit-step loop; counter a simple non-aliased signed 32-bit local /
value param, DFA-proven unmodified in the body (32-bit so the widened int64
crossover math m+-1 / hi+1 cannot overflow);
* body is exactly one if whose condition is i <rel> m, rel in < <= > >=
(monotone in i; = and <> rejected), m an ordinal constant or a simple
non-aliased ordinal (<=32-bit) local / value param DFA-proven unmodified in
the body (evaluating it once == each iteration, cannot trap);
* neither branch contains break/continue/goto/label/exit/raise (a break in one
sub-loop would leave the other running);
* not TP/Mac, not inline candidate, not -Os, range/overflow checking off.
The crossover is clamped into [lo, hi+1] in an int64 domain and each sub-loop is
emitted under an `if lo<=c-1` / `if c<=hi` guard, which both skips a statically
empty sub-range and keeps the narrowing of c back to the 32-bit counter type in
range. The two loops tile [lo,hi] contiguously, so the counter is left with
exactly the value a single for-loop leaves (hi when it ran, unchanged if not).
Wiring: cs_opt_loopsplit switch (globtype.pas + ppudump table), added to
genericlevel4optimizerswitches; OptimizeLoopSplit in optloop.pas run in psub.pas
just before the vectorizer; notes cg_n_loop_split (06071) /
cg_n_loop_not_split (06072).
Tests (unleashed/tests/testfiles/optsplit, 5 files): exhaustive crossover sweep
(m below/inside/above the range, empty/short/long ranges) for all four relations
and the swapped operand order vs a scalar reference; counter-after value (hi when
run, untouched when empty); 8 declined shapes (=, <>, expression bound, fully
invariant condition, multi-statement body, break in a branch, downto, 64-bit
counter) each staying correct; -O4-default firing; the padded-stencil border
shape. Full suite 855 pass / 6 fail (the 6 known pre-existing); -O4 stage-2
self-compile stays clean. Interior loop verified branch-free in the emitted
assembly (the per-iteration i<m test is hoisted entirely into the prologue).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ports the gcc/LLVM loop-fusion transform to FPC (the inverse of the already
landed -OoLOOPDISTPAT distribution): two ADJACENT counted for-loops over the
identical iteration space are merged into one loop whose body is the two
original bodies concatenated, so an intermediate result A writes and B
immediately re-reads stays in registers/cache for one pass instead of being
streamed out to DRAM by the first loop and reloaded by the second:
for i:=lo to hi do A(i); -> for i:=lo to hi do begin A(i); B(i) end;
for i:=lo to hi do B(i);
This is exactly neural-api's consecutive element-wise passes over one
TNNetVolume -- a bias/scale add immediately followed by an activation, or a
gradient accumulate followed by a weight update -- each otherwise a separate
memory-bandwidth-bound loop. The fused element-wise loop is left as a single
for-node so the vectorizer (which runs right after) can still pack it.
DEPENDENCE SAFETY (a wrong fusion is a miscompile; the recognizer is strict and
anything unmatched compiles exactly as before). Fusion is legal iff no iteration
i of loop 2 needs a value loop 1 has not yet produced for that same index i.
This is guaranteed by one blunt, provably sufficient rule instead of a general
dependence test:
* EVERY array-element reference in both bodies, read or write, is indexed by
*exactly* the loop counter (unit stride, no a[i-1]/a[i+1] offset). Then
everything either body touches at "time i" lives at array index i, so after
the reorder body1(i) still runs before body2(i) and no iteration ever
reaches an element another iteration owns. This holds regardless of
aliasing: two dynamic arrays can only fully alias at offset 0 or be disjoint
(a shifted overlap is impossible -- the same fact LOOPDISTPAT used), and a
full alias with identical [i] indexing is still element-wise safe; static
arrays / fields likewise, since the index is the same i on both sides.
* The ONLY writes allowed are to such an a[counter] element (the write flag
sits on the vecn; the check is on the assignment *target* shape, not a
write/modify flag, because writing a[i] marks the array base -- e.g. a
field-backed dynamic array Self.FData[i]:=x -- nf_modify too, and that base
is a safe element access, not a scalar write). A scalar / field / pointer
write is the channel by which a value could cross the loop boundary, so it
is declined. Because no scalar or field is ever written by either body,
every scalar/field a body reads (and every variable the shared bounds
mention) is loop-invariant across the whole fused region.
* No calls, no pointer derefs, no nested loops, no break/continue/goto/label/
exit/raise, only plain (:=) assignments; -Cr/-Co checked code is declined
(fusion reorders the per-element checks). Inline intrinsics are declined
except a whitelist of pure single-arg arithmetic ones (abs/sqr/sqrt and the
min/max family) -- Min/Max is what FPC's if-conversion turns a
`if a[i]<0 then a[i]:=0` ReLU activation into (in_max_*), the very
activation-after-scale shape this targets.
* Both loops ascending, unit step, bounds structurally equal (tnode.isequal)
and free of calls/derefs. Counters may be the same variable or two distinct
simple non-aliased ordinal locals; in the latter case loop 2's body keeps
its counter c2 and the fused body prepends c2:=c1 each iteration, which both
makes loop 2 see c2=i and leaves c2 holding hi if the loop ran / unchanged
otherwise, matching a stand-alone for-loop's post-value with no separate
guarded fixup (same-counter fusions carry no such assignment and vectorize
unchanged). The fused loop leaves loop 1's counter at hi as the original
first loop did.
* The pass greedily folds a whole run L1;L2;L3;... into one loop in a single
visit (the survivor keeps loop 1's counter; its already-checked,
counter-preserving body is trusted on the re-fold so the synthesized node's
missing DFA info is not re-queried). Non-loop statements between two loops
(only nothing-nodes are skipped) block adjacency.
New switch cs_opt_loopfuse (-OoLOOPFUSE), node pass OptimizeLoopFuse in
optloop.pas firing on statement-list nodes, call site in psub before the
vectorizer / strength reduction / the for->while lowering, added to
genericlevel4optimizerswitches so plain -O4 enables it. Per-loop diagnostics
06073/06074 (-vn) mirror the sibling loop passes. ppudump switch-name table
updated.
Benchmark (best-of, `array of single`, identical checksum on/off proving equal
output; -O4 fused vs -O4 -OoNOLOOPFUSE):
scale+bias then ReLU, 4 MB, 60 reps : 113 ms -> 97 ms ~1.16x
accumulate then weight-update, 3x16 MB, 25 reps : 266 ms -> 234 ms ~1.14x
Memory traffic drops by one array round-trip per fused pair: the intermediate
that loop 1 stored and loop 2 reloaded (a[i], resp. the gradient g[i]) now stays
in a register -- for accumulate-then-update, 6 array streams through DRAM per
rep fall to 5 (~17%), matching the measured speedup; the rest is bandwidth
bound.
Tests: unleashed/tests/testfiles/optfuse/ -- element-wise producer/consumer
(same and different counters, boundary lengths 0/1/2), post-loop counter value
(hi when run, unchanged when empty, for both counter shapes), a >2-loop greedy
chain and a same-then-different-counter mix, the field-backed TNNetVolume
bias+ReLU shape, the -O4 default set, and a negatives file asserting each
decline path (different bound, cross-iteration a[i-1], downto, non-whitelisted
Trunc intrinsic, scalar carry, non-adjacent statement, nested loop) still
computes the same result. Full unleashed suite 855 -> 861 pass / 6 fail (the 6
are the documented pre-existing failures); stage-2 -O4 self-compile builds clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Completes the branch-predication -> vectorization pipeline for the element-wise
min/max activations neural-api runs over a TNNetVolume. FPC's existing -O2 if-
conversion (nflw.pas internalsimplify) already rewrites a branch-predicated
for i:=0 to high(a) do if a[i]<0 then a[i]:=0; { ReLU }
for i:=0 to high(a) do if a[i]>hi then a[i]:=hi; { one-sided clamp }
for i:=0 to high(a) do if a[i]<b[i] then a[i]:=b[i]; { element-wise max }
into a branch-free single-precision min/max intrinsic (a[i]:=max/min(...)), but
that scalar maxss/minss stayed a per-element operation the autovectorizer could
not widen (it only knew +,-,*). -OoIFCONVERT teaches the vectorizer's recognizer
the min/max activation shape and widens it across four SSE lanes to a packed
maxps/minps main loop plus a scalar tail, so the data-dependent per-element
branch disappears entirely and the loop streams at SIMD width. Under an AVX
fputype the VEX v-forms (vmaxps/vminps) are used.
CORRECTNESS (a wrong widening is a miscompile; the recognizer reuses the strict
OptimizeVectorize gate and anything unmatched compiles exactly as before):
* The body must be exactly one plain a[i] := max/min(u,v) assignment where the
destination is a single-precision dynamic-array element indexed by exactly
the loop counter, and the loop is an ascending unit-step counted loop over a
simple non-aliased signed 32/64-bit counter (all shared with the existing
autovectorizer checks; -Cr/-Co checked code and inline candidates declined).
* Only the single-precision intrinsics in_min_single/in_max_single are matched
(a double/integer min/max would change the per-lane precision). Each operand
u,v is independently either an array element of the counter (loaded as a
128-bit window) or a provably loop-invariant single scalar (broadcast once
into a [s,s,s,s] slot before the loop, as the arr_scalar shape already does).
* NaN / signed-zero fidelity: maxps/minps return their SECOND source operand
when a lane is unordered. The recognizer maps that second operand to opB --
the min/max node's parameter-list head, i.e. the same NaN-preferred value the
scalar maxss/minss uses -- and the packed body emits max/minps opB,opA so
every lane is bit-identical to the if-converted scalar result (verified for
NaN, +-0.0 and +-Inf on both the SSE and AVX paths). No reassociation, no
fast-math gate: this is a pure widening of an already-branch-free body.
New switch cs_opt_ifconvert (-OoIFCONVERT), added to genericlevel4optimizer-
switches so plain -O4 enables it; ppudump switch-name table updated. The pass is
driven by OptimizeVectorize (whose psub gate now fires on either -OoVECTORIZE or
-OoIFCONVERT): a new vok_minmax tvectoropnode kind (nbas.pas) carries the two
operand windows and an ismax flag, lowered by tx86vectoropnode.pass_generate_code
to MAXPS/MINPS (or the VMAXPS/VMINPS VEX forms). Per-loop diagnostic 06075/06076
(-vn) mirror the sibling loop passes.
Tests: unleashed/tests/testfiles/optifconvert/ -- ReLU / both one-sided clamps /
element-wise max checked bit-exact against a scalar recompute over every tail
residue 0..3 (SSE and AVX fputypes), NaN/signed-zero/Inf special-value fidelity,
the plain -O4 default set, and a negatives file asserting each decline path
(double precision, an open-array parameter, a two-statement two-sided clamp, a
downto loop, a shifted a[i+1] index) still computes the correct result. Full
unleashed suite 861 -> 866 pass / 6 fail (the 6 are the documented pre-existing
failures); stage-2 -O4 -Sew self-compile builds clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…th gated)
Ports gcc's -freassoc / LLVM's reduction reassociation to FPC for the sum /
dot-product reduction shape that dominates neural-api's inner products and L2
norms over a TNNetVolume:
for i:=lo to hi do acc := acc + expr(i); (sum)
for i:=lo to hi do acc := acc + a[i]*b[i]; (dot product)
A single serial accumulator is a loop-carried dependency: iteration i+1's add
cannot start until iteration i's has retired, so the loop runs at one FP-add
latency per element however wide the machine is. The pass splits acc into K=4
INDEPENDENT partial accumulators, each summing every fourth element, combined
after the loop:
s1:=0; s2:=0; s3:=0; i:=lo;
while i <= hi-3 do begin
acc:=acc+expr(i); s1:=s1+expr(i+1); s2:=s2+expr(i+2); s3:=s3+expr(i+3);
i:=i+4;
end;
acc := (acc+s1) + (s2+s3); { combine }
while i <= hi do begin acc:=acc+expr(i); i:=i+1 end; { scalar tail }
The four chains are independent, so four adds are in flight at once and the loop
becomes throughput- rather than latency-bound (~3-4x on a long single/double
reduction on a 3-4-cycle-latency, 1/cycle-throughput FP adder), and for exactly-
representable data the result is bit-identical to the serial loop.
CORRECTNESS. Reassociating the additions changes the grouping and hence the FP
rounding, so a FLOATING-POINT accumulator is split ONLY under fast-math
(cs_opt_fastmath), matching gcc's -fassociative-math requirement; the switch is
in genericlevel4optimizerswitches but its FP action is gated on fast-math being
active, exactly as the task requires. INTEGER accumulators use two's-complement
addition, which is exactly associative even on overflow, so they are always split
(-Co checked code is declined since a wrap would trap on a different add). The
recognizer is strict; anything unmatched compiles exactly as before:
* ascending unit-step counted loop over a simple non-aliased signed 32/64-bit
counter (so i+1..i+3 and hi-3 cannot wrap for any index the loop reached);
* body is exactly ONE plain acc:=acc+expr / acc:=expr+acc whose target acc is
a simple non-aliased, non-address-taken, non-volatile local/value-param FP or
integer scalar (the single-statement shape guarantees acc and the counter are
written nowhere else in the loop);
* expr is side-effect free and never mentions acc: no call, assignment, address-
of, nested loop, control transfer, write/modify of any location, or non-pure
inline (only abs/sqr/sqrt and the min/max family are allowed) -- so
duplicating expr four times with the counter shifted i+1..i+3 reads the same
values the serial loop would and adds no side effect (aliasing is irrelevant:
nothing but the local acc is stored);
* -Cr/-Co checked code and provably tiny constant-trip loops (< 2*K) are left
alone; procedures with labels are skipped at the psub call site.
The counter-shift copies substitute each plain read of i with (i+delta) and
typecheck the new node immediately: the surrounding expr is a copy of already-
typechecked body nodes that do_firstpass will not re-descend, but (i+delta) has
the identical type as the plain read it replaces, so the ancestors' cached
resultdefs stay valid. New switch cs_opt_reassoc (-OoREASSOC), node pass
OptimizeReassoc in optloop.pas firing on for-nodes, call site in psub after loop
peeling and BEFORE strength reduction (which would rewrite the a[i] index nodes
into pointer walks the counter-shift could no longer find) and the for->while
lowering. Per-loop diagnostics 06077/06078 (-vn) mirror the sibling loop passes;
ppudump switch-name table updated.
Tests: unleashed/tests/testfiles/optreassoc/ -- FP dot/sum and integer sum split
results checked bit-exact against an address-taken (hence declined, strictly
serial) reference over trip counts 0..40 and 997..1000 (all hi-3 residues); the
fast-math gate (integer still split / FP left serial under -OoNOFASTMATH, both
correct); a fast-math tolerance test (the split single-precision dot product
within a few ULP of an extended sequential reference); and a negatives file
asserting each decline path (a non-inline call in expr, a downto loop, a two-
statement body) still computes the correct result. Full unleashed suite 866 ->
870 pass / 6 fail (the 6 are the documented pre-existing failures); stage-2
-O4 -Sew self-compile builds clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ports gcc's -funroll-and-jam / LLVM's loop-unroll-and-jam to FPC for a perfect
(or near-perfect) two-level counted loop nest: the OUTER loop is unrolled by
factor K=4 and the four duplicated INNER loops are fused (jammed) into one inner
loop, so a value the inner body loads once (b[j] in a matmul-shaped nest
for i .. for j .. c[i]:=c[i]+a[i,j]*b[j]) is reused across the four unrolled
outer rows straight from a register instead of being reloaded every outer pass,
and a per-outer-iteration scalar accumulator is register-blocked:
for i:=lo to hi do --> i:=lo;
begin while i<=hi-3 do begin
s:=0; s0:=0; s1:=0; s2:=0; s3:=0;
for j:=lo2 to hi2 do for j:=lo2 to hi2 do begin
s:=s+a[i,j]*b[j]; s0:=s0+a[i ,j]*b[j]; { b[j] loaded }
c[i]:=s; s1:=s1+a[i+1,j]*b[j]; { b[j] reused }
end; s2:=s2+a[i+2,j]*b[j];
s3:=s3+a[i+3,j]*b[j]; end;
c[i]:=s0; c[i+1]:=s1; c[i+2]:=s2; c[i+3]:=s3;
i:=i+4;
end;
while i<=hi do begin { scalar remainder }
s:=0; for j:=lo2 to hi2 do s:=s+a[i,j]*b[j];
c[i]:=s; i:=i+1;
end;
This is exactly neural-api's register-blocked inner products / convolution nests
over a TNNetVolume (pletora/neural-api/neural/neuralvolume.pas). Unlike -OoREASSOC
the jam does NOT reassociate: each of the four accumulators sums exactly one outer
row in the original j order, so every produced value is BIT-IDENTICAL to the
untransformed nest for any input (no fast-math needed).
DEPENDENCE / LEGALITY (a wrong jam is a miscompile; the recognizer is strict and
anything not matched compiles exactly as before). Jamming the four inner loops is
the same reordering as fusing four copies of the inner body running on outer
indices i..i+3, legal iff those copies touch provably disjoint memory (no
loop-carried dependence across the outer loop). One blunt sufficient rule:
* The ONLY writes the outer body performs are (a) to a simple non-aliased LOCAL
SCALAR accumulator -- renamed to a fresh per-copy temp in copies 1..3 (the
register-blocking payoff) and required DEAD outside the whole nest (every
reference lies inside the outer for-node) so the rename cannot change any
observable value and a scalar that reduces ACROSS the outer loop is declined
-- or (b) to an ARRAY ELEMENT whose subscript chain mentions the outer counter
i, so copy k writes the i+k slice and the four stores are disjoint.
* The outer counter i may appear in the body ONLY as an exact array subscript
[i] (unit stride, no i+/-c offset, never in a scalar computation or the inner
bounds); enforced by counting that every read of i coincides with a bare [i]
subscript. Hence every i-touching access in copy k lives at i+k (disjoint
across copies), arrays not mentioning i are never written (read-only, shared
safely), and the inner iteration space is i-invariant so the jam is well-
defined.
* No calls, pointer derefs, address-of, non-pure inline intrinsics (only the
abs/sqr/sqrt and min/max whitelist), and EXACTLY ONE nested loop -- the inner
counted for, appearing as a direct statement of the outer body so the
prologue/epilogue split around it is unambiguous -- with no other loop; no
break/continue/goto/label/exit/raise/try; only plain (:=) assignments.
* -Cr/-Co checked code (global and per-region) is declined; both loops
ascending, unit step, over simple non-aliased signed 32/64-bit counters i<>j
(so i+1..i+3 and hi-3 cannot wrap for any index the loop reached). DFA proves
i unmodified in the body; procedures with labels are skipped at the psub call
site. Non-constant bounds are handled by snapshotting lo/hi once and emitting
the scalar-outer remainder; a provably tiny constant-trip outer loop (< K) is
left alone.
The counter-shift copies substitute each plain read of i with (i+delta) and each
accumulator read/write with its per-copy temp, typechecking every new node in the
substitution callback (the surrounding body is a copy of an already-typechecked
tree do_firstpass will not re-descend into, but each substituted node has the
identical type of what it replaced -- the REASSOC substitution gotcha).
New switch cs_opt_unrolljam (-OoUNROLLJAM), node pass OptimizeUnrollJam in
optloop.pas firing on for-nodes, call site in psub before the vectorizer /
strength reduction / the for->while lowering, added to
genericlevel4optimizerswitches so plain -O4 enables it. Per-loop diagnostics
06079/06080 (-vn) mirror the sibling loop passes; ppudump switch-name table
updated.
Tests: unleashed/tests/testfiles/optunrolljam/ -- a scalar-accumulator matmul row
and an array-element accumulator checked bit-exact against a strictly-serial
(descending-outer, hence declined) reference and an independent flat-loop absolute
reference across outer trip counts 0..8 (below factor, factor, every residue),
inner lengths 0..5, and larger sizes 200..205 hitting every main-loop residue; a
remainder file sweeping residues 0,1,2,3 with non-constant inner bounds; a
negatives file asserting each decline path (loop-carried outer dependence,
break in the inner body, offset a[i+1,j], address-taken accumulator, scalar
reduction across the outer loop) still computes the correct result; and an
-O4-default smoke test. Full unleashed suite unchanged at 861 pass / 19 fail --
the identical 19 (6 documented pre-existing plus runner-inaccuracy/environment
artifacts) fail on the pristine baseline with the same runner, so zero regression.
Stage-2 -O4 -Sew self-compile builds clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ports gcc's -fpredictive-commoning to FPC. In a counted loop that re-reads
the same array location across successive iterations through a small window of
constant offsets -- the classic stencil a[i]:=b[i-1]+b[i]+b[i+1] and 1-D
convolution inner loops sliding a window over an array -- the value b[i+c]
loaded at iteration i is exactly b[i+c-1] as seen at iteration i+1. Instead of
re-loading every offset each iteration, the window is carried in a rotating set
of scalar temporaries and only the LEADING EDGE b[i+maxoff] is loaded per
iteration:
if lo<=hi then begin t0:=b[lo-1]; t1:=b[lo]; end; { guarded preheader }
for i:=lo to hi do begin
t2 := b[i+1]; { leading edge -- the only load }
a[i] := t0 + t1 + t2; { window reads served from temps }
t0:=t1; t1:=t2; { rotate down for the next iteration }
end;
This is cross-*sequential*-iteration reuse in a single loop, which LICM
(invariant loads only), same-iteration CSE and unroll-and-jam (cross-*outer*-
iteration reuse) each miss. Measured on the stencil above, per-iteration
memory reads of b drop from 3 to 1 (the two trailing offsets come from
registers). No reassociation: each produced value is BIT-IDENTICAL to the
re-loaded one (FP included -- values are only carried), so the transform is
exact for any input.
LEGALITY (a wrong commoning is a miscompile; the recognizer is strict and
anything not matched compiles exactly as before):
* The reuse base B is a plain 1-D dynamic or normal (non-packed) static array
VARIABLE (a direct load, so field/pointer-backed arrays are declined) with
an unmanaged 8/16/32/64-bit integer or Single/Double element, read as
B[i+c] for constant offsets |c|<=16.
* B must be READ-ONLY across the loop: declined if any store target resolves
(through the vec chain) to B's variable. Disjointness from the stores is
the fork's trusted array-identity disambiguation that LOOPFUSE / LOOPDISTPAT
/ UNROLLJAM already rely on -- a store to a different array variable A
(A<>B by symbol) is assumed not to alias B. An in-place stencil that stores
into B itself is thus declined (its store target resolves to B), sidestepping
the stale/fresh-value hazard entirely.
* The commoned offsets must form a CONTIGUOUS window [minoff..maxoff] (every
integer between appears as a B read) of width 2..8. Contiguity makes the
preheader safe: it loads B[lo+minoff .. lo+maxoff-1] and the loop's leading
edge loads B[i+maxoff]; every index either touches is one the ORIGINAL first
iteration i=lo (which reads the whole contiguous window) already read, and
B[i+maxoff] is read by the original iteration i, so NO load is introduced
that the original did not perform. The if lo<=hi guard suppresses the
preheader on an empty / too-short loop, so a zero-trip loop performs no
speculative or out-of-bounds load (a hard requirement under -Cr and at the
edges of an array). All body reads are unconditional (no if/case/short-
circuit and/or), so the whole window is genuinely read every iteration.
* The loop is ascending, unit step, over a simple non-aliased signed 32/64-bit
counter not modified in the body (DFA); -Cr/-Co, downto and non-unit steps
are declined. Bounds are snapshotted into temporaries evaluated once, and
the transformed loop stays a real for-node so the counter's post-value is
exactly what the original left.
* The body is built only from a whitelist of side-effect-free constructs --
plain (:=) assignments to an array element or a simple scalar, reads,
constants and pure arithmetic -- with NO call, pointer deref, address-of,
as-cast, nested loop, if/case, break/continue/goto/label/exit/raise/try,
short-circuit and/or, or non-pure inline intrinsic (only the abs/sqr/sqrt
and min/max whitelist). Procedures with labels are skipped at the psub call
site like the sibling loop passes.
The reload-to-temp substitution typechecks each new tempref immediately (the
surrounding body is a copy of an already-typechecked tree do_firstpass will not
re-descend into, and the tempref has the identical element type of the vecn it
replaces -- the REASSOC substitution gotcha).
New switch cs_opt_predcom (-OoPREDCOM), node pass OptimizePredCom in optloop.pas
firing on for-nodes, call site in psub before the vectorizer / strength
reduction / the for->while lowering, added to genericlevel4optimizerswitches so
plain -O4 enables it. Per-loop diagnostics 06081/06082 (-vn) mirror the sibling
loop passes; ppudump switch-name table updated.
Tests: unleashed/tests/testfiles/optpredcom/ -- a 3-wide double stencil, a
2-wide int32 sliding window and a 3-wide Single stencil checked element-wise
bit-exact against an independent reference across trip counts 0,1,2,3 (window
boundary and the guard) and sizes 200..205 hitting every residue; a negatives
file asserting each decline path (in-place store into the base, read+write same
base, call in body, conditional in body, downto, address-taken counter) still
computes the correct result; a -Cr/-Co file (pass declines, no new range error
at the window edges or trip count 0/1); and an -O4-default 5-wide stencil smoke
test. Full unleashed suite 874 -> 878 pass / 6 fail (the identical 6 pre-
existing failures on the pristine baseline, zero regression). The -O4 -Sew
stage-2 self-compile builds clean (the resulting -O4-self-compiled binary is
non-functional on pristine HEAD too, an unrelated pre-existing condition).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ports gcc's -ftree-sra to FPC. A local variable of record type whose address never escapes is split field-by-field into independent synthesized scalar temporaries (one tlocalvarsym per field, of the field's type), and every rec.field access is rewritten to a plain load of the matching temp. The fields then live in registers and feed the existing data-flow passes (constant propagation, DFA, dead-store elimination) instead of round-tripping through the stack frame -- the motivating pattern being small geometry/stride records and accumulator aggregates threaded through hot code. New switch -OoSRA (cs_opt_sra), on by default at -O4. Runs first in tcgprocinfo.TransformNodeTree, before constant propagation / DFA, so those passes see the scalar temps. Skipped for routines with inline assembler. The transform is gated by a conservative single-walk escape analysis (foreachnodestatic) per candidate record: a record is scalar-replaced only when EVERY load of it is the direct base of a field subscript. That one invariant, plus an address-of rule and a by-reference-argument rule, declines all the unsafe shapes: * address taken of the record or any field (@rec, @rec.field) * record or a field passed as var/out/constref * whole-record assignment (rec := other), function-result stores * whole record passed by value / as a method Self, typecasts of the record * FillChar/Default zeroing (passes @rec) and untyped move/fillchar * a `with rec do` that the front-end could not resolve to plain subscripts (a simple local-record `with` IS resolved and is safely handled) Structural gates decline the record type up front: unions, variant/case parts, `absolute`/overlapping-offset fields, packed/bitpacked records, and any record carrying a non-scalar or managed field (v1 handles flat records whose fields are all unmanaged register-sized ordinal/enum/float/pointer types -- ref-counting is never at risk because managed fields are declined). Assignment-target and read-modify-write markers (nf_write/nf_modify) are carried from the replaced subscript onto the fresh load so DFA still sees a definition there; the new load is typecheck+firstpassed in the substitution (the surrounding tree is already lowered, cf. the REASSOC nil-resultdef gotcha). Tests in unleashed/tests/testfiles/optsra/: sra_correct_01 (integer/float/pointer fields, read-modify-write, loop-threaded records -- results checked against an independent reference), sra_negatives_01 (every escape gate -- behavioural equality with the record left in memory), sra_constprop_01 (SRA enabling constant propagation and dead-store elimination down to a constant return). Full unleashed suite: identical PASS=878 FAIL=6 (the documented pre-existing failures) with and without the change; stage-2 -O4 -Sew self-compile builds clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ports gcc's -fstore-merging to FPC. A straight-line run of adjacent narrow
constant stores to consecutive addresses off the SAME base register -- the
shape left behind by record field-by-field initialisation and small constant
fills that the code generator lowers to per-field "mov $const,off(base)" -- is
coalesced into one wider, naturally-aligned store whose immediate is the
little-endian composition of the component constants. Two adjacent byte stores
become one 16-bit store, two 16-bit stores one 32-bit store, two 32-bit stores
one 64-bit store, etc.
New switch -OoSTOREMERGE (cs_opt_storemerge), on by default at -O4.
Approach: an x86-64 assembler-list peephole (TX86AsmOptimizer.TryStoreMerge,
called from PostPeepholeOptMov). The peephole level is the pragmatic fit for
FPC here: at this point taicpu MOV records carry fully-resolved base+offset
reference operands and known constant sources, so the "provably consecutive
addresses off the same base" and "constants compose little-endian" tests are
direct, and no earlier tree pass sees the per-field stores as one object. It
also catches stores the front-end leaves unmerged (e.g. out-of-order field
initialisation) that a node-tree pass keyed on a single record variable would
not.
Legality (all gates are structural and local):
* The run is collected forward from p via GetNextInstruction (which skips only
reg-alloc / line-info markers). Collection stops at the first instruction
that is not itself a qualifying constant store to the same base register, so
a call, a label/jump, a memory read, a base-register overwrite, a volatile
ref or any other instruction between two stores terminates the run and those
stores are never merged together. Consequence: between the instructions
that actually get merged there is provably nothing but const-stores-to-base,
and such a store neither reads memory nor writes its own base register.
* A qualifying store is "mov $const, off(base)" with a plain base+offset ref:
real base register, index = NR_NO, no symbol/relsymbol, no segment override,
refaddr = addr_no, empty volatility set. Symbol-bearing refs (rip-relative
globals) are declined, so only register-based (stack / frame / pointer)
stores are touched.
* Any pair of stores in the run that overlaps at all aborts the whole attempt
(overlapping stores are order-sensitive); every surviving store therefore
writes a distinct byte range, so composing them and placing the one wide
store at p's slot preserves the final memory contents.
* A merged group must exactly tile an aligned window [w0, w0+tw), tw in
{2,4,8}, w0 the natural-alignment boundary at or below p's offset (floored,
handling negative %rbp-relative offsets), fully covered by >= 2 component
stores. Anchoring the window on p guarantees p is a component, so p is
widened in place (its offset lowered to w0 when p is the high half) and the
other components are removed.
x86-64 has no move of a full 64-bit immediate to memory, so a 64-bit merge is
emitted only when the composed value fits a sign-extended imm32; otherwise the
next smaller width is tried, so a general 64-bit constant group coalesces into
two 32-bit stores rather than four 16-bit ones -- still a win, same as gcc.
Unaligned stores are cheap on x86-64 but the natural-alignment window is kept as
a mild quality gate.
Tests in unleashed/tests/testfiles/optstoremerge/: correct_01 (out-of-order
byte inits coalesce to a 32-bit store whose imm bytes spell a tag asserted with
%CHECKBIN_HAS, plus a 64-bit merge; every byte read back individually),
disabled_01 (same source at -O4 -OoNOSTOREMERGE -- the plain code generator does
NOT merge this out-of-order shape, so %CHECKBIN_LACKS the tag; proves the
peephole, not codegen, is responsible), endian_01 (little-endian composition of
byte->word and word->dword merges verified byte by byte), negatives_01 (call
between stores, aliasing pointer write between stores, overlapping stores,
non-adjacent offsets -- all declined and behaviourally correct).
Full unleashed suite identical PASS/FAIL set with and without the change (the 6
documented pre-existing failures only); RTL rebuild clean; stage-2 -O4 -Sew
self-compile builds clean (the flow analyser's spurious "items may be
uninitialised" hint on the run buffer is locally silenced with {$warn 5036 off}
so -Sew stays clean).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Port gcc/tree-switch-conversion.cc's jump-table / bit-test *clustering* to
FPC's case-node codegen. Where tcgcasenode previously picked ONE whole-
statement strategy (genjumptable / genlinearlist / genjmptree), the new
tcgcasenode.gencaseclusters partitions the sorted labels by dynamic
programming into an optimal mix of clusters and dispatches between them with
a balanced binary comparison tree:
- jump-table cluster: a dense run (>=4 labels, >=40% density, span capped)
lowered through the existing target genjumptable (holes -> else);
- bit-test cluster: several labels within an ALU-word span sharing at most
3 target blocks, lowered to one unsigned range check + "1 shl (x-low)" +
an AND-mask membership test per target (the classic
"case c of 'a','e','i','o','u'" shape becomes a shift+AND, not 5 compares);
- plain-compare singletons for the leftovers.
Wired into pass_generate_code ahead of the classic single-strategy
selection (gated on cs_opt_casecluster, only on the ordinal non-64-bit-alu
path where hregister/opsize/elselabel/jmp_lt/jmp_le/jumptable_no_range are
already set up); returns false and falls back to the tuned default lowering
when nothing beats one compare per label or only a lone jump table results.
The generic implementation uses only hlcg/generic helpers so it stays correct
on every target sharing this code path.
cs_opt_casecluster ('CASECLUSTER') added to globtype + ppudump and to
genericlevel4optimizerswitches, so plain -O4 enables it; -OoNOCASECLUSTER
disables. Constant-arm-to-static-array switch-conversion (gcc feature (c))
is left as a follow-up.
Robustness fixes over the initial draft: the DP span computation now bails on
TConstExprInt overflow (labels straddling the type range no longer wrap past
the width guards and mint an absurd table/mask); jump-table clusters run
genjumptable on a private copy of the selector register so its in-place
SUB does not clobber the value other dispatch-tree paths still read; the
per-cluster jumptable_no_range is saved/restored and only omits the bounds
check when the cluster covers the whole selector-type range.
Tests (unleashed/tests/testfiles/optcasecluster): exhaustive semantic
correctness across cluster boundaries, holes and type bounds at -O4;
a bit-test whose composed mask spells an ASCII tag asserted present via
%CHECKBIN_HAS and absent under -OoNOCASECLUSTER (%CHECKBIN_LACKS); signed
selectors with negative labels/ranges; -Cr/-Co range-checked shapes; and
negative cases (few labels, enum/boolean/string) that must stay correct.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ports gcc's -fcrossjumping to FPC. When two or more predecessor blocks end
in an identical instruction sequence and converge on the same successor (the
if/else branches that share a trailing tail, case arms ending with the same
cleanup, and the per-arm Exit(TError...) boilerplate this repo's TResult style
produces in every codec), one copy of the shared tail is kept and the other
predecessors are redirected to jump into it. Purely a code-size / I-cache win.
New switch -OoCROSSJUMP (cs_opt_crossjump), on by default at -O4.
Approach: a late, whole-block assembler-list pass in the x86 optimiser
(TX86AsmOptimizer.DoCrossJump, run from an override of PostPeepHoleOpts after
the ordinary post-peephole). At this point taicpu records carry fully
resolved, register-allocated operands, so equality is operand-exact and the
transform has zero semantic risk: the merged instruction stream that executes
is byte-for-byte the one that executed before, only its storage is shared, so
calls, RIP-relative loads and everything else are handled with no special
casing precisely because they must match textually to be merged.
Algorithm:
* Collect the block's "anchors" -- unconditional jumps to a label and RET.
* Group anchors that are operand-exact equal (InstructionsEqualExact compares
opcode, size, condition, segment prefix and every operand; references via
RefsEqual, which also rejects any volatility). The earliest anchor of each
class is the canonical copy; each later identical anchor is a candidate.
* For a candidate, walk backwards from both anchors in lock-step
(GetLastInstruction) counting identical trailing instructions. The walk
stops at the first mismatch, at any referenced label (GetLastInstruction
returns those as a boundary, so a merged tail can never contain a side-entry
label), at any interior unconditional transfer, and via an overlap guard
that keeps the deleted region strictly after the kept region.
* If at least 3 trailing instructions match (a single shared instruction is
not worth a jump), a fresh label is inserted before the kept tail, a
"jmp <newlabel>" replaces the duplicated tail, and the duplicated tail is
removed with correct tasmlabel ref-counting (each removed jump decrefs its
target, the new jump increfs the new label).
Correctness guards beyond operand-exact equality:
* The duplicated region is refused (RegionSafeToDelete) if it physically
contains anything whose removal could corrupt debug or unwind information --
any label (a zero-jump-refcount label can still be referenced by DWARF line
tables, CFI/EH frame data or exception tables), any CFI directive, variable
location note, alignment, symbol or constant. Only instructions and
disposable bookkeeping (reg/temp-alloc, comments, line markers) may be
dropped. This is what makes epilogue tails carrying .cfi directives, and
tails bracketed by eh_frame .Lc labels, safe by declining to merge them.
* Merge artefacts (the inserted label and redirect jump) are themselves walk
boundaries, so repeated merges compose soundly.
Tests in unleashed/tests/testfiles/optcrossjump/: correct_01 (multi-exit shared
tails, every path validated against a reference at -O4), disabled_01 (same
source at -O4 -OoNOCROSSJUMP, equally correct), exceptions_01 (managed
ansistring tails inside try/finally with a raising arm -- ref counting and the
exception frame survive the merge), casecleanup_01 (case arms with a common
cleanup epilogue), negatives_01 (a one-operand difference and a live goto label
must not be merged across).
Full unleashed suite: 896 PASS / 6 FAIL, the 6 being the documented
pre-existing failures (3x composable_records_rtti_flatten needing the unbuilt
Rtti package, multi_var_init_double_01, inline_vars_inferred_array_string_
widestring_01) plus incfile_length_matches_size_02, which is a shell-runner
cwd artifact (it opens its own source by relative path; passes from its own
directory) and compiles at -O0 where the pass never runs. Code-size at -O4 on
the repo's own CLIs (only the app code is recompiled; RTL is prebuilt without
the pass): calc .text 360048 -> 360016 (-32 B), sed .text 339024 -> 338976
(-48 B); both binaries produce identical output with and without -OoNOCROSSJUMP.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ports gcc's -freorder-blocks cold-region sinking to FPC. Lays out each
routine so the straight-line fall-through follows the likely (hot) edge and
sinks cold error/raise regions out of the hot path to the end of the routine.
A cold region is the code guarded by a conditional jump that jumps over it to
the hot successor and whose body unconditionally calls a runtime error/raise
helper (fpc_raiseexception, fpc_reraise, fpc_handleerror, assert, RunError,
range/overflow/divbyzero/invalidcast/object-check helpers). This is exactly
the shape this repo's TResult/raise/Assert style emits in every codec's guard
clauses. Sinking it makes the hot path branch-not-taken and packs the cold
raise boilerplate away from the I-cache-hot straight line.
New switch -OoBLOCKORDER (cs_opt_blockorder), on by default at -O4.
Approach: a late, whole-block assembler-list pass in the x86 optimiser
(TX86AsmOptimizer.DoBlockOrder, run from the PostPeepHoleOpts override right
after DoCrossJump). At this point taicpu operands are fully resolved and
register-allocated, so the relocation is a pure placement change: the same
instruction stream still executes, only its address changes.
Algorithm per routine:
* FindAnchor locates the last top-level unconditional control transfer (RET,
unconditional JMP, or a provably-noreturn helper call). Sunk regions are
spliced in immediately after it -- still inside the function's CFI/size
bracket, but unreachable by fall-through.
* For each conditional guard jump p to a label L, TrySinkAt walks the region
[p.Next .. L) requiring it to be straight-line (no interior branch/RET; a
CALL is allowed, it is how the raise fires), to actually contain an error
helper call, and to close exactly on L. Interior labels are permitted only
if alt_jump (plain jump targets that travel with the region and still reach
the one preserved exit); CFI/debug-range labels, alignment, symbols and
constants force a refusal so unwind/line tables are never disturbed.
* The region is relocated after a fresh out-of-line label, the guard is
inverted and retargeted at it (hot successor becomes the fall-through), and
a rejoin "jmp L" is appended only when the terminal helper can return
(assert); a provably-noreturn raise/error region falls off the end with no
rejoin. tasmlabel ref-counts are maintained (old target decref, new label
and any rejoin increfs).
* Bounded by MaxRegionLen / MaxTransforms and gated on CanDoJumpOpts so
pathological input can't blow up; rescans from the top after each transform
since the list and anchor change.
Tests in unleashed/tests/testfiles/optblockorder/: correct_01 (guard-clause
raises, a loop body holding a guarded arm, two independent cold arms in one
routine, a returning assert arm that keeps its rejoin jump, and nested
try/finally over a managed type with a raising arm -- every hot path validated
and every sunk arm still raises the caught exception), disabled_01 (same source
at -O4 -OoNOBLOCKORDER, byte-for-byte identical behaviour -- the control that
proves only placement changes), exceptions_01 (distinct exception classes sunk
from cold guards, matching handler and message must survive, finally runs
exactly once), negatives_01 (branchy routines with no cold region -- the pass
must find nothing to sink and change nothing).
Full unleashed suite: 900 PASS / 6 FAIL, the 6 being the documented
pre-existing failures (3x composable_records_rtti_flatten needing the unbuilt
Rtti package, multi_var_init_double_01, inline_vars_inferred_array_string_
widestring_01, and incfile_length_matches_size_02, a shell-runner cwd artifact
that compiles at -O0 where the pass never runs). Verified on parseDigit: with
the pass the guard becomes "ja .Lcold" falling through to the hot result and
ret, with the fpc_raiseexception region sunk past the ret; with -OoNOBLOCKORDER
the raise stays inline mid-routine. Both builds pass every test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… prefetch Two cooperating opt-in loop passes (OptimizeUnrollPrefetch in optloop.pas, run from psub after the vectorizer and before strength reduction) for the long contiguous array-walk loops the stock -OoLOOPUNROLL never fires on, since it only fully unrolls small compile-time constant trip counts. -OoUNROLLDYN unrolls a counted for-loop of UNKNOWN trip count by 4 -- four back-to-back (body; inc) copies guarded by i<=hi-3 with a scalar remainder while-loop for the tail. The copies keep the original serial evaluation order, so a floating-point accumulation chain stays a single serial accumulator and the result is bit-identical to the scalar loop (no reassociation). -OoPREFETCH inserts a PREFETCHNTA of base[i+64] once per iteration group for each distinct streamed dynamic-array base (skipping pure store destinations). Semantically a no-op: an x86 prefetch of an out-of-range address never faults. Sound subset (correctness over coverage): ascending unit-step for-loop; a simple non-aliased signed 32/64-bit counter DFA proves is never modified in the body; straight-line body (no calls, control flow, nested loops, try/raise/asm); every array access is a 1-D array indexed by exactly the bare counter; -Cr/-Co disables it; inline-candidate procs and procs with labels are skipped. Opt-in; NOT part of the -O4 defaults. Adds the two switches to globtype.pas and ppudump.pp, three bit-exact/order semantics tests under unleashed/tests/testfiles/optunrolldyn/, and a codegen + semantics check script unleashed/tests/unrolldyn_check.sh. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The initial -OoPARTIALINLINE landing (99fe9f1) split only PROCEDURES and deferred functions: when the tiny inline header was inlined at a call site, the guard arm's exit(value) and the tail's result assignment could land in different result locations (the funcret temp vs. the return register), so one arm returned the wrong value. Root cause: after typecheck an "exit(value)" is lowered by the exit node's pass_typecheck into "$result := value; exit", whose assignment targets the routine's own funcret local. When the leading guard is deep-copied into the header, that assignment still pointed at the *body's* funcret local (parameter remapping only touched paravarsyms), while the header had no funcret local of its own at all (getcopyas / create_procdef_alias does not copy the body's localst). The two arms of the header therefore wrote different locations. Function-safe header shape: * proc_eligible now accepts potype_function as well as potype_procedure, still gated to simple register-returned result types (simple_split_type) and additionally skipping any result returned via a hidden pointer parameter (paramanager.ret_in_param) so the header never has to forward the caller's result pointer. * partialinline_make_header gives the header its own funcret local via insert_funcret_local(headerpd) for non-void routines. * remap_load additionally re-points the guard's "$result := value" assignment from the body's funcret local onto the header's funcret local (tracked in tremap.old/newfuncret). * the forwarding tail is built as "$result := Body(params)" (function) instead of a bare "Body(params)" (procedure), writing the *same* header funcret local as the remapped guard arm. Because both arms now assign the identical header funcret local, the inliner's funcret substitution (createlocaltemps mapping vo_is_funcret -> funcretnode) routes both writes to the single funcretnode the caller materialised: no temp-vs-register divergence. Procedures are unaffected (funcretsym is nil, so the new remap/tail branches are skipped). Tests (unleashed/tests/testfiles/optpartialinline): * optpartialinline_func_semantics_01: both guard and tail arms for integer, floating-point, enum and pointer results, at compare-use, direct-assignment (funcret-reuse) and expression-nested call sites; switch ON. * optpartialinline_func_disabled_01: identical assertions with the pass OFF (baseline), pinning that enabling the switch does not change semantics. * optpartialinline_func_checkbin_01 / _off_01: a guard-true constant call site whose outlined function body (and its cold-only string constant) is smart-linked away only when split (%CHECKBIN_LACKS) vs. a NOPARTIALINLINE control asserting it survives (%CHECKBIN_HAS). Verified: full unleashed suite (973 tests) has the same pass/fail set as baseline both with per-file options and with -OoPARTIALINLINE forced globally (966 pass, 0 regressions, 7 pre-existing known failures); unrolldyn_check.sh and slp_codegen_check.sh still pass. Disabling the new funcret remap reproduces the documented miscompile on the function tests, confirming the fix is load-bearing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…oaden pure eligibility to const-aggregate readers Two follow-ups to the landed -OoPURE interprocedural pure/const discovery: * -OoGVNPRE now value-numbers calls to routines proven PURE by -OoPURE. A direct, non-method call to a pure routine (reads but never writes global state, no I/O, cannot raise/trap) whose arguments are side-effect-free is treated as a memory-reading GVN candidate, keyed on its argument's safe locals. Two structurally-identical such calls with the arguments and the pure-read memory unchanged in between are commoned to a single call reused from a temp. A pure call is no longer a straight-line barrier (new gvn_has_sideeffects), so `x := f(a)` participates; its arguments are still scanned so a side effect hiding in one re-arms the barrier. Kill semantics ride the existing gvn_mem machinery: any store to memory, an impure call, or a loop that writes memory invalidates the value. Gated on -OoPURE; inert when the pure pass did not run. * -OoPURE eligibility broadened: a routine that reads through a const/constref aggregate parameter (unmanaged record / fixed array / set) but never writes it is now a PURE reader -- its field/element reads are pure memory reads and a const parameter cannot be written. Result type stays a simple scalar and methods (a mutable self alias) remain out of scope, keeping the gates conservative. These are pure-not-const, so they feed the new GVN-PRE CSE (LICM, which needs const, is unaffected). Tests: optgvnpre_purecall_01 (pure-call CSE correctness oracle: full/partial redundancy plus kill-on-arg-write and kill-on-global-store soundness), optgvnpre_purecall_disabled_01 (disabled-switch parity), optpure_constaggr_01 (const record/array reader eligibility, memory-kill soundness, var-aggregate stays impure). Full unleashed suite green (968 pass / 8 known-skip / 0 fail); unrolldyn_check.sh and slp_codegen_check.sh still pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… bodies via jump thunks Ports the gcc -fipa-icf / gold --icf idea to FPC as an opt-in -OoICF pass that runs intra-unit at the assembler-list level, after every routine of a module has been generated into al_procedures and before the object file is emitted (hook in create_objectfile, pmodules.pas). For each routine it canonicalizes the body (opcode/opsize/condition, every operand encoding, memory references with base/index/scale/segment/offset/ refaddr) with the routine's own symbol and local labels symbolized to positional tokens and other relocations abstracted to the referenced symbol name. Routines with equal canonical signatures emit byte-identical machine code, so the first is kept and each later duplicate has its instruction body replaced by a single `jmp <kept>` thunk. The fold is always a thunk, never a symbol alias: every duplicate keeps its own distinct symbol and therefore its own distinct address, so @f<>@g survives even when a folded routine's address is taken and compared, and FPC's non-DWARF exception model is unaffected. All labels are preserved so the already-generated debug/CFI list never dangles. Conservative: only straight-line bodies (no embedded data/cfi/unhandled operand kinds) fold, and only when the routine has enough instructions that a 5-byte thunk is a net shrink. x86 only; a no-op elsewhere. Opt-in; NOT part of the -O4 defaults. New cs_opt_icf switch (globtype.pas) + 'ICF' name + ppudump table entry. Tests under unleashed/tests/testfiles/opticf: two identical routines fold (a runtime probe asserts exactly one begins with the 0xE9 jmp opcode) while a near-identical routine does not; addresses stay pairwise distinct; results match ICF on vs off; and a generics-instantiation case. Full suite green (972 pass, 7 pre-existing ignores) both normally and with -OoICF forced on globally; unrolldyn_check.sh and slp_codegen_check.sh still pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…cal dynamic arrays
Intraprocedural escape analysis that proves a local dynamic-array variable
never outlives its routine and replaces its heap buffer with a frame-local
stack buffer, eliding the fpc_getmem/fpc_dynarray_setlength call. This is the
HotSpot / gcc -fipa-pta stack-allocation idea, the safe dynamic-array subset.
A local dynamic-array var A of a NON-managed element type qualifies when its
ONLY SetLength is SetLength(A,N) with N a positive compile-time constant and
header+payload within a 4 KiB frame budget, and A provably never escapes:
it appears only as A[i], as the operand of Length/High/Low or of that one
SetLength, is never address-taken, captured by a nested routine, assigned
to/from another location, or passed whole to a callee. Recursive routines,
routines using exceptions and assembler routines are disqualified.
The rewrite runs BEFORE do_firstpass (next to -OoREFELIDE), while SetLength is
still an in_setlength_x inline node and A[i]/Length are plain nodes. It
allocates a hidden local record record refcount,high:sizeint; data:array[
0..N-1] of elem end and lowers the SetLength (guarded by if A=nil so loop
re-execution stays the RTL's SetLength-to-same-length no-op) to:
FillChar(R,sizeof(R),0); R.refcount:=-1; R.high:=N-1;
PPtrUInt(@A)^ := PtrUInt(@R.data);
refcount:=-1 is exactly FPC's own constant/read-only dynarray header sentinel
(aasmcnst begin_dynarray_const), so the scope-end finalization the compiler
still emits for A (fpc_dynarray_decr_ref) neither frees nor finalizes it, and
fpc_dynarray_incr_ref treats it as constant. The stack buffer has precisely
the heap buffer's lifetime, so behaviour is bit-identical.
Follows the established fork pattern: cs_opt_stackalloc in globtype.pas (+
OptimizerSwitchStr and ppudump table), OptimizeStackAlloc in optloop.pas,
call site in psub.pas. Opt-in (-OoSTACKALLOC); NOT part of the -O4 defaults.
Tests: unleashed/tests/stackalloc_codegen_check.sh asserts the non-escaping
array emits no allocator call (FILLCHAR instead) while a returned one still
allocates; testfiles/stackalloc/ covers non-escape zero-fill/index/Length
semantics, the loop-reexecution guard, each escape route (returned, stored to
global, aliased) staying heap-allocated and correct, managed-element arrays
not transformed, and by-var element passing. Full suite 978/985 (only the 7
known pre-existing failures) both normally and with -OoSTACKALLOC forced on
globally; unrolldyn_check.sh and slp_codegen_check.sh still pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…caller-save spills to a callee's proven volatile clobber set
Ports the idea of gcc's -fipa-ra to FPC, intra-unit and x86_64 only. When a
routine's code is generated its body is already register-allocated, so the
register allocator knows exactly which physical registers it uses
(rg[bank].used_in_proc -- the same set FPC uses to pick prologue callee-saved
pushes). optipara.RecordProcClobbers snapshots, per register bank, the
intersection of that set with the ABI volatile (caller-saved) mask and records
it on the tprocdef. That summary is already transitively complete: every inner
call allocates its callee's clobbers around itself, so a routine calling an
un-analysed target ends up with the full mask (no unsafe reduction downstream);
one that only calls small proven leaves inherits just their small sets.
Consumer side (tcgcallnode.pass_generate_code): for a DIRECT call to an
already-recorded routine, the volatile registers allocated around the call are
narrowed to the callee's clobber set, so caller values living in untouched
volatile registers survive the call without a spill/reload. On a small-leaf hot
loop this removes the rbx/r12/r13/r14 evacuation entirely.
Both integer and XMM/MM clobbers are tracked. Conservative fallbacks to the full
ABI mask (a too-small set silently corrupts caller state):
- indirect/procvar, virtual, method-pointer, external, syscall, interrupt and
assembler callees;
- routines not yet generated in this compilation (forward order);
- inline-asm-containing or exception-using callees (marked ipara_full);
- and, crucially, any call inside a caller that itself has exception handling
or an implicit finally -- an exception unwind (longjmp) restores only
callee-saved registers, so a caller value kept in a volatile register across
a call that raises and is caught here would be lost.
The per-procdef summary is transient (never written to / read from the ppu), so
on load a routine defaults to "not analysed" => callers use the full ABI mask.
Gated behind -OoIPARA; NOT added to -O4 this round. Adds the cs_opt_ipara switch
(globtype OptimizerSwitchStr + ppudump table), an ipara_codegen_check.sh
assertion and runtime regression tests under testfiles/optipara/ (int leaves,
XMM leaves, and every fallback path). Full unleashed suite: no new failures with
the switch off or forced on globally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…t-in) Port of gcc's -ftree-scev-cprop (scalar-evolution final-value replacement) plus the whole-loop deletion its control-dependent DCE performs. New optimizer unit compiler/optfinalvalue.pas, wired as opt-in -OoFINALVALUE (globtype.pas enum + name table, ppudump.pp mirror, psub.pas call site before the DFA/loop passes so the still-structured for-nodes' bounds are read directly). NOT in the -O4 defaults. Transform: a counted for-loop whose whole body is a single loop-invariant accumulator update of a plain local integer (inc(s,c) / dec / s:=s+c / s:=s-c), whose counter's exit value is dead, is replaced by the closed form of s's exit value if a<=b then s:=s+(b-a+1)*c (to<=from for downto) and the loop deleted; an empty-bodied dead counted loop is deleted outright. Post-loop uses of s then read the closed form directly. Sound conservative subset (correctness over coverage): integer inductions only; counter a plain local integer <=32 bits (so the symbolic trip count b-a+1 is exact in 64-bit); accumulator a plain local integer distinct from the counter; c and the bounds loop-invariant, side-effect-free and independent of i/s; counter/accumulator rejected if address-taken, captured by a nested scope or volatile; zero-trip loops handled by the a<=b guard; two's-complement wraparound preserved so the result is bit-identical, hence DISABLED under -Co/-Cr; break/continue/exit/call/store bodies never match. Tests: unleashed/tests/testfiles/optfinalvalue/ (ascending/downto/zero-trip accumulators, byte/word/shortint wraparound, invariant step, empty-loop deletion, break/global/live-counter/nested-capture/addr-taken no-transform cases, switch round-trip, -Co disable) all self-checking bit-exact vs a direct reference; finalvalue_check.sh proves at the assembly level that eligible loops lose their back-edge while rejected loops keep it, and that ON/OFF output is identical. Full unleashed suite: 991 pass / 6 known-ignored, both normally and with -OoFINALVALUE forced on every test; ipara/slp/stackalloc/unrolldyn checks still pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add one generic per-procdef "optimizer summary" blob persisted through the PPU, then consume it for cross-unit -OoPURE (pure/const CSE of used-unit calls) and cross-unit -OoIPARA (narrowing caller-save spills around a used-unit leaf call). Mechanism (compiler/symdef.pas tprocdef.write_optimizer_summary / load_optimizer_summary): a self-describing (tag,len,payload) list streamed inside the ibprocdef entry, terminated by optsum_end. An older/absent summary yields no tags -> the existing conservative fallback. Tags live in ppu.pas (optsum_pure/optsum_ipara, optsum_icf reserved). Framing is versioned once via CurrentPPULongVersion 30 -> 31; new per-pass payloads are just new tags. - PURE: the DERIVED pure/const verdict is written as two booleans (computed at write time via a hook optpure registers, so the whole defining unit is already analysed). A used-unit routine carries pure_ppu_valid + the verdict; dfs_pure treats it as a leaf with that verdict. Kept in the interface crc so a flipped verdict recompiles dependents. - IPARA: the volatile-register clobber mask is written guarded by a target/ABI/-Cf instruction-set signature; a mismatching or full summary is dropped to the full ABI mask. The ncgcal consumer already had no same-unit gate, so cross-unit narrowing works once the mask is loaded. - ppudump dumps the new blob (PURE verdict, IPARA signature + mask, unknown tags skipped by length). Tests: unleashed/tests/optsummary/ two-unit fixtures + optsummary_check.sh (cross-unit pure call CSEs 3->1 under -OoGVNPRE -OoPURE, absent summary keeps 3, bit-exact/correct in every config); ipara_codegen_check.sh extended with a cross-unit case (used-unit leaf narrows callee-saved pushes 4->0, falls back when the callee unit lacks the switch). Full unleashed suite green normally and with -OoPURE -OoIPARA -OoGVNPRE forced on (992 pass / 0 unexpected / 5 ignored); RTL rebuilt for the version bump; all *_check.sh pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two follow-ups to the intra-unit -OoICF pass (compiler/opticf.pas):
1. Symbol-alias mode. A proven address-never-observed duplicate is now
collapsed to a *zero-byte* symbol alias (its entry symbol emitted as an
additional label at the survivor's address, the same two-labels-one-location
mechanism the codegen already uses for main/PASCALMAIN) instead of a 5-byte
jmp thunk. Gated conservatively via a new source-level address-taken proof:
- tprocdef.icf_addrtaken, set at the canonical proc->procvar conversion
(ttypeconvnode.create_proc_to_procvar) and at tloadnode.create_procvar,
captures every @proc / procvar assignment / procvar argument;
- the routine must be source-invisible to other units (procsym in the
static/local symtable, not exported/public/external/virtual, not a
method), so no source anywhere can form @dup;
- single entry symbol only (multi-aliasname routines fall back to thunk).
Address-taken / interface-visible / method duplicates keep the
address-preserving thunk, so @f<>@g stays observable where guaranteed. We do
NOT gate on the ELF global bind: under default section-per-symbol
smartlinking every routine symbol is emitted global, yet an
implementation-section routine is still un-nameable from another unit.
2. Cross-unit folding via the shared PPU optimizer-summary mechanism. Each
globally-visible routine that survives intra-unit folding stores a 128-bit
digest of its full canonical instruction stream on its tprocdef, persisted as
the (previously reserved) optsum_icf ppu tag guarded by the IPARA-style
target/ABI/-Cf signature and carrying the survivor's global name (captured at
write time; mangledname is not yet resolvable at load). A later unit registers
every used unit's survivors and folds a digest-matching routine into a jmp
thunk to that external global symbol. Cross-unit folds are always thunks
(never aliases): the survivor lives in another object and a thunk keeps every
duplicate's own address intact. Collision stance: cross-unit we trust the
128-bit digest of the full canon alone (no body to compare) -- birthday-bound
~n^2/2^128, and the identical canonicalisation + signature guard must also
match; documented in the unit header. The intra-unit bucket now keys on the
digest and still byte-verifies via full canon compare (fixing latent
truncation of >255-char canons, since mangled names are shortstrings here).
No PPU version bump (new length-prefixed, skippable summary tag). Survivors are
kept alive under smartlinking by the thunk/alias reference; verified with
default flags and -XX -CX -Xs.
Tests: unleashed/tests/testfiles/opticf/opticf_alias_01.pp (alias correctness);
unleashed/tests/icf_check.sh + icf/ fixtures (zero-byte alias asm shape,
address-taken thunk fallback with @ping<>@pong, cross-unit external jmp,
non-matching bodies never fold, bit-exact vs ICF-off and under smartlink).
Full unleashed suite: 992 pass / 0 unexpected. With -OoICF forced on: only
opticf_disabled_01 "fails" (it asserts ICF-off, so forcing it on is by design).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Extend the -OoVECTORIZE loop autovectorizer from single-only to also handle `double` dynamic arrays, 2 lanes per 128-bit window. Every existing shape now fires on double: array-array a[i]:=b[i] op c[i] (+ - *), scalar-broadcast a[i]:=b[i] op s / s op b[i] (incl. non-commutative subtraction and constant literal), plain copy a[i]:=b[i], and the sum / dot-product reductions (including the FMA-contracted form) under fast-math. Backend: tvectoropnode gains an `isdouble` flag (nbas.pas); the x86 emitter (nx86inl.pas) selects movupd/addpd/subpd/mulpd, movsd+unpcklpd for the splat, xorpd+movsd for the reduction seed, and an unpckhpd+addsd 2-lane horizontal sum, with the VEX v-forms under an AVX fputype (still 128-bit). min/max if-conversion stays single-precision only. Recognizer (optloop.pas): the precision is fixed from the destination array (store shapes) or the accumulator (reductions); every source array, scalar operand and RHS/addend must match it, so any single/double mix is rejected and stays scalar (a wrong lane width would miscompile). All single-precision soundness gates are preserved unchanged: plain-counter index, -Cr/-Co disable, globals/address-taken/volatile scalar rejection, self-alias safety, bit-exact vs the scalar path (incl. NaN/Inf/-0.0 and tail residues), and the fast-math gate on reductions. The SLP straight-line vectorizer stays single-only. AVX-256 (ymm) widening is intentionally NOT taken here: it needs a 256-bit horizontal-reduction path and OS_M256 codegen across the node, which is invasive/higher-risk; left as a documented follow-up. Tests: unleashed/tests/testfiles/optvect/vect_double_* (10 runtime bit-exact tests mirroring the single ones: per-op array-array, scalar-broadcast, copy, sum+dot reductions, store & reduction special-values, self-alias, mixed-type rejection, -Cr disable, AVX) plus vectorize_double_check.sh (asserts double loops pack to addpd/movupd, stay scalar with the switch off, and that a mixed single/double loop never vectorizes). Full unleashed suite green normally and with -OoVECTORIZE forced on (1000 pass / 0 unexpected). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(d) Emit a -vh hint per analyzed routine when -OoPURE discovers it pure/const (messages 06087/06088, once at AnalyzeProcPurity time, gated by hint verbosity). Reports the strongest verdict (const implies pure). (e) Broaden purity eligibility to read-only instance/class methods (the classic function TFoo.GetX:Integer;begin Result:=FX end accessor): - proc_eligible now accepts non-virtual, non-abstract, non-ctor/dtor methods; self is skipped as a read-only reference parameter. - lvalue_write_is_side_effect treats ANY store reachable through self as a side effect regardless of how self is passed (a by-value class pointer or a by-ref record self), so a method that writes a field is correctly impure -- the soundness-critical gate. - A method reads self's referenced memory, so it is at best "pure", never "const" (pure_reads_global stays set). No new PPU summary flavour is needed: the "pure" bit already means memory-dependent, and the -OoGVNPRE consumer treats every pure call as a memory reader. -OoGVNPRE method-call CSE (memory dependence): gvn_pure_call now accepts a provably-direct pure method call (non-virtual/abstract procdef, side-effect-free methodpointer); gvn_candidate value-numbers it as gvn_mem keyed on the locals read by self AND the arguments. Any intervening store (including a field write to the object) or any call therefore invalidates it -- two getter calls with a field write or an impure call between them are NOT commoned; two with nothing between are. Virtual/abstract methods are never folded. Cross-unit works via the existing optsum_pure bit (a pure method serializes exactly like a pure function). Tests: optpure_method_01.pp (bit-exact runtime: record-self getter commons, class getter with intervening field write / call / object reassignment does not); pure_method_check.sh (codegen call-count assertions incl. virtual-not-folded); pure_hint_check.sh (verdict text, silent for impure and with -OoPURE off). Full unleashed suite 1003 pass / 0 unexpected, bit-clean normally and with -OoPURE -OoGVNPRE forced on; all *_check.sh green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The level2 range-test optimization in nadd.pas (is_range_test) rewrites (C1 <= expr) and (expr < C2) into the unsigned test (expr - C1) <= (C2 - C1). When the lower bound C1 is negative, the low-bound constant is stored as an nf_internal *unsigned* ordinal (adaptrange converts e.g. -149 to qword(-149) = $FFFFFFFFFFFFFF6B). When such a helper is inlined at a call site where "expr" folds to a compile-time constant, the const-fold in taddnode.simplify evaluated e.g. 0 - qword(-149) which sets the tconstexprint overflow flag (0 < $FF..F6B in unsigned arithmetic) and raised a spurious user-facing "Overflow in arithmetic operation" ERROR at any level enabling cs_opt_level2 (-O2/-O3/-O4) -- even though this is intentional modulo-2^n wraparound in compiler-generated code. This made pletora/neural-api's neuralvolume.pas uncompilable at -O2+: inside pcr_powf -> is_exact_pf the range test "(-149 <= Trunc(y)*e) and (...)" folded once y=1/2.4 gave Trunc(y)=0, and the error surfaced (via the inlined body's stored fileinfo) at lab2rgb's three float "/const" statements, neuralvolume.pas(2147,1). Fix: in the addn/subn/muln integer const-fold, do not raise parser_e_arithmetic_operation_overflow for nf_internal nodes; keep the wrapped value (create_simplified_ord_const masks it to the result type). User-level overflow reporting is unchanged for non-internal nodes. Regression test const_fold_range_test_neg_lowbound_01 fails to compile at -O2 with the fix reverted (clean at -O1) and compiles+runs correctly with the fix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Port gcc's -foptimize-sibling-calls as an opt-in -OoSIBCALL pass, distinct from
the self-recursion loop rewrite (-OoTAILREC/cs_opt_tailrecursion) and from
partial inlining. When a routine's last action is a direct tail call to ANOTHER
routine (result := OtherFunc(args) or a bare tail procedure call), the frame
teardown is hoisted above the call and the call is rewritten to a jmp, so the
callee reuses the caller's return slot: O(depth) stack becomes O(1) for mutually
recursive pairs and continuation-style dispatch.
Implemented at the x86-64 assembler-peephole level (TX86AsmOptimizer.OptSibCall,
run from PostPeepholeOptCall, gated on cs_opt_sibcall). It also handles the
result-forwarded-through-a-callee-saved-register shape that the -O4
CallFrameRet2Jmp deliberately punts on -- the shape the register allocator emits
for `result := OtherFunc(args)`, i.e. exactly the mutual-recursion case:
recognises call X / mov %rax,%cs / [shared label] / mov %cs,%rax / <rsp
release + callee-saved pops> / ret, hoists only the pops+release (never the
result movs -- the jmp leaves the result in rax), and jmps to X.
Sound subset (x86-64 SysV/Linux only; anything not provably safe falls back to a
plain call):
* direct call to a symbol (indirect/procvar rejected);
* teardown = leaq/addq rsp release and/or pops of callee-saved integer regs;
* optional single result forward rax->callee-saved->rax (identity);
* no outgoing stack arguments in the routine (maxpushedparasize=0), so a
callee whose stack-arg area exceeds our zero incoming area is excluded;
* caller convention register/cdecl/stdcall (safecall + exotic excluded);
* CurrentProcAllowsSiblingTailFrameReuse: no try/finally/exception frame, no
dynamic stack alloc, no nested-frame capture, no address-taken local/param
(so no @Local can escape into the callee's arguments).
Opt-in; NOT added to genericlevel4optimizerswitches.
Switch wired in globtype.pas (cs_opt_sibcall + 'SIBCALL') and the ppudump name
table; -Oo parsing is table-driven.
Tests: testfiles/sibcall/ (even/odd + continuation runtime correctness, and a
disqualifier correctness file) pass under the suite both normally and with
-OoSIBCALL forced on the whole suite (1006 pass / 6 known pre-existing fails,
unchanged either way). unleashed/tests/sibcall_check.sh asserts the jmp in the
eligible case, a plain call with the switch off, and a plain call for each of
the four disqualifiers. An even/odd pair to depth 10^7 completes under
ulimit -s 512 with the flag and segfaults without it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Extends the opt-in final-value-replacement / dead-loop-elimination pass with two sound follow-ups: * POINTER-STRIDE accumulators: a counted loop whose body advances a plain local pointer by a loop-invariant stride via inc(p,stride)/dec(p,stride)/ inc(p) is replaced by the closed form p := p + (b-a+1)*stride, built as a pointer+integer add so the compiler applies the same element-size scaling the loop body did. The still-unlowered inc/dec inline form carries the raw (unscaled) element stride; the p:=p+stride assignment form is deliberately rejected because by this point the stride is already element-size-scaled and rebuilding a pointer+integer closed form would scale it a second time. * MULTI-STATEMENT bodies: a body of several statements, each an independent accumulator update of a DISTINCT plain local (integer or pointer), matches -- one closed form per accumulator under the shared trip-count guard. Two updates of the same accumulator (double-count) reject the loop. Every increment must be loop-invariant and reference neither the counter nor any accumulator, so the accumulators evolve independently and statement order is irrelevant. All prior soundness gates (<=32-bit counter, dead counter, no overflow/range checking, zero-trip guard, two's-complement wraparound) are preserved. Still opt-in; not in the -O4 defaults. Tests: finalvalue_check.sh grows codegen assertions (pointer-inc and multi- accumulator loops deleted; duplicate-accumulator and pointer-assign loops kept). New runtime fixtures fv_pointer_stride_01 (typed + PChar pointers, unit and non-unit strides, inc/dec, ascending/downto, zero-trip) and fv_multi_accum_01 (two/three integer accumulators, mixed integer+pointer body), plus dup-accumulator and pointer-assign kept-loop cases in fv_no_transform_01, all checked bit-exact against a direct reference and ON/OFF byte-identical. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e elimination The extended (record-field / static-array-element) DSE in optdeadstore.pas treated every call as a hard barrier that flushed the whole pending-store set. With -OoPURE on, a call whose resolved direct target carries a pure or const verdict is now relaxed, precise about the pure-vs-const asymmetry: * const call: reads and writes no memory -> full non-barrier; pending stores survive (only its argument expressions, scanned normally, perform reads); * pure call: writes nothing but MAY READ global/heap state -> treated as a potential reader: pending stores to globally reachable slots (static vars, by-ref const params) are kept live, while stores to non-address-taken locals and by-value params (which no callee can name) still survive; * impure / indirect / procvar / virtual / aggregate-returning calls remain full barriers, as do deref and asm nodes. Soundness first. Both the non-candidate-assignment scan and the previously unconditional "any other statement" barrier for bare call statements go through the new relaxed path (el_classify_call / el_haz_scan / apply_relaxed_barrier). Tests: unleashed/tests/pure_dse_check.sh asserts on codegen that the dead store IS removed across a const call and across a pure call on a local base, and is NOT removed across an impure call or across a pure call on a static base, and that the baseline (no -OoPURE) keeps everything; runtime fixture testfiles/optpure/optpure_dse_01.pp is deterministic and self-checking at -O1/-O2/-O3+flags/-O4+flags. Full suite 1009 pass / 6 known pre-existing fails normally AND with -OoPURE forced suite-wide; all 12 *_check.sh PASS. (The 7 optvect failures under forced -Oodeadstore are a pre-existing -Oodeadstore+VECTORIZE-reduction interaction, reproduced with this change stashed; not introduced here.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tale accumulator seed)
Root cause
----------
The autovectorizer's reduction path (OptimizeVectorize, vok_reduce_sum /
vok_reduce_dot in optloop.pas) seeds lane 0 of the packed accumulator with the
incoming scalar accumulator s -- i.e. the user's s := <start> store just
before the loop. That seed read of s was embedded directly inside the
create_reduce_init backend (tvectoropnode) node:
acc := reduce_init( acctemp, load(s) )
A backend node's operand reads are not reliably modelled by data-flow analysis
(the finish/horizontal-sum store already worked around this by writing s with a
plain assignment from a temp, per the comment at the store site). Because DFA
did not see s being read at the seed, the incoming s := <start> def looked
dead, and with dead-store elimination enabled (-Oodeadstore) the stock DSE pass
in optdeadstore.pas removed it. Lane 0 was then seeded from a stale stack slot,
producing a wrong sum / dot-product result.
This is why forcing -Oodeadstore suite-wide made 7 optvect runtime tests fail
(vect_reduce_01, vect_reduce_avx_01, vect_reduce_disabled_01,
vect_reduce_special_values_01, vect_double_avx_01, vect_double_reduce_01,
vect_double_reduce_special_values_01) while the compiler was otherwise correct
without -Oodeadstore. The bug is pre-existing and independent of -OoPURE.
Fix
---
Mirror the store-side workaround for the read side: copy the incoming s into a
memory-backed scalar temp with a PLAIN assignment first, and seed the packed
accumulator from that temp:
seedtemp := s; { DFA-visible read of s keeps its def live }
acc := reduce_init( acctemp, seedtemp )
The plain seedtemp := s assignment is an ordinary loadn read of s that DFA
tracks, so the incoming def of s stays live and DSE no longer removes it. Emitted
code is otherwise identical (one extra scalar copy that the register allocator
folds away in the common case).
Test
----
New runtime fixture unleashed/tests/testfiles/optvect/vect_reduce_deadstore_01.pp
forces -OoVECTORIZE -Oodeadstore via %OPT and checks single/double sum and dot
reductions against a strict sequential (downto) oracle over exactly-representable
inputs for every trip count; it fails before this change and passes after.
Full unleashed suite bit-clean both normally and with -Oodeadstore -OoVECTORIZE
forced suite-wide (1011 pass / 5 pre-existing known fails, no new failures); all
30 optvect tests green with the flags forced; all 12 *_check.sh codegen scripts
PASS.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…atic/global accumulators Lands the three remaining -OoFINALVALUE follow-ups, soundness-first (a wrong final value or wrongly-deleted loop is a miscompile, so the pass still leaves any non-provable loop alone). * 64-BIT COUNTERS: the counter may now be any integer up to 64 bits (max size raised 4 -> 8). The exact trip count is not needed: b-a+1 is computed in 64-bit two's-complement (the true iteration count reduced mod 2^64) and each accumulator product is truncated to the accumulator's own width w<=64, so (count*c) mod 2^w is bit-identical to the repeated addition; a pointer stride's count*stride element count is likewise correct mod 2^64. * STATIC / GLOBAL accumulators: a unit-level (static) variable may be an accumulator (never the counter -- a static counter's cross-unit uses the whole-routine deadness scan cannot see). Sound because the matched body has no calls and the routine no exception paths, so nothing observes the global's intermediate states cross-routine within this thread; the closed form stores the identical final value. different_scope (normally set on any global reached from a subroutine) is no longer treated as the nested-capture hazard -- that check now applies to locals only; addr_taken / volatile still reject, and threadvars / externals stay out. * RANGE CHECKING (-Cr, without -Co): re-enabled, restricted to native full-range integer accumulators (int64/qword) whose s:=s+c performs no narrowing and so is never range-checked, matching the nf_internal closed form bit-for-bit. Sub-native / subrange / pointer accumulators are range-checked on their narrowing store, so they are declined (loop kept). Bound range faults are preserved (the guard copies the original bound expressions verbatim), and the empty-dead-loop shortcut is disabled under -Cr so a narrowing bound still faults at loop entry. * -Co (overflow checking) stays DISABLED by design: a native accumulator loop traps on the overflowing iteration while the wrap-around closed form does not, and reproducing that per-iteration trap from a single closed-form step is not robustly sound (sub-native trap-freedom relies on a fragile promotion detail; a native accumulator's count*c can overflow int64 in exactly the trapping case). The pass exits early whenever overflow checking is on. Tests: new runtime fixtures fv_counter_64bit_01, fv_global_accum_01, fv_rangecheck_native_01; finalvalue_check.sh extended with codegen asserts (64-bit-counter / native / global loops deleted, deleted under -Cr; sub-native kept under -Cr; every loop kept under -Co). fv_no_transform_01 updated (global accumulator is now transformed). Full unleashed suite 1013 pass / 0 fail, byte-identical normally and with -OoFINALVALUE forced suite-wide. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Extend TX86AsmOptimizer.OptSibCall to recognise result forwarding through more than one location. The old matcher accepted only a single callee-saved register identity pair (rax->cs->rax). It now walks a chain of forward-saves that stage one or more integer return registers (rax, or rax:rdx for a 128-bit result) through callee-saved registers and/or rsp-relative frame slots, each with a matching restore before the teardown. Sources and destinations must be pairwise distinct and every save must be restored before any teardown, so the save/restore round-trip is a proven identity on the return registers; the transform then hoists the teardown, turns the call into a jmp, and drops the (now dead) forwarding moves exactly as before. This is the shape the register allocator emits for a 128-bit rax:rdx record result staged through two frame slots, and for a result split across a callee-saved reg and a stack slot. Also harden a pre-existing soundness hole in the frame-teardown gate (CurrentProcAllowsSiblingTailFrameReuse): symtable_has_addrtaken only tracks named locals/parameters, so compiler temporaries whose address escapes to the callee -- an open array of const, a @temp passed by reference -- slipped through and the frame was torn down before the callee read them (dangling stack). Scan the routine and decline when any lea/mov materialises a frame-relative address into a general register. This fixes a real miscompile that surfaced only under a forced suite-wide -O2 -OoSIBCALL (match_as_expression_in_arg_01); it does not affect the eligible register-only mutual-recursion / continuation cases. Tests: sibcall_forward128_01.pp (deep 128-bit + single-reg forwarding proving O(1) stack and correct results), sibcall_frameaddr_negative_01.pp (escape cases decline and stay correct), and new firing/decline codegen assertions in sibcall_check.sh (multi-location forward jmps, non-tail and frame-address declines). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…icter case-exhaustiveness check in optloop.pas Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- ProducedExePath appended .exe unconditionally, but unix binaries have no extension: every run-phase test reported 'compiled but exe not found' - inject -Facthreads on non-Windows targets so the threaded-feature tests (parallel for, async/await, lock, thread static) get the cthreads driver the compiler itself only hints about With these, 1200/1202 tests pass on x86_64-linux; the two failures (inline_vars widestring RTTI name, asyncawait reraise flake) reproduce identically on the pre-merge compiler. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fibodevy
requested changes
Jul 9, 2026
fibodevy
left a comment
fibodevy
left a comment
Member
There was a problem hiding this comment.
"Adds regression tests under unleashed/tests/testfiles/dead_store_normalize/ (all fail on the old compiler)"
All four pass on current main (-O3 -OoDEADSTORE):
[1/4] dead_store_normalize\dsnorm_ifexpr_nested_loops_01.pp ... PASS
[2/4] dead_store_normalize\dsnorm_match_in_loop_01.pp ... PASS
[3/4] dead_store_normalize\dsnorm_ifexpr_in_loop_01.pp ... PASS
[4/4] dead_store_normalize\dsnorm_funcref_compiles_01.pp ... PASS
The IE 200108231 fix landed upstream about a month ago (!1442), so the normalize() bug has been gone there since then. Today's sync pulled it into our main (acc89a1), so as of today it's fixed on our side too - which is why these pass instead of fail. "all fail on the old compiler" only held against the pre-sync base.
So the fix part is redundant now - unless I've overlooked something, in which case can you point to the case it still handles that acc89a1 doesn't? The regression tests themselves are welcome (they pass and lock the fix in), so I'd just take those.
Also worth checking the other fixes in the PR the same way - since this one was already resolved upstream, some of the others may have landed there too and be redundant after today's sync.
Author
|
Many thanks for the feedback. I will work on it (eventually). I will review: the nadd.pas const-fold spurious overflow, the -Oodeadstore × -OoVECTORIZE stale-seed miscompile, the testtool unix run-phase fix, and the case-exhaustiveness fixup. |
Add overloaded declarations bound to the same external symbols so callers can pass a variable directly instead of a pointer for the out-parameters of clGetPlatformInfo, clEnqueueMapBuffer, clEnqueueMapImage, clGetDeviceInfo, clCreateContext, clCreateContextFromType, clGetContextInfo, clGetCommandQueueInfo, clSetCommandQueueProperty, clCreateBuffer, clCreateImage2D, clCreateImage3D, clGetSupportedImageFormats, clGetMemObjectInfo, clGetImageInfo, clCreateSampler, clGetSamplerInfo, clCreateProgramWithSource, clCreateProgramWithBinary, clGetProgramInfo, clGetProgramBuildInfo, clCreateKernel, clCreateKernelsInProgram, clGetKernelInfo, clGetEventInfo and clGetEventProfilingInfo. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add a clCreateCommandQueue overload taking 'var errcode_ret: cl_int' alongside the existing pcl_int version; mark both with 'overload'. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…arus checkout rebuildu.sh chains the full toolchain rebuild (compiler clean cycle seeded via fpc -PB, RTL, packages, and the LCL when ../lazarus exists), detects the host CPU instead of assuming x86_64, and warns about stray package PPUs shadowing rtl/units. lazbuildu.sh now passes --lazarusdir pointing at the sibling ../lazarus checkout (LAZDIR overridable) so dependency packages are rebuilt from source with the in-tree compiler instead of failing with "PPU Invalid Version" against the system Lazarus install's precompiled units. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fpmake's NeedsCompile never compares package PPUs against the compiler or RTL (it only recompiles when a dependency package was built in the same fpmake session, or the unit's own sources/includes are newer), so a default rebuild left packages compiled by the previous compiler against the previous RTL next to the fresh toolchain. Make the packages clean unconditional; --no-clean-packages restores the old fast path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
New -O4 tree-level optimization passes, x86 peepholes, and supporting fixes
I have not properly tested. This is for you to know what I am working on.
Summary
This PR adds a family of new optimization passes to the Free Pascal compiler, enabled at
-O4(each individually controllable via its own-Oo…/-OoNO…switch) or opt-in via their own switch, together with x86-specific code-generation improvements, interprocedural analyses with cross-unit PPU summaries, bug fixes uncovered by self-compiling the compiler at-O4, and an extensive regression-test suite underunleashed/tests/(~320 files changed, ~40,000 insertions across 69 commits).New tree-level optimizer passes (
-O4)Loop optimizations (
compiler/optloop.pas, ~6,000 new lines)-OoLICM— loop-invariant code motion: hoists invariant computations out of loops.-OoLOOPUNSWITCH— loop unswitching: moves loop-invariant conditionals out of the loop, duplicating the loop body per branch (with a size cap).-OoLOOPPEEL— full peeling of small constant-trip-count loops.-OoLOOPSPLIT— loop splitting on an induction-variable crossover point.-OoLOOPFUSE— fusion of two adjacent loops iterating over the same space.-OoLOOPDISTPAT— loop-distribution pattern idiom recognition.-OoUNROLLJAM— outer-loop unroll-and-jam.-OoPREDCOM— predictive commoning of cross-iteration reloads.-OoIFCONVERT— loop if-conversion to packedmaxps/minps.-OoREASSOC— floating-point reduction reassociation (gated on fast-math).-OoUNROLLDYN(opt-in) — dynamic-trip loop unrolling: unrolls a counted for-loop of unknown trip count by 4 with a scalar remainder loop, keeping the original serial evaluation order so FP accumulation stays bit-identical.-OoPREFETCH(opt-in) — software prefetch: inserts aPREFETCHNTAofbase[i+64]once per iteration group for each distinct streamed dynamic-array base (skipping pure store destinations); semantically a no-op.-OoFINALVALUE(opt-in) — final value replacement + dead loop elimination (gcc-ftree-scev-cprop, new unitcompiler/optfinalvalue.pas): a counted for-loop whose body is only loop-invariant accumulator updates of plain locals, with a dead counter, is replaced by the closed formif a<=b then s := s + (b-a+1)*cand the loop deleted. Extended to pointer-stride accumulators (inc(p,stride)), multi-statement bodies (one closed form per distinct accumulator), 64-bit counters (exact mod 2^64 arithmetic), static/global accumulators, and native full-range accumulators under-Cr; disabled under-Coby design (the closed form cannot reproduce the per-iteration overflow trap).Vectorization
-OoVECTORIZE— conservative SSE loop autovectorization of element-wise array loops (add/mul/sub, copy, and scalar-broadcast shapes), with alias, range-check, and special-value safety guards. Includes per-loop diagnostics (notes 06065/06066) explaining why a loop was or was not vectorized.-OoVECTORIZEto the two canonical single-precision reductions,s := s + a[i](sum) ands := s + a[i]*b[i](dot product), widened to a 4-lane packed SSE/AVX accumulator with a scalar residue loop; recognizes the FMA-contracted dot shape on FMA targets. Fast-math gated (it reassociates FP adds, like-OoREASSOC). New note 06086.-OoVECTORIZEfrom single-only todoubledynamic arrays, 2 lanes per 128-bit SSE/AVX window: every existing shape (array-array+ - *, scalar-broadcast, copy, and the sum/dot reductions including the FMA-contracted form) now fires on double, withmovupd/addpd/subpd/mulpdand a 2-lane horizontal sum in the backend. Any single/double mix is rejected and stays scalar; all single-precision soundness gates are preserved.-OoSLP(opt-in) — superword-level parallelism (gcc tree-slp-vectorize / LLVM SLPVectorizer): packs a run of ≥4 adjacent, isomorphic scalar single-precision assignments over consecutive elements of the same array base — hand-unrolled straight-line code with no surrounding loop — into one 128-bit SSE packed op, reusing the loop vectorizer's backend node. Bit-identical to scalar (no reassociation, no fast-math gate).Value-range and idiom recognition
-OoRANGEELIM— value-range-analysis-driven range-check elimination (including dynamic-arrayHigh/Length-1bounds and must-check preservation for off-by-one and variable bounds).-OoVRP— interval value-range propagation with branch folding: forward-propagates integer intervals seeded from declared subrange/ordinal type bounds, for-loop counter bounds, and straight-lineconst/mod/andfacts, then folds everyifthe intervals already decide, deleting the dead arm.-OoBITIDIOM— recognition of bit-manipulation idioms: population count, andtzcnt/bsrbit-scan loops.modby a non-power-of-two constant: magic-number strength reduction (compiler/x86/nx86mat.pas).loopvar * invariantmultiplies inside array-index / pointer-offset expressions.Scalars, stores, and control flow
-OoSRA— scalar replacement of non-escaping local aggregates (new unitcompiler/optsra.pas).-OoSTOREMERGE— coalescing of adjacent narrow stores into wider stores.-OoJUMPTHREAD— jump threading / elimination of nested re-tests of the same condition.compiler/optdeadstore.pas) — DSE now also handles record-field and static-array-element stores.-OoSINK— partially-dead code sinking (the symmetric counterpart of LICM): a pure assignment immediately preceding anifwhose value is consumed on only one arm (and dead after the branch) is relocated into that arm, so paths that never use it stop paying for it.-OoSTOREMOTION— loop store motion / scalar promotion (gcc-fgcse-sm): a global of unmanaged scalar type repeatedly loaded/stored in a loop is promoted to a register temp — loaded once before the loop, stored once after — under a strict no-call/no-alias/no-trap whitelist over the entire loop.-OoGVNPRE(opt-in) — dominator-based global value numbering / full-redundancy elimination across statements and rejoining branches: a side-effect-free scalar expression available on every path to a point is computed once into a temp and reused. Availability tables are copied into branch arms and intersected at rejoins; killed conservatively by operand writes, memory stores, and calls. Validated by a 20k-case randomized differential and a self-compile with the pass active (816 firings). New note 06085.-OoPARTIALINLINE(opt-in) — partial inlining / function splitting (gcc-fpartial-inlining, new unitcompiler/optpartialinline.pas): a routine that begins with a cheap, side-effect-free early-exit guard followed by an expensive body is split into a tiny inlinable header (guard + forwarding call) that takes over the original procsym, and an out-of-line body keeping the full original code — so the common guarded-exit path pays no call at all. Covers both procedures and functions (the function case gives the header its own funcret local and remaps the guard'sexit(value)onto it, so both arms write the same result location). A 97%-early-exit guard in a hot loop runs ~16% faster.-OoSTACKALLOC(opt-in) — escape-analysis stack allocation of non-escaping local dynamic arrays: a local dynamic array of unmanaged element type whose onlySetLengthhas a positive constant length (within a 4 KiB frame budget) and which provably never escapes is given a frame-local buffer with FPC's own read-only refcount sentinel (-1), eliding the heap allocation entirely; scope-end finalization neither frees nor finalizes it.Interprocedural analyses and cross-unit PPU summaries
-OoPURE(opt-in) — interprocedural pure/const discovery (gcc-fipa-pure-const, new unitcompiler/optpure.pas): walks the unit's routines bottom-up over the call graph and proves per routine whether it is const (result depends only on by-value parameters) or pure (may read but never writes global memory, no I/O or other side effects), with a greatest-fixpoint DFS so mutually-recursive SCCs are still proven. Anything not provably harmless is impure. Eligibility covers standalone routines, const/constref-aggregate readers, and read-only instance/class methods (the classic field getter). Emits-vhpurity hints (messages 06087/06088). Consumed by:-OoLICM— hoists proven-const calls with loop-invariant arguments out of loops;-OoGVNPRE— CSEs identical pure calls (including direct pure method calls), value-numbered as memory readers killed by any intervening store or impure call;-OoIPARA(opt-in) — interprocedural register allocation (gcc-fipa-ra): records each generated routine's proven volatile-register clobber set, and narrows the caller-save spills around a direct call to that set, so caller values in untouched volatile registers survive without a spill/reload. Conservative full-ABI-mask fallbacks for indirect/virtual/external/assembler callees, exception contexts, and not-yet-generated routines.-OoICF(opt-in) — identical code folding (gcc-fipa-icf/ gold--icf): canonicalizes each generated routine body at the assembler-list level and replaces byte-identical duplicates with a 5-bytejmpthunk to the kept copy (addresses stay pairwise distinct, so@f<>@gsurvives). A proven address-never-observed, unit-private duplicate is instead collapsed to a zero-byte symbol alias at the survivor's address.(tag,len,payload)summary blob streamed inside the PPUibprocdefentry (versioned once viaCurrentPPULongVersion30 → 31; new passes add tags without further bumps), dumped byppudump. Consumed for cross-unit-OoPURE(pure/const CSE of used-unit calls, verdict kept in the interface CRC), cross-unit-OoIPARA(clobber mask guarded by a target/ABI/-Cfsignature), and cross-unit-OoICF(128-bit digests of surviving routine bodies; a digest-matching routine in a later unit folds to a jmp thunk to the external symbol).Case-statement lowering
-OoCASECLUSTER— gcc-style case clustering: partitions sorted case labels by dynamic programming into an optimal mix of jump-table clusters, bit-test clusters (thecase c of 'a','e','i','o','u'shape becomes one shift+AND membership test), and plain-compare singletons, dispatched via a balanced binary comparison tree.-OoSWITCHTABLE— case-to-lookup-table conversion (gcc-ftree-switch-conversion): a fully-covered, hole-free case whose arms only assign compile-time constants to the same ordered set of ordinal variables is replaced by per-variable static lookup tables plus a single range guard (no guard at all under full type coverage). New notes 06083/06084.x86 code generation
compiler/x86/aoptx86.pas): tail calls from a framed routine now reuse the caller's frame, with negative guards for managed types, pointer arguments, andtry/finallycontexts.-OoSIBCALL(opt-in) — sibling-call optimization (gcc-foptimize-sibling-calls), distinct from the self-recursion rewrite: when a routine's last action is a direct tail call to another routine, the frame teardown is hoisted above the call and the call rewritten to ajmp, so O(depth) stack becomes O(1) for mutual recursion and continuation-style dispatch (an even/odd pair to depth 10⁷ completes underulimit -s 512). Handles result forwarding through a chain of callee-saved registers and/or frame slots (including 128-bitrax:rdxrecord results). Hardened frame-teardown gate: anylea/movmaterialising a frame-relative address into a general register (escaping compiler temporaries such as open-array-of-const) declines the transform.-OoCROSSJUMP— cross-jumping / tail merging (gcc-fcrossjumping) as a late post-peephole pass: predecessor blocks ending in an operand-exact identical instruction tail keep one copy and jump into it. Pure code-size/I-cache win; refuses to delete regions containing labels, CFI directives, or debug/unwind data.-OoBLOCKORDER— static hot/cold basic-block layout (gcc-freorder-blocks): sinks cold error/raise regions (guard clauses ending infpc_raiseexception,RunError, range/overflow helpers, etc.) out of the hot path to the end of the routine, inverting the guard so the hot successor is the fall-through.-OoREE— redundant sign/zero-extension elimination (gcc-free) as a post-peephole pass: deletes amovzx/movsxwhose value already carries the required extension from its nearest definition (an earlier extension, anand-mask,xor reg,reg, or a small non-negative immediate move), reaching past the adjacency limit of the existing peepholes.-OoSHRINKWRAP(opt-in) — shrink-wrapping (gcc-fshrink-wrap): sinks a push-only callee-saved prologue below an initial volatile-only guard clause, so early-exit fast paths (nil check, zero-length, cache hit) run prologue-free with a bareret. DWARF CFI stays correct (position labels relocate with the pushes); validated withobjdump --dwarf=framesand a full self-host with the pass forced on.Managed types
-OoREFELIDE(opt-in) — managed refcount elision (ARC-style pair elimination): an ansistring borrowa := bwhereais thereafter only read is lowered to a plain pointer copy with the incref and the finalization decref both removed; the source provably keeps the buffer alive fora's lifetime. Verified leak-free with-ghon every fixture.Bug fixes and infrastructure
opttree: fixednormalize()hoisting block-expressions out of control-flow bodies, which broke-Oodeadstoreonif-expressions andmatchinside loops.REASSOC: fixed a miscompile / internalerror 200306031 when the loop counter appears in the accumulated expression — substituted(i+delta)copies kept stale cachedresultdef/pass1 flags; the whole substituted expression is now re-firstpassed.nadd.pasconst-fold: fixed a spurious "Overflow in arithmetic operation" error at-O2+ when the level-2 range-test rewrite'snf_internalunsigned wraparound constants (a negative low bound stored asqword) fold at an inlined call site — internal nodes now keep the wrapped value instead of raising; user-level overflow reporting is unchanged. (Madeneural-api'sneuralvolume.pascompilable at-O2+.)optvectorize: fixed a-Oodeadstore×-OoVECTORIZEreduction miscompile — the packed accumulator's lane-0 seed read ofswas embedded in a backend node invisible to DFA, so DSE deleted the incomings := <start>store; the seed now goes through a plain DFA-visible scalar copy.testtool: fixed the run phase on unix targets (.exewas appended unconditionally) and injected-Facthreadson non-Windows so threaded-feature tests link the cthreads driver; 1200/1202 tests now pass on x86_64-linux.optloop.pas: satisfiedun_main's stricter case-exhaustiveness check after cherry-picking the optimizer work.-O4stage-2 self-compile of the compiler build clean: fixed DFA false positives and case-exhaustiveness warnings.Doublefor real-literal initializers; documented theRttiunit build for tests.compiler/msg/errore.msg;ppudumpupdated for the new option flags.fpcu.sh(invoke the in-treeppcx64with the built unit paths) andlazbuildu.sh(lazbuildusing the in-tree compiler) wrapper scripts; README Quick Start section on compiling programs and Lazarus projects with-O4.Testing
unleashed/tests/testfiles/(optlicm,optunswitch,optvect,optvrp,optvrpfold,optbitidiom,optbitscan,optsibcall,optstoremerge,optdistpat,optreassoc,optcasecluster,optcrossjump,optblockorder,optsink,optstoremotion,optswitchtable,optree,optshrinkwrap,optgvnpre,optrefelide,dead_store_ext,optpure,optpartialinline,optslp,optunrolldyn,optipara,opticf,stackalloc,optfinalvalue,sibcall,optsummary, and more), including must-fire, must-not-fire, safety, and disabled-switch cases, plus per-pass codegen assertion scripts (*_check.sh) proving at the assembly level that eligible shapes fire and rejected shapes do not.-O4with all new passes enabled (including full self-hosts with the opt-inSHRINKWRAPandGVNPREpasses forced on); the full unleashed suite passes with zero regressions beyond the documented pre-existing failures.Commits (oldest first)
ce247f6f99add -OoLICM loop-invariant code motion at -O4f85029776etests: optlicm loop-invariant code motion coverage5942317888add -OoLOOPUNSWITCH loop unswitching at -O4e3c1c8aca7add -OoBITIDIOM bit-population-count idiom recognition at -O4a146378dccadd -OoRANGEELIM value-range-analysis range-check elimination at -O4b58a765960add -OoVECTORIZE conservative SSE loop autovectorization87ef474c9eSigned mod by non-power-of-two constant: magic-number strength reduction612dd73ee3add tzcnt/bsr bit-scan idiom recognition to -OoBITIDIOM at -O4a417889ad1strength-reduce loopvar*invariant multiplies inside array-index/pointer offsetsb4892c6230add -OoVECTORIZE per-loop vectorization diagnostic (notes 06065/06066)68259ed54f-OoVECTORIZE: add scalar-broadcast and copy element-wise shapes9a57af019foptdeadstore: extend DSE to record-field and static-array-element stores6451f6addcaoptx86: sibling-call frame reuse for tail calls from a framed routine445fbd6c8aAdd -OoJUMPTHREAD jump threading / nested re-test elimination at -O4fd7296cf5bopttree: fix normalize() hoisting block-exprs out of control-flow bodies (-Oodeadstore)e203bc363fcompiler: make an -O4 stage-2 self-compile build clean (DFA false positives + case exhaustiveness)d67e6b8af7inline vars: infer Double for real-literal init; document Rtti unit build for tests (task 273)77a02770c5add -OoLOOPDISTPAT loop-distribution pattern idiom recognition at -O4babb3b5494add -OoLOOPPEEL full loop peeling of small constant-trip loops at -O4971e03dd61add -OoLOOPSPLIT loop splitting on an induction-variable crossover at -O49293ace3feadd -OoLOOPFUSE loop fusion of two adjacent same-space loops at -O4aedba022e2add -OoIFCONVERT loop if-conversion to packed maxps/minps at -O4324f28bea6add -OoREASSOC floating-point reduction reassociation at -O4 (fast-math gated)0e1bef5300add -OoUNROLLJAM outer-loop unroll-and-jam at -O453f858294cadd -OoPREDCOM predictive commoning of cross-iteration reloads at -O4455f28051cadd -OoSRA scalar replacement of non-escaping local aggregates at -O410b1ba8d85add -OoSTOREMERGE adjacent narrow-store coalescing at -O4a1f1438bbfadd -OoCASECLUSTER gcc-style case clustering at -O4a6978eadf3add -OoCROSSJUMP cross-jumping / tail merging at -O48e5adcad82add -OoBLOCKORDER static hot/cold basic-block layout at -O4776b611da0add -OoSINK partially-dead code sinking at -O4c156929359add -OoSTOREMOTION loop store motion / scalar promotion at -O48b4171b34cadd -OoVRP interval value-range propagation with branch folding at -O4ad7351332dadd -OoREFELIDE managed refcount elision (opt-in)a484c09620fix REASSOC miscompile/internalerror when the counter appears in the accumulated expression1cb1ab4e9badd -OoSWITCHTABLE case-to-lookup-table conversion at -O462aeea0f98add -OoREE redundant sign/zero-extension elimination at -O4f7f8d2c030add -OoSHRINKWRAP shrink-wrapping (opt-in), sinking a push-only prologue below a guard clausecf11e776f4add -OoGVNPRE dominator-based global value numbering / full-redundancy eliminationd4645bc067add Quick Start section to README: compiling programs and Lazarus projects with -O40e595fe7acadd fpc.sh wrapper: invoke in-tree ppcx64 with -O4 and built unit paths5f238a038dextend -OoVECTORIZE with single-precision sum / dot-product reduction vectorizatione45bbf4b59rename fpc.sh to fpcu.sh to avoid confusion with the fpc driver; document it in README Quick Startb1213a0f2adrop forced -O4 from fpcu.sh so debug builds stay debuggable; adjust READMEa7486cf488add lazbuildu.sh wrapper: lazbuild with the in-tree compiler via fpcu.sh14f577ffa7lazbuildu.sh: use the lazbuild on PATH instead of a sibling checkoutcfdeaf5297add -OoPURE interprocedural pure/const discovery, consumed by -OoLICM1de4a6465badd -OoPARTIALINLINE partial inlining / function splitting (procedures)118bcd0684add -OoSLP superword-level parallelism (SLP) straight-line vectorization247fba5de0add -OoUNROLLDYN dynamic-trip loop unrolling and -OoPREFETCH software prefetch98d932ba51extend -OoPARTIALINLINE partial inlining to functions (non-void return)e9e466506d-OoGVNPRE: consume the -OoPURE verdict (CSE identical pure calls); broaden pure eligibility to const-aggregate readersce7d807420add -OoICF identical code folding (opt-in), folding duplicate routine bodies via jump thunks4ea8de1765add -OoSTACKALLOC escape-analysis stack allocation of non-escaping local dynamic arrays95c78053ecadd -OoIPARA interprocedural register allocation (opt-in), narrowing caller-save spills to a callee's proven volatile clobber setfe0a4fbbe0add -OoFINALVALUE final value replacement + dead loop elimination (opt-in)62275fd46ashared cross-unit PPU optimizer summaries: cross-unit -OoPURE + -OoIPARA39786cb38c-OoICF: symbol-alias mode + cross-unit identical code foldingc2a48b0437-OoVECTORIZE: double-precision loop autovectorization (128-bit SSE/AVX)7ac3c112fe-OoPURE: -vh purity hints + read-only method/accessor eligibilitye620c046ccFix cs_opt_level2 const-fold spurious overflow on internal wraparound4bf0beabd2add -OoSIBCALL sibling-call optimization (opt-in)a7a109c1bf-OoFINALVALUE: pointer-stride accumulators + multi-statement bodies26da550200-OoPURE: treat pure/const calls as non-barriers in extended dead-store eliminationad3964f734optvectorize: fix -Oodeadstore x -OoVECTORIZE reduction miscompile (stale accumulator seed)f7be469997-OoFINALVALUE follow-ups: 64-bit counters, -Cr range-check regime, static/global accumulators5898a20945-OoSIBCALL: multi-location result forwarding + frame-address escape gate1f5e957d97fixup after cherry-picking main optimizer work: satisfy un_main's stricter case-exhaustiveness check in optloop.pasc95bce2901testtool: fix run phase on unix targets