Skip to content

WIP: ast-fuzz assertion detector, mutation kinds, and an ast_verify check for base-class nodes - #3862

Open
aleksisch wants to merge 23 commits into
masterfrom
aleksisch/more-fuzzer
Open

WIP: ast-fuzz assertion detector, mutation kinds, and an ast_verify check for base-class nodes#3862
aleksisch wants to merge 23 commits into
masterfrom
aleksisch/more-fuzzer

Conversation

@aleksisch

Copy link
Copy Markdown
Collaborator

The AST fuzzer could not see the bugs it was built to find. Every sweep ran against a Release binary, where DAS_NO_ASSERTIONS compiles out all 318 internal assertions, so a run was only detectable when it segfaulted. This branch adds an assertions-enabled path, five mutation kinds, and a verifier check for the one defect class the work turned up.

The verifier now reports a base class used as a node. ExprConst and ExprMakeLocal are bases that a concrete node derives from, and neither overrides visit, so a walk reaching one asserts inside Expression::visit with no diagnostic. The walk it kills includes the verifier's own, so the scan reads the module's gc list instead of descending. This matters outside the fuzzer: a macro that builds an ExprMakeLocal crashes a Debug build today and gets a located error instead.

The fuzzer gained mutation kinds for function signatures and for arbitrarily deep type or statement nesting, and a --mut-depth knob. Four defects in the fuzzer's own instrumentation were fixed, each of which had been producing numbers that measured nothing: two concurrent sweeps shared one victim file, --synth-skip was inert for the main generation path, mutations aimed at a required module were never re-checked, and timeouts were judged against a baseline that excluded the macro module's own load cost.

No compiler bug was found. Roughly 11,000 source-expressible mutants across Release and Debug produced zero crashes and zero assertion failures, with the detector independently proven able to fire.

Ledger

Validation

Full preflight and the AOT sweep were not run - this is a WIP branch and CI is the first full check. Run locally: rebase onto origin/master, lint and format over all changed .das, the ast-fuzz suite (19/19), and the make-pr sync and ast-verify gates. The Debug binary used for the assertion work was built in a separate worktree, since CMake pins output to <src>/bin.

The verifier check was negative-controlled: 60 real test files compiled under --ast-verify produced 0 false positives, and 26 of 31 generator repros that previously asserted now report a located error.

Claims - stated, not tested

The remaining 5 repros reach the compiler's own Program::visit before any verifier entry runs. No daslib-side check can precede that walk; closing it would need a compiler-side pre-pass, which is not attempted here. A break would look like a Debug assert in Expression::visit with no AST verify line before it.

The ast-verify gate skipping utils/internal/ast-fuzz/selftest/ is a pre-existing gap, not one this branch introduced: cycle.das and alias.das emit the same reports under the gate's own command.

aleksisch and others added 23 commits August 25, 2026 20:49
Typed mode builds each expression bottom-up from a requested type, so the
program is well-typed by construction and inference runs to the end instead
of stopping at the first error. Run mode gives the victim a main that calls
every synthesized function, so the same program can be executed twice and
its output compared.

probe.das is the source-level counterpart to the AST generator: it writes
ordinary source text, so a crash it finds has a repro anyone can paste. Its
mismatch matrix found the const-initializer folding crash.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The relative path broke every Bash call as soon as the working directory
moved out of the project root.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The const-initializer folding crash the matrix found is fixed upstream, so all
6900 cells report cleanly and the matrix can guard the fix. The gate samples
every fifth cell (25s); a full sweep drops --sample.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three axes past the mismatch matrix: declarations that refer to themselves, text
that is not a program, and mutations of real corpus files. Mutation swaps stay
inside a category (type for type, operator for operator) and rewrite one token
inside its own line, so the mutant still parses and the diagnostic comes from
inference rather than the grammar - 119 of 122 sampled mutants reach inference,
against 2 of 10 for cross-category swaps.

Each corpus builder logs the file and mutant counts it produced: a builder that
finds nothing has to say so instead of leaving a sweep that proves nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A compile-time blowup that finishes is invisible to the per-probe timeout: the
optimizer findings only surfaced because they happened to exceed it under load.
Every probe is now timed and anything past --slow-secs is reported and kept.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Materializing every mutant up front cost gigabytes and got the sweep OOM-killed
on a corpus the size of daslib. A mutant now stores the source path, line, token
and replacement, and the text is built when the probe is written.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An absolute threshold flags every mutant of a big corpus file: daslib/aot_cpp.das
compiles in 6s by itself, so twelve parallel shards pushed its mutants past 8s and
reported five false positives. A corpus mutant is now only slow if it exceeds
max(5x, +slow-secs) of the unmutated file, measured once per file per shard.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Mutating source text is the wrong lever for inference: most mutants die in the
parser, and the ones that do not are a worse-targeted version of what the AST
generator already reaches. The synthesized matrices stay - they are gated tests.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The generator builds trees from nothing, which is why its findings are shapes only
a macro can build. Mutation starts from a real parsed program and applies one edit
a person could have typed - a different operator, literal, argument order, name or
declared type - so inference sees an ordinary program and a crash is a compiler bug.

--mutate <file> reuses the whole existing driver: the seed range sweeps the
candidate node index, and subprocess isolation, --threads and the
crash/verifier/timeout classification are unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The node index a mutation targets only exists if the file has that many candidates
of that kind: a fixed seed range spends almost every run on a no-op and reports it
as clean. The driver now measures the count per file and kind from the macro's own
report and sweeps exactly that range, over every file in a directory and every kind,
in one invocation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Swapping an operator or a literal is what inference handles all day, which is why
those kinds find nothing. Seven more, each still something a person could type:
drop a declared type and make inference derive it, add const, wrap a type in a
container, rewrite an assignment as a move or a clone, put a statement inside
unsafe / try-recover / a bare scope, move a statement into an invoked closure, and
duplicate a statement.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One well-formed edit is something inference handles every day, which is why single
mutations of real programs find nothing. Several at once put it in a state no single
edit reaches, while each edit on its own stays source-expressible - so a crash there
is a compiler bug and not a shape only a macro can build.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The calibration compile also ran in count mode, so it never reported a candidate
count and every file was silently skipped.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two kinds were building trees the parser never produces, so a crash from them would
have been a verifier finding wearing a bug's clothes. Renaming a variable now drops
the resolved pointer and its flags, the way an unresolved name looks; a statement
moved into a closure gets the return type a closure carries. Both now report the
ordinary diagnostics source would: cannot locate variable, not expecting a return
value.

Running a mutation sweep with --verify is the check for this: verifier-caught should
stay zero, because a mutant is supposed to be a program, not a malformed tree.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five mutation kinds on shapes the existing set never reached: const argument,
default value on the last argument, result type to auto, and a let type or a
statement nested to an arbitrary depth. Each is a shape source can spell, so a
crash from one is a compiler bug rather than a malformed-tree report.

A timeout now keeps a repro on disk, the same way a crash does.

Only the module being compiled is mutated. A required module is already
inferred by the time the macro runs, so edits to it are never re-checked - a
sweep aimed at one reports every run clean while measuring nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An assertions-enabled build reports an internal invariant violation and traps.
The banner identifies it even where the trap is swallowed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two sweeps in one tree shared __wrapN.das and __workerN.log, so each compiled
the other's victim and both reported numbers that measured nothing. A run token
in the name keeps them apart.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Requiring the macro module costs ~1s in Release and ~10s in Debug before any
edit is applied. Judged against a bare-file baseline, an ordinary victim reads
as a compile-time blowup.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The skip set was filled at one of the six generator construction sites, so the
flag was inert for --synth-funcs and --synth-decls. Bisecting a crash with it
excluded nothing and pointed at the wrong kind.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ExprConst and ExprMakeLocal are bases a concrete node derives from. Neither
overrides visit, so a walk that reaches one asserts inside Expression::visit on
an assertions-enabled build, with no diagnostic - and the walk it kills includes
this verifier's own. The scan therefore reads the module's gc list instead of
descending, and returns before the visitor starts.

All three entry points check: verify_module, verify_module_after_infer, and
verify_function, which has no module of its own and asks the program being
compiled for one.

The pair is what a macro can build and a visitor cannot walk, measured rather
than read off the header: of the 43 classes without their own visit, 39 inherit
one and 2 cannot be constructed from daslang at all.

Of 31 generator repros that asserted, 26 now report a located error. The rest
reach the compiler's own Program::visit before any verifier entry runs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each selftest fixture breaks its own AST and asserts the report that follows, so
the gate read those reports as a node built wrong and went red for any PR that
touched one. Pre-existing: cycle.das and alias.das behave the same.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant