WIP: ast-fuzz assertion detector, mutation kinds, and an ast_verify check for base-class nodes - #3862
Open
aleksisch wants to merge 23 commits into
Open
WIP: ast-fuzz assertion detector, mutation kinds, and an ast_verify check for base-class nodes#3862aleksisch wants to merge 23 commits into
aleksisch wants to merge 23 commits into
Conversation
Typed mode builds each expression bottom-up from a requested type, so the program is well-typed by construction and inference runs to the end instead of stopping at the first error. Run mode gives the victim a main that calls every synthesized function, so the same program can be executed twice and its output compared. probe.das is the source-level counterpart to the AST generator: it writes ordinary source text, so a crash it finds has a repro anyone can paste. Its mismatch matrix found the const-initializer folding crash. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The relative path broke every Bash call as soon as the working directory moved out of the project root. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The const-initializer folding crash the matrix found is fixed upstream, so all 6900 cells report cleanly and the matrix can guard the fix. The gate samples every fifth cell (25s); a full sweep drops --sample. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three axes past the mismatch matrix: declarations that refer to themselves, text that is not a program, and mutations of real corpus files. Mutation swaps stay inside a category (type for type, operator for operator) and rewrite one token inside its own line, so the mutant still parses and the diagnostic comes from inference rather than the grammar - 119 of 122 sampled mutants reach inference, against 2 of 10 for cross-category swaps. Each corpus builder logs the file and mutant counts it produced: a builder that finds nothing has to say so instead of leaving a sweep that proves nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A compile-time blowup that finishes is invisible to the per-probe timeout: the optimizer findings only surfaced because they happened to exceed it under load. Every probe is now timed and anything past --slow-secs is reported and kept. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Materializing every mutant up front cost gigabytes and got the sweep OOM-killed on a corpus the size of daslib. A mutant now stores the source path, line, token and replacement, and the text is built when the probe is written. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An absolute threshold flags every mutant of a big corpus file: daslib/aot_cpp.das compiles in 6s by itself, so twelve parallel shards pushed its mutants past 8s and reported five false positives. A corpus mutant is now only slow if it exceeds max(5x, +slow-secs) of the unmutated file, measured once per file per shard. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Mutating source text is the wrong lever for inference: most mutants die in the parser, and the ones that do not are a worse-targeted version of what the AST generator already reaches. The synthesized matrices stay - they are gated tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The generator builds trees from nothing, which is why its findings are shapes only a macro can build. Mutation starts from a real parsed program and applies one edit a person could have typed - a different operator, literal, argument order, name or declared type - so inference sees an ordinary program and a crash is a compiler bug. --mutate <file> reuses the whole existing driver: the seed range sweeps the candidate node index, and subprocess isolation, --threads and the crash/verifier/timeout classification are unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The node index a mutation targets only exists if the file has that many candidates of that kind: a fixed seed range spends almost every run on a no-op and reports it as clean. The driver now measures the count per file and kind from the macro's own report and sweeps exactly that range, over every file in a directory and every kind, in one invocation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Swapping an operator or a literal is what inference handles all day, which is why those kinds find nothing. Seven more, each still something a person could type: drop a declared type and make inference derive it, add const, wrap a type in a container, rewrite an assignment as a move or a clone, put a statement inside unsafe / try-recover / a bare scope, move a statement into an invoked closure, and duplicate a statement. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One well-formed edit is something inference handles every day, which is why single mutations of real programs find nothing. Several at once put it in a state no single edit reaches, while each edit on its own stays source-expressible - so a crash there is a compiler bug and not a shape only a macro can build. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The calibration compile also ran in count mode, so it never reported a candidate count and every file was silently skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two kinds were building trees the parser never produces, so a crash from them would have been a verifier finding wearing a bug's clothes. Renaming a variable now drops the resolved pointer and its flags, the way an unresolved name looks; a statement moved into a closure gets the return type a closure carries. Both now report the ordinary diagnostics source would: cannot locate variable, not expecting a return value. Running a mutation sweep with --verify is the check for this: verifier-caught should stay zero, because a mutant is supposed to be a program, not a malformed tree. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five mutation kinds on shapes the existing set never reached: const argument, default value on the last argument, result type to auto, and a let type or a statement nested to an arbitrary depth. Each is a shape source can spell, so a crash from one is a compiler bug rather than a malformed-tree report. A timeout now keeps a repro on disk, the same way a crash does. Only the module being compiled is mutated. A required module is already inferred by the time the macro runs, so edits to it are never re-checked - a sweep aimed at one reports every run clean while measuring nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An assertions-enabled build reports an internal invariant violation and traps. The banner identifies it even where the trap is swallowed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two sweeps in one tree shared __wrapN.das and __workerN.log, so each compiled the other's victim and both reported numbers that measured nothing. A run token in the name keeps them apart. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Requiring the macro module costs ~1s in Release and ~10s in Debug before any edit is applied. Judged against a bare-file baseline, an ordinary victim reads as a compile-time blowup. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The skip set was filled at one of the six generator construction sites, so the flag was inert for --synth-funcs and --synth-decls. Bisecting a crash with it excluded nothing and pointed at the wrong kind. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ExprConst and ExprMakeLocal are bases a concrete node derives from. Neither overrides visit, so a walk that reaches one asserts inside Expression::visit on an assertions-enabled build, with no diagnostic - and the walk it kills includes this verifier's own. The scan therefore reads the module's gc list instead of descending, and returns before the visitor starts. All three entry points check: verify_module, verify_module_after_infer, and verify_function, which has no module of its own and asks the program being compiled for one. The pair is what a macro can build and a visitor cannot walk, measured rather than read off the header: of the 43 classes without their own visit, 39 inherit one and 2 cannot be constructed from daslang at all. Of 31 generator repros that asserted, 26 now report a located error. The rest reach the compiler's own Program::visit before any verifier entry runs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each selftest fixture breaks its own AST and asserts the report that follows, so the gate read those reports as a node built wrong and went red for any PR that touched one. Pre-existing: cycle.das and alias.das behave the same. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The AST fuzzer could not see the bugs it was built to find. Every sweep ran against a Release binary, where
DAS_NO_ASSERTIONScompiles out all 318 internal assertions, so a run was only detectable when it segfaulted. This branch adds an assertions-enabled path, five mutation kinds, and a verifier check for the one defect class the work turned up.The verifier now reports a base class used as a node.
ExprConstandExprMakeLocalare bases that a concrete node derives from, and neither overridesvisit, so a walk reaching one asserts insideExpression::visitwith no diagnostic. The walk it kills includes the verifier's own, so the scan reads the module's gc list instead of descending. This matters outside the fuzzer: a macro that builds anExprMakeLocalcrashes a Debug build today and gets a located error instead.The fuzzer gained mutation kinds for function signatures and for arbitrarily deep type or statement nesting, and a
--mut-depthknob. Four defects in the fuzzer's own instrumentation were fixed, each of which had been producing numbers that measured nothing: two concurrent sweeps shared one victim file,--synth-skipwas inert for the main generation path, mutations aimed at a required module were never re-checked, and timeouts were judged against a baseline that excluded the macro module's own load cost.No compiler bug was found. Roughly 11,000 source-expressible mutants across Release and Debug produced zero crashes and zero assertion failures, with the detector independently proven able to fire.
Ledger
Validation
Full preflight and the AOT sweep were not run - this is a WIP branch and CI is the first full check. Run locally: rebase onto origin/master, lint and format over all changed
.das, the ast-fuzz suite (19/19), and the make-pr sync and ast-verify gates. The Debug binary used for the assertion work was built in a separate worktree, since CMake pins output to<src>/bin.The verifier check was negative-controlled: 60 real test files compiled under
--ast-verifyproduced 0 false positives, and 26 of 31 generator repros that previously asserted now report a located error.Claims - stated, not tested
The remaining 5 repros reach the compiler's own
Program::visitbefore any verifier entry runs. No daslib-side check can precede that walk; closing it would need a compiler-side pre-pass, which is not attempted here. A break would look like a Debug assert inExpression::visitwith noAST verifyline before it.The ast-verify gate skipping
utils/internal/ast-fuzz/selftest/is a pre-existing gap, not one this branch introduced:cycle.dasandalias.dasemit the same reports under the gate's own command.