fix(extraction): replace fixed traversal stacks with growable arena-allocated stacks#217
Draft
AhmedHamadto wants to merge 1 commit intoDeusData:mainfrom
Draft
Conversation
…llocated stacks AST traversal functions use fixed-size TSNode stack[] arrays. When the DFS stack fills up, the child-push loop exits silently, dropping entire subtrees without warning. Confirmed: extracting 600 TypeScript imports returns 511 (capped at ES_IMPORT_STACK_CAP = 512). Added ts_node_stack.h — a growable stack backed by the existing arena allocator. Initial capacity matches previous fixed caps (zero extra memory for small files), doubles on overflow instead of truncating. Applied to all 14 TSNode fixed stacks across 9 extraction files. Not changed: walk_defs (uses walk_defs_frame_t, different struct type). 7 regression tests added. Full suite: 2727 passed, 0 failed. Fixes DeusData#199
Owner
|
Hey, thanks for publishing this PR. I will tackle this immediately after the 15th of April :) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
AST traversal functions in the extraction layer use fixed-size
TSNode stack[N]arrays. When the DFS stack fills up, the child-push loop exits silently via&& top < CAP, dropping entire subtrees without warning.This affects files with many top-level siblings (e.g., 600 ES imports, 130 Express routes). Reported in #199 — a Node.js routes file with ~130 routes lost data past the 512 cap.
Confirmed: extracting 600 TypeScript imports returns 511 (capped at
ES_IMPORT_STACK_CAP = 512).Fix
Added
ts_node_stack.h— a 53-line growable stack backed by the existing arena allocator. Initial capacity matches the previous fixed caps (no extra memory for small files), but doubles on overflow instead of truncating.Applied to all 14
TSNodefixed stacks across 9 extraction files:extract_calls.cwalk_callsextract_imports.cwalk_es_imports,walk_wolfram_importsextract_semantic.cwalk_throws,walk_readwritesextract_defs.cextract_elixir_call,walk_variables_iter,push_nested_class_nodesextract_usages.cwalk_usagesextract_env_accesses.cwalk_env_accessesextract_type_refs.cwalk_body_type_refs,walk_type_refsextract_type_assigns.cwalk_type_assignsextract_channels.cscan_string_consts,cbm_extract_channelsNot changed:
walk_defsuseswalk_defs_frame_t(different struct type), left as-is with its 4096 cap.Tests
Added
test_stack_overflow.cwith 7 regression tests:ts_imports_exceed_512js_calls_exceed_512python_calls_exceed_512go_calls_exceed_1024express_routes_exceed_512js_deeply_nested_callsyaml_vars_exceed_256Full suite: 2,727 passed, 0 failed.
Design notes
arena_destroyat end of file extraction). Norealloc, no mixed lifetimes.#definecaps, added growable stack header).clang-tidy/cppchecknot available on build machine — CI will validate.Fixes #199