✨ Add TypeScript and JavaScript comment type support (issue #69)

[arnoox]

Summary

This PR implements #69 by adding TypeScript as a supported comment_type. Because the TSX grammar used to parse it is a strict superset of TypeScript, which is itself a superset of JavaScript, the same comment type also covers the JavaScript family — no separate grammar or comment_type value.

Changes

• Add ts to supported comment_type values in source discovery config.
• Map ts discovery extensions to .ts, .tsx, .mts, .cts, .js, .jsx, .mjs, .cjs (.mts/.cts are TypeScript's ESM/CJS module variants).
• Add tree-sitter TypeScript dependency and parser wiring in analyse utilities.
• Add TypeScript scope/comment parsing tests.
• Add discovery fixture coverage across the whole extension family.
• Add integration analyse test coverage for TypeScript input.
• Add declarative extraction fixture cases + snapshots (default_oneliner_typescript, jsx_oneliner_tsx) so the extraction contract is pinned by captured reference output.
• Add FE_TS feature and the corresponding impl markers for traceability.
• Update docs for supported languages and discover examples.
• Add changelog entry under Upcoming.

Verification

Ran:

• tox -e py312-sphinx8-needs5

Result:

• 366 passed, 1 skipped (60 snapshots)

Linked issue

• Closes Feature: Add TypeScript language support #69

[patdhlk]

The TSX grammar is not a superset of the TypeScript grammar — .ts files using angle-bracket type assertions silently lose all markers after the first cast.

TypeScript deliberately forbids <T>expr assertions in .tsx because they are ambiguous with JSX; that is exactly why tree-sitter-typescript ships two grammars. Under language_tsx() a plain-TS cast becomes an unterminated JSX element and the rest of the file collapses into one ERROR node whose contents are lexed as jsx_text — so the comments in it are never captured at all.

Reproduced against this branch:

// @m1
function a() {}

const x = <string>value;   // legal .ts, illegal .tsx

// @m2
function b() {}

// @m3
class C {}

language_tsx():         has_error=True   comments found = 1   (@m2, @m3 silently gone)
                        tree: (program (comment) (function_declaration ...) (ERROR ...))
language_typescript():  has_error=False  comments found = 3

This is silent data loss (FAULT_TS_1), not a degraded scope. The same false "strict superset" claim is repeated in source_discover/config.py:14, docs/source/components/features.rst:278, and the changelog, so it will be trusted by future readers.

Fix: pick the grammar from the file suffix — language_tsx() for .tsx/.jsx, language_typescript() for .ts/.mts/.cts/.js/.mjs/.cjs. That requires init_tree_sitter to receive the path (or SourceAnalyse to cache one parser per grammar), which is the deeper fix the "one grammar for everything" shortcut is avoiding.

[patdhlk]

SCOPE_NODE_TYPES[CommentType.ts] omits TypeScript's type-level declarations, so a marker documenting one mis-associates with the next unrelated declaration.

find_next_scope walks forward until something matches, so an unmatched declaration is not "no scope" — it is a wrong scope. Verified on this branch (find_associated_scope(comment, CommentType.ts)):

source	reported tagged_scope
interface Foo {…} then function unrelated(){}	function unrelated() {}
enum E { A } then function unrelated(){}	function unrelated() {}
type T = number; then function unrelated(){}	function unrelated() {}
abstract class Base {} then function unrelated(){}	function unrelated() {}
function* gen(){} then function unrelated(){}	function unrelated() {}

That is FAULT_TS_2 ("hallucinates traceability objects") for five of the most common TS declaration forms. abstract_class_declaration is the starkest: class_declaration is in the set, so class Foo {} works and abstract class Foo {} silently points at the wrong code.

Missing node types: interface_declaration, enum_declaration, type_alias_declaration, abstract_class_declaration, generator_function_declaration. Compare CommentType.rust, which does include struct_item/enum_item/trait_item.

[patdhlk]

TS_FUNCTION_VALUE_TYPES is too narrow — generator, class, and cast-wrapped values fall through the new guard and the comment jumps to an unrelated declaration.

The guard correctly stops a plain const from stealing the association, but everything it fails to recognise is treated as a plain const, so find_next_scope keeps walking. Verified:

// @m
const gen = function* () {};      // value type: generator_function
function unrelated() {}           // -> tagged_scope = "function unrelated() {}"

// @m
const A = class {};               // value type: class
function unrelated() {}           // -> tagged_scope = "function unrelated() {}"

// @m
const f = (() => {}) as Handler;  // value type: as_expression
function unrelated() {}           // -> tagged_scope = "function unrelated() {}"

Suggest adding generator_function and class, and unwrapping the TS expression wrappers (as_expression, satisfies_expression, parenthesized_expression, non_null_expression) before the type test — const f = (() => {}) as Handler is idiomatic typed-callback style.

Suggested change

	TS_FUNCTION_VALUE_TYPES = {"arrow_function", "function_expression"}
	TS_FUNCTION_VALUE_TYPES = {
	"arrow_function",
	"function_expression",
	"generator_function",
	"class",
	}
	# expression wrappers to look through before testing the value type
	TS_VALUE_WRAPPER_TYPES = {
	"as_expression",
	"satisfies_expression",
	"parenthesized_expression",
	"non_null_expression",
	}

[patdhlk]

This marker is in the wrong field order for the default one-line style, so the new integration test asserts on a garbage need.

OneLineCommentStyle() defaults to needs_fields = [title, id, type(default impl), links(default [])]. // @type,TS_REQ_002,TypeScript one-line test therefore parses as:

{'title': 'type', 'id': 'TS_REQ_002', 'type': 'TypeScript one-line test', 'links': []}

(actual output from running SourceAnalyse on this file). The need type becomes the free-text string TypeScript one-line test, which is not a registered need type, and no link is produced. demo.tsx has the same problem (type: 'TypeScript JSX component test').

num_oneline_needs: 1 passes regardless, so the "integration analyse test coverage for TypeScript" never exercises the canonical marker shape that every other language fixture uses (tests/data/rust/demo.rs, tests/data/jsonc/demo.jsonc):

Suggested change

	// @type,TS_REQ_002,TypeScript one-line test
	// @TypeScript one-line test, TS_REQ_002, impl, [REQ_TS_001]

…and the links assertion is worth adding so the shape is actually pinned.

[patdhlk]

This comment's claim is false: nothing in the suite pins the production grammar choice.

the TSX-grammar choice is pinned by test_analyse_utils.py's has_error check instead

test_find_associated_scope_typescript_jsx_no_parse_error parses with the init_typescript_tree_sitter fixture, which builds Language(tree_sitter_typescript.language_tsx()) itself (tests/test_analyse_utils.py:68). It never calls utils.init_tree_sitter(CommentType.ts), so it asserts that tree-sitter's TSX grammar parses JSX — a fact about tree-sitter, not about this code.

Verified: swapping utils.init_tree_sitter to language_typescript() keeps every new test green. tests/data/typescript/demo.tsx is the only fixture that goes through production wiring, and its assertions (num_comments: 1, num_oneline_needs: 1) hold under both grammars, because the leading // comment is still captured from the error tree:

demo.tsx  tsx: has_error=False  comments=1
demo.tsx  ts:  has_error=True   comments=1   <- assertions still pass

Either build the fixture from utils.init_tree_sitter(CommentType.ts) so the tests exercise the real wiring, or add an assertion on the grammar actually selected.

[patdhlk]

JavaScript's dominant function-definition forms are not scope types, so markers on them mis-associate — and .js/.cjs/.mjs are newly auto-discovered by this PR.

The scope set covers only ES declaration syntax. Assignment-based definitions — the norm in CommonJS, which is precisely what .cjs exists for — produce assignment_expression, which is not handled at any level:

// @m
module.exports = function f(){};
function unrelated(){}            // -> tagged_scope = "function unrelated(){}"

// @m
Foo.prototype.bar = function(){};
function unrelated(){}            // -> tagged_scope = "function unrelated(){}"

Anonymous default exports (the standard shape for React/Next.js page modules) get no scope at all:

// @m
export default () => {};          // -> tagged_scope = None
// @m
export default function () {};    // -> tagged_scope = None

Widening comment_type = "ts" to the whole JS family (config.py:17) advertises support for these files; the scope table should cover their idioms, or features.rst should state the limitation explicitly.

[patdhlk]

Class-field arrow functions resolve to the enclosing class instead of the field.

public_field_definition is missing, so the very common React/Angular class-property handler form loses precision:

class A {
  // @m
  handler = () => {};
}

find_next_scope finds nothing (the sibling is a public_field_definition), then find_enclosing_scope walks up to class_declaration, so tagged_scope becomes the entire class body (verified: class A {\n // @m\n handler = () => {};\n…) rather than the field the comment documents. method_definition is already handled, so getters/methods are fine — this is the field-assigned-arrow gap, which is exactly the case the new _is_function_like_lexical_declaration logic was written to handle at statement level.

[patdhlk]

.d.ts declaration files are pulled into discovery by the ts suffix but no ambient node type is a scope, so every marker in them mis-associates or gets a null scope.

SourceDiscover matches on filepath.suffix.lower(), and Path("api.d.ts").suffix == ".ts" — so declaration files are discovered with no way to opt out except a user-supplied exclude. Their entire content is ambient declarations, none of which SCOPE_NODE_TYPES[CommentType.ts] recognises:

declare function f(): void;   // ambient_declaration > function_signature -> None
declare module "x" { }        // ambient_declaration > module              -> None
interface I { doThing(): void; }  // -> None

Either add ambient_declaration / function_signature / module to the scope table, or exclude .d.ts from discovery under comment_type = "ts" — the latter mirroring the existing _json_starts_with_comment gate that keeps plain .json out of the jsonc type.

[patdhlk]

A TypeScript-only container type is hardcoded into the shared traversal, applied to every language.

{"block", "export_statement"} now mixes a C/C++ node type with a TS/JS one in a single set consulted for all nine comment types, and _matches_scope (line 223) hardcodes comment_type == CommentType.ts inside an otherwise language-agnostic helper. The module already has the right mechanism for this — the per-language SCOPE_NODE_TYPES table, whose header comment explains carefully which languages participate and why.

Suggest a parallel per-language table so the next language adds a dict entry instead of another or-clause in shared code, e.g. SCOPE_CONTAINER_TYPES: dict[CommentType, set[str]] (cpp → {"block"}, ts → {"block", "export_statement"}) plus SCOPE_PREDICATES: dict[CommentType, dict[str, Callable[[Node], bool]]] for the lexical_declaration refinement. As written, TS's other wrapper (ambient_declaration) is already missed by the same shortcut.

[patdhlk]

Sixth verbatim copy of the same query string.

CPP_QUERY, C_SHARP_QUERY, YAML_QUERY, JSONC_QUERY, BASH_QUERY and now TYPE_SCRIPT_QUERY are all exactly """(comment) @comment""". One shared constant (SIMPLE_COMMENT_QUERY) referenced by the six branches would remove the copy without losing the per-language traceability marker, which can stay on the init_tree_sitter branch.

Relatedly, the if/elif chain in init_tree_sitter is now nine branches of identical shape (import, Language(...), Query(...)); a dict[CommentType, tuple[str, str]] of (module_name, query) driven by importlib.import_module would make the whole function ~6 lines and make the "add a language" edit a single-line data change.

[patdhlk]

Adding one language now requires five unsynchronised edits, and divergence fails with a bare KeyError.

ts had to be added to: the CommentType enum (here), COMMENT_FILETYPE (line 17), SCOPE_NODE_TYPES (analyse/utils.py:31), a new *_QUERY constant (utils.py:78), and an elif branch in init_tree_sitter (utils.py:137). Nothing enforces the join:

• COMMENT_FILETYPE is keyed by raw str, not CommentType, so an enum member without a dict entry blows up as KeyError in SourceDiscover.__init__ (line 37) with no diagnostic.
• Conversely init_tree_sitter raises ValueError, and a missing SCOPE_NODE_TYPES entry silently falls back to the C++ scope set (utils.py:236) rather than erroring — a wrong-language default.

A single per-language record (extensions + grammar module + query + scope types), with the enum derived from it, would collapse the five edits into one and make the fallback impossible. At minimum, a test asserting set(COMMENT_FILETYPE) == {c.value for c in CommentType} and that every member has a SCOPE_NODE_TYPES entry would catch the divergence.

[patdhlk]

The advertised {/* … */} JSX capability does not work in its idiomatic single-line form — it produces a warning and no need.

Inside JSX children // is impossible, so {/* … */} on one line is the way to comment there. With the documented default style it silently fails (run against this branch):

const App = () => (
  <div>
    {/* @Tsx Title, IMPL_TSX, impl, [REQ_TSX] */}
  </div>
);

oneline_needs: []
warnings:      ['not_start_or_end_with_square_brackets']

The default end_sequence is \n, so the trailing */} is swallowed into the links field and the bracket check rejects it. The same applies to the plainly-documented /* */ style for .ts (analyse.rst:50): /* @Blk Title, IMPL_BLK, impl, [REQ_BLK] */ yields no need either.

tests/data/extraction/oneline.yaml acknowledges this and works around it by putting the marker on its own line inside the block, explicitly "deliberately not exercised by the shared fixtures". That leaves the newly advertised feature's most common shape both broken and untested. Either handle a block-comment terminator as an implicit end sequence, or narrow the docs to say markers must be on their own line inside a block/JSX comment.

[patdhlk]

This describes an architecture that does not exist, and points future agents at a recipe that cannot be followed.

analyse/projects.py is 78 lines containing a single AnalyseProjects class. There is no LANGUAGE_ANALYZERS dict and no per-language analyzer class anywhere in the repo:

$ grep -rn LANGUAGE_ANALYZERS src/    # no matches

Line 66 then tells the reader that adding a language "follow[s] a short recipe documented in AGENTS.md ... follow those rather than inventing a new approach" — but that recipe (AGENTS.md:400-417) instructs creating a BaseAnalyzer subclass and registering it in LANGUAGE_ANALYZERS, which is fiction. This PR itself could not follow it; TypeScript support landed via SCOPE_NODE_TYPES + init_tree_sitter in analyse/utils.py, which the new file never mentions.

Also: line 13's language list omits Bash, which shipped in 1.4.0.

Since a wrong CLAUDE.md actively misdirects, it is worth either correcting these two sections against analyse/utils.py, or dropping the architecture section and deferring to AGENTS.md. Separately, adding CLAUDE.md is unrelated to issue #69 and would be easier to review as its own PR.

[patdhlk]

Heading deviates from the established convention.

Every prior pre-release cycle in this file used Unreleased — e.g. at adf35cc the in-progress section was:

Unreleased
----------

The PR description also says the entry was added "under Upcoming", so the actual heading matches neither. Suggest Unreleased for consistency with the release tooling and history.

Suggested change

	Under development
	Unreleased
	----------