feat(markdown): add CommentType.markdown backed by tree-sitter-markdown

[HartmannNico]

Closes #94

Summary

Adds comment_type: markdown backed by tree-sitter-markdown, enabling <!-- @needs … --> HTML-comment markers in Markdown files to be ingested as traceability nodes by sphinx-codelinks.

Primary use case: AI agent and skill definition files (.claude/agents/*.md, .github/prompts/*.md) that are the actual implementation artefacts of agentic systems and need to participate in a requirements → architecture → design → implementation traceability chain.

Changes (3 files, ~24 lines)

source_discover/config.py

• Added "markdown": ["md", "markdown"] to COMMENT_FILETYPE
• Added markdown = "markdown" to CommentType enum

analyse/utils.py

• Added MARKDOWN_QUERY = """(html_block) @comment""" — tree-sitter-markdown maps standalone <!-- … --> blocks to html_block nodes
• Added elif comment_type == CommentType.markdown: branch in init_tree_sitter() importing tree_sitter_markdown
• No SCOPE_NODE_TYPES entry: Markdown has no function/class scopes; oneline-only mode never invokes scope association

pyproject.toml

• Added tree-sitter-markdown>=0.5.1

Caller note

Because html_block node text includes the full <!-- … --> delimiters, callers must configure end_sequence: " -->" (not the default "\n") in their oneline_comment_style. This is the only behavioural difference from all other comment_type entries.

Example conf:

"my_agents": {
    "source_discover": {"src_dir": ".claude/agents", "comment_type": "markdown"},
    "analyse": {
        "get_oneline_needs": True,
        "get_need_id_refs": False,
        "oneline_comment_style": {
            "needs_fields": [...],
            "start_sequence": "<!-- @needs ",
            "end_sequence": " -->",
        },
    },
}

Relation to other PRs

Analogous to PR #92 (bash). PR #82 (TypeScript) was the structural template.
This PR is independent of PR #92 — it is based directly on main and does not require bash support to be merged first.

[ubmarco]

Nice, I will check this out. We are building the features in codelinks in sync with ubCode, so bear with me a little while I do that :)

[ubmarco]

I approved the CI, it fails.

[HartmannNico]

pushed tests and doc as well. used pr #92 bash as template.

[patdhlk]

(html_block) captures every block-level HTML element, not just HTML comments.

html_block is the CommonMark HTML block node: <div>, <details>, <table>, <script>, <br/> — all of them are handed to the extractor as "comments". Verified against tree-sitter-markdown==0.5.1:

<details>
<summary>@need-ids: FAKE_1</summary>
body
</details>

get_need_id_refs defaults to True in SourceAnalyseConfig (and the Sphinx path only force-sets get_oneline_needs), so the above yields a need-ID reference with id FAKE_1</summary> — a phantom link with HTML markup baked into the ID, which is exactly FAULT_MARKDOWN_2. Same for <script>var x = 1; // @need-ids: REQ_9</script> → ref REQ_9.

A #match? predicate fixes it (verified working on the pinned tree-sitter~=0.25.1):

Suggested change

	MARKDOWN_QUERY = """(html_block) @comment"""
	MARKDOWN_QUERY = r"""((html_block) @comment (#match? @comment "^[ \t]*<!--"))"""

[patdhlk]

Mandating end_sequence: " -->" silently opts Markdown out of the #88 prose anchor, so HTML comments containing prose hallucinate needs.

oneline_parser only anchors a marker to the start of the comment when end_sequence == UNIX_NEWLINE:

if oneline_config.end_sequence == UNIX_NEWLINE and any(
    char.isalnum() for char in oneline[:start_idx]
):
    return None

With " -->" that guard never runs, so every @ anywhere inside an HTML comment is parsed. Verified end-to-end:

• <!-- reviewers: @alice, @bob, please check --> → need {title: "alice", id: "@bob", type: "please check"} — a bogus ID that raises InvalidNeedException in Sphinx-Needs (the exact failure the 1.4.0 "Anchor newline-terminated one-line markers" fix was written for).
• <!-- @format --> (the Prettier pragma, common in Markdown) → too_few_fields warning, which fails sphinx-build -nW.

Markdown files are the most prose-heavy input this project accepts and .md discovery sweeps every README.md/CHANGELOG.md under src_dir, so this is a high-volume false-positive source. Consider extending the anchor check to treat a delimiter-shaped end_sequence as line-anchored too, or strip <!--/--> in the Markdown path so the default newline end_sequence keeps working.

[patdhlk]

end_sequence: " -->" requires exactly one space before -->; ...--> silently extracts nothing.

<!-- @A Title, IMPL_A, impl, [R1]--> is valid HTML and a very common way to write the closing delimiter, but oneline.rfind(" -->") returns -1, so oneline_parser returns None: zero needs, zero warnings, no diagnostic. Verified against the real pipeline.

Same class of silent loss when two markers share a line: <!-- @A ... --> <!-- @B ... --> → rfind takes the last -->, both comments merge into one field list and you get a spurious too_many_fields warning instead of two needs.

Worth a fixture for the no-space form so the sharp edge is at least pinned by a test.

[patdhlk]

Multi-line HTML comment blocks extract nothing under the mandated end_sequence: " -->".

The natural way to declare several Markdown markers is one HTML comment holding several lines:

<!--
@A Title, IMPL_A, impl, [R1]
@B Title, IMPL_B, impl, [R2]
-->

extract_oneline_need parses the html_block text line by line; no inner line contains " -->" and the closing --> line contains no @, so the result is 0 needs and 0 warnings. With the default end_sequence: "\n" the same input yields both needs correctly (verified).

So the two Markdown comment forms are mutually exclusive per project config — one-line markers need " -->", multi-line blocks need "\n" — and picking the documented one makes the other fail silently. This is the real argument for handling the <!-- … --> delimiters inside the Markdown path instead of pushing end_sequence onto callers.

[patdhlk]

Nothing enforces the end_sequence requirement, so the default config silently corrupts need fields.

A user who configures only comment_type = "markdown" keeps end_sequence = "\n", and generate_project_configs force-enables get_oneline_needs. Verified results:

• <!-- @Md Title, IMPL_MD, impl --> → need with type == "impl -->" — silently corrupted, no warning.
• <!-- @Md Title, IMPL_MD, impl, [REQ_MD] --> → not_start_or_end_with_square_brackets warning, marker dropped.

A comment in COMMENT_FILETYPE and a .. note:: in the docs are the only things standing between a user and corrupted traceability data. This deserves enforcement in code rather than prose — e.g. a check in SourceAnalyseConfig.check_fields_configuration() that errors when comment_type is CommentType.markdown and oneline_comment_style.end_sequence == UNIX_NEWLINE, or a per-language default end_sequence.

[patdhlk]

This justification is factually wrong: scope association is invoked for Markdown, on every comment, regardless of those flags.

SourceAnalyse.extract_marked_content calls it unconditionally before looking at either flag (analyse.py):

tagged_scope = utils.find_associated_scope(
    src_comment.node, self.analyse_config.comment_type
)
if self.analyse_config.get_need_id_refs:
    ...
if self.analyse_config.get_oneline_needs:
    ...

Because CommentType.markdown is absent from this table, find_next_scope/find_enclosing_scope silently fall back to the C++ scope set:

scope_types = SCOPE_NODE_TYPES.get(comment_type, SCOPE_NODE_TYPES[CommentType.cpp])

so every html_block is walked to the end of the document looking for function_definition/class_definition. It returns None only because the Markdown grammar happens to have no node with those names — correct by luck, not by design, and O(blocks²) on a Markdown-heavy tree. An explicit CommentType.markdown: set() entry (or short-circuiting find_associated_scope for Markdown, as it already does for YAML/JSONC) would be both honest and cheaper.

[patdhlk]

This claim is inaccurate and understates the false-positive surface.

"Because tree-sitter-markdown exposes standalone HTML comments as html_block nodes, only block-level <!-- … --> markers are captured" — html_block is not comment-specific. <div>, <details>, <table>, <script> and <br/> blocks are all captured and scanned for markers (verified). What's true is the narrower statement that only block-level HTML is captured; inline HTML is not.

Either fix the query (see the comment on MARKDOWN_QUERY) and keep this sentence, or state plainly that all block-level HTML is scanned so users understand why <details> sections can produce phantom need IDs.

[patdhlk]

The added tests cover only the happy path; every sharp edge of this feature is untested.

AGENTS.md: "Test coverage: Write tests for all new functionality and bug fixes." The four behaviours most likely to bite users all pass through untested:

1. a non-comment html_block (<div>, <details>, <script>) being captured as a comment;
2. [R1]--> (no space before the delimiter) silently extracting nothing;
3. a multi-line <!-- … --> block extracting nothing under the documented end_sequence;
4. the default end_sequence leaking " -->" into a str field.

Cases 2–4 are one-line additions to tests/data/extraction/oneline.yaml; case 1 fits here as an expected_count row.

[patdhlk]

Nits in the new tests.

• @pytest.mark.parametrize("code", [ ...one value... ]) over a single input adds a parametrize id for no benefit — inline the bytes literal.
• comments = utils.extract_comments(...) or [] in test_extract_comments_markdown masks the return contract: extract_comments returns list | None, and the expected_count == 0 case passes whether it returns None or []. Assert the shape you expect instead of coercing it.
• init_markdown_tree_sitter re-implements utils.init_tree_sitter(CommentType.markdown) (same language, same query); the second test already proves the real factory works, so the fixture only adds a second construction path that can drift.

[patdhlk]

Commit and PR titles don't follow the documented format.

AGENTS.md, Commit Message Format: <EMOJI> <KEYWORD>: Summarize in 72 chars or less (#<PR>) with keywords ✨ NEW: / 🐛 FIX: / 🧪 TEST: …, and PR Title and Description Format: "Use the same as for the commit message format, but for the title you can omit the KEYWORD and only use EMOJI".

This branch uses Conventional Commits instead:

• 1c0402c feat(markdown): add CommentType.markdown backed by tree-sitter-markdown
• 09585c9 test(markdown): add tests, fixtures, docs following PR #92 pattern
• PR title: feat(markdown): add CommentType.markdown backed by tree-sitter-markdown

Also, 1c0402c's body still carries the unfilled placeholder Closes #NNN. (the PR description says Closes #94).

[patdhlk]

Auto-discovering .md makes source-page names collide with real documents.

generate_code_page derives the generated page name by stripping the suffix:

pagename = str((file_path.relative_to(app.outdir)).with_suffix(""))

Until now no discoverable extension was also a Sphinx source extension. With .md discoverable, tracing README.md yields pagename README; in a MyST project (or any project with .md documents), html-collect-pages runs after the normal pages are written, so the source-tracing page silently overwrites the real document's HTML. Worth either documenting "don't point src_dir at your doc sources" or namespacing the generated pagename.

[patdhlk]

Ninth branch on a chain that a table would replace.

Adding a language now means touching five places: COMMENT_FILETYPE, CommentType, SCOPE_NODE_TYPES, a *_QUERY constant, and this if/elif chain — and the chain is at 10 branches against ruff's PLR0912 limit of 12, so the next two languages break it.

A single registry keeps the per-language knowledge in one row and makes "no scope types" an explicit value rather than a missing key:

LANGUAGE_REGISTRY = {
    CommentType.markdown: ("tree_sitter_markdown", MARKDOWN_QUERY, frozenset()),
    ...
}

with init_tree_sitter doing one importlib.import_module + Query(...) lookup. Not required for this PR, but this is the third language added on the same chain.