Skip to content

fix(extract): snap docstring truncation to UTF-8 codepoint boundary (#1017)#1094

Open
harshitaajoshi wants to merge 2 commits into
DeusData:mainfrom
harshitaajoshi:fix/utf8-docstring-truncation-1017
Open

fix(extract): snap docstring truncation to UTF-8 codepoint boundary (#1017)#1094
harshitaajoshi wants to merge 2 commits into
DeusData:mainfrom
harshitaajoshi:fix/utf8-docstring-truncation-1017

Conversation

@harshitaajoshi

@harshitaajoshi harshitaajoshi commented Jul 15, 2026

Copy link
Copy Markdown

Fixes #1017

What does this PR do?

Docstring/comment truncation at MAX_COMMENT_LEN (500 bytes) in extract_comment_text() cuts at a raw byte offset, which can split a multi-byte UTF-8 character and leave an incomplete byte sequence in the stored nodes.properties JSON. This breaks any strict UTF-8 consumer reading the SQLite database (e.g. Python's sqlite3 module).

This PR snaps the truncation point back to the last complete UTF-8 codepoint boundary in extract_comment_text(), and adds the same guard to the append_json_string() escape loops in both pass_definitions.c and pass_parallel.c (kept in sync per codebase convention) as defense in depth.

A regression test (docstring_utf8_truncation_boundary_issue1017) crafts a Go comment with 3-byte CJK characters straddling the 500-byte boundary and validates every byte of the extracted docstring is well-formed UTF-8.

Checklist

  • Every commit is signed off (git commit -s) -- required, CI rejects
    unsigned commits (DCO, see CONTRIBUTING.md)
  • Tests pass locally ( make -f Makefile.cbm test )
    ASan hangs on macOS Tahoe 26.5 (Apple Clang 17) for any binary, including a trivial int main(){return 0;}. All 6294 tests pass when built without sanitizers (SANITIZE="").
  • Lint passes (make -f Makefile.cbm lint-ci)
  • New behavior is covered by a test (reproduce-first for bug fixes)

…eusData#1017)

Signed-off-by: Harshita Joshi <j.harshitaa06@gmail.com>
@harshitaajoshi harshitaajoshi requested a review from DeusData as a code owner July 15, 2026 01:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docstring truncation splits multi-byte UTF-8 characters, producing invalid UTF-8 in nodes.properties (non-ASCII comments, e.g. Japanese)

1 participant