fix(pdf): read a comment as white space, and half a pair as one mark - #688
Merged
Conversation
Four new pdfs in the private corpus rendered as nothing at all. Three ended in `Invalid UTF-16`: a code the `ToUnicode` cmap does not map falls back to its own numeric value as a utf-16 unit, and a value in the surrogate range is not a code point, so converting the run threw and took the whole document with it. `u16string_to_string` now spends a replacement mark on a surrogate that completes no pair, wherever the half pair came from - the same throw was reachable from every engine that decodes utf-16. The fourth carries `%<ODT_ATTRIBUTION_LABEL_SIGNAL>` in its content stream. A comment runs from `%` to the end of the line and stands wherever white space may (7.2.4), but the parser read the `<` as the start of a hex string and threw on the first letter that is not a hex digit. Objects and content-stream operators now skip comments. The file structure reads its `%PDF-` and `%%EOF` markers as entries of its own, which is why this is a separate call and not `skip_whitespace` itself. The four pdfs and an odt template join the corpus with their reference output; no existing reference file changes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J8PCMZUVxuGstmmgQzEdif
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1cf5765f0c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Three gaps the first pass left, all from review: `%` is a delimiter (7.2.2), so `Tj%note` is an operator and a comment rather than one bareword that swallows the operation; the separators inside an indirect reference may carry one, so `5 0%note\nR` resolves; and the gaps around an indirect object's body may too, so a commented catalog or page still parses. The entry-level skips stay blind, which is what keeps `%PDF-` and `%%EOF` readable as entries of their own. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J8PCMZUVxuGstmmgQzEdif
andiwand
force-pushed
the
fix/pdf-content-stream-comments-and-utf16
branch
from
August 14, 2026 17:39
5b1af65 to
ccc1b81
Compare
This was referenced Aug 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 Generated with Claude Code
Five files new to the corpus — four pdfs and an odt template. Every one of the pdfs rendered as nothing at all, in two different ways.
Invalid UTF-16(HUK_Fahrrad-Schutzbrief,Laserentfernungsmesser_PLEM50_D5,Thule_E-Flexi_G2_959). A code theToUnicodecmap does not map falls back to its own numeric value as a single utf-16 unit. A two-byte code landing inD800–DFFFis then half a surrogate pair, which is not a code point, soutf16to8threw and one bad text run cost the whole document.u16string_to_stringnow spends a replacement mark on a surrogate that completes no pair — the same throw was reachable from every engine that decodes utf-16, not just this fallback.invalid character in hex_char_to_int(FNIRSI_2C53T_de). Its content stream carries%<ODT_ATTRIBUTION_LABEL_SIGNAL>. A comment runs from%to the end of the line and stands wherever white space may (7.2.4), but the parser read the<as the start of a hex string and threw on the first letter that is not a hex digit. Objects and content-stream operators now skip comments. The file structure reads its own%PDF-and%%EOFmarkers as entries, so this isskip_whitespace_and_commentsrather than a change toskip_whitespaceitself.Test data: the input files and their reference output are on
mainof the four data repos, andtest/data.cmakemoves to those pins. A full local run differs from the previous reference set only by the five new directories — no existing reference file changes.Not addressed here: two of the pdfs extract garbled text where their fonts map no unicode (two ascii bytes packed into one utf-16 unit read as CJK). That is the pre-existing shape of the fallback, and their reference output records it as it stands today.