fix: keep inter-sentence whitespace when a sentence ends with a closing quote

[winklemad]

Related Issues

• No existing issue — I found this while checking that sentence chunks still map back onto the source text. Happy to open one if you'd rather track it separately.

Proposed Changes:

SentenceSplitter(keep_white_spaces=True) silently drops the whitespace between two sentences when the first one ends with a closing quote:

from haystack.components.preprocessors.sentence_tokenizer import SentenceSplitter

splitter = SentenceSplitter(keep_white_spaces=True)
text = 'He said "Hello there." Then he left.'
spans = splitter.split_sentences(text)

[(s["start"], s["end"]) for s in spans]     # [(0, 22), (23, 36)]  <- char 22 is in no span
"".join(s["sentence"] for s in spans)       # 'He said "Hello there."Then he left.'

Root cause. CustomPunktLanguageVars.period_context_re extends SentEndChars with closing brackets only (self._re_sent_end_chars + r"[\)\]}]*"). For ." the whitespace sits behind the closing quote, so the \s* in _period_context_fmt never reaches it; nltk's boundary realignment then ends the sentence at the quote but moves the next break past the whitespace, and those characters end up in none of the returned spans. He said (hi.) Then left. round-trips correctly today, which is exactly the bracket case that is already covered — closing quotes are simply missing from that character class.

Impact. The characters are lost from the chunk text and every following split_idx_start shifts, so chunks no longer index back into the source. This reaches DocumentSplitter (split_by="sentence", and split_by="word" with respect_sentence_boundary=True), RecursiveDocumentSplitter, MarkdownHeaderSplitter and EmbeddingBasedDocumentSplitter — any prose with a quoted sentence. It also breaks the assumption stated in embedding_based_document_splitter.py that "chunks are contiguous substrings of the original text ... so the character offset is simply accumulated".

Fix. Match closing quotes as well as closing brackets after a sentence ending. Because a span now carries the trailing whitespace, _needs_join compares the split rules against the end of the sentence itself (text[start:end].rstrip()); without that line the existing "a cited question is not a sentence boundary" rule would silently stop firing, so there is a test guarding it.

Chunk boundaries change for text containing quoted sentences, so re-indexing an existing corpus produces different chunks — this is called out in the release note.

I deliberately did not touch recursive_splitter.py: the overlap problem in #12281 / #12284 has a different root cause and is already being worked on.

How did you test it?

Unit tests, added first and confirmed failing on unmodified code (7 failed, 2 passed), then passing with the fix (9 passed):

• test_split_sentences_keeps_white_spaces_after_a_closing_quote — parametrized over ", ', “ ”, ‘ ’, « » and !", asserting no character is lost and that the spans still tile the text. It includes the bracket case He said (two.) Three. as a control that already passed before the fix.
• test_split_sentences_keeps_a_cited_question_joined — guards the existing split rule for cited questions.
• test_run_split_by_sentence_quoted_text_keeps_offsets_aligned — checks at DocumentSplitter level that every split_idx_start still indexes its own chunk in the source text.

Also ran hatch run test:unit test/components/preprocessors/ (367 passed, 358 before the new tests), the full hatch run test:unit, hatch run test:types (clean) and hatch run fmt. Since the pattern is version-sensitive, I re-ran the preprocessor tests against the declared floor nltk==3.9.1 as well — 73 passed there too.

Notes for the reviewer

• The closer set is ) ] } " ' ’ ” ». If apostrophe-heavy languages are a concern I'm happy to drop ' from it — the remaining characters still fix the common ." and ” cases, just say the word and I'll push the change.
• I kept the set to actual closing characters. nltk's own re_boundary_realignment also lists the opening ‘ “ «; including them made no difference on any case I tried, so I left them out.
• The one-line change in _needs_join is required by the fix, not a drive-by: it keeps the existing quote/numeration rules working now that a span can end with whitespace.

Checklist

• I have read the contributors guidelines and the code of conduct.
• I have updated the related issue with new insights and changes.
• I have added unit tests and updated the docstrings.
• I've used one of the conventional commit types for my PR title: fix:.
• I have documented my code.
• I have added a release note file, following the contributors guidelines.
• I have run pre-commit hooks and fixed any issue.

[vercel]

@winklemad is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
haystack/components/preprocessors
sentence_tokenizer.py
Project Total

This report was generated by python-coverage-comment-action

[bogdankostic]

Thanks for your PR @winklemad! I found a case where the changes alter the behavior that is intended, I left an in-line comment about this.

Also, I'm interested, did you run into the splitting problem that your PR should fix yourself while processing your own documents?

[bogdankostic]

This moves the boundary such that the rule documented three lines down — # sentence.", sentence -> no split — stops firing:

s = SentenceSplitter(language="en", keep_white_spaces=True)
[x["sentence"] for x in s.split_sentences('He said "Hi.", then left.')]
# main:    ['He said "Hi.", then left.']
# this PR: ['He said "Hi."', ', then left.']

Same for ."+;/:/— and the 'Hi.', form. Via DocumentSplitter you get a chunk starting with a comma.

Why. The quote span is (8, 13). On main the boundary lands at 12, inside the quote, so quote_start < end < quote_end joins. The widened closer class moves it to 13 — exactly quote_end — where only the ? rule applies. rstrip() can't help: there's no whitespace to strip, the next char is ,.