fix: populate _split_overlap metadata for word/token units in RecursiveDocumentSplitter

[Osamaali313]

Related Issues

None — found while reviewing RecursiveDocumentSplitter overlap handling (a follow-up to the split_idx_start family, #11710/#11711, which fixed a different site).

Proposed Changes

_add_overlap_info computed the overlap length with self._chunk_length(prev_doc.content), which returns a word/token count for split_unit="word"/"token", while curr_pos and split_idx_start are character offsets. Mixing units made overlap_length negative, so the if overlap_length > 0 guard never fired and the _split_overlap metadata was silently left empty for word/token splitting with overlap (char units were unaffected, since there len == _chunk_length).

Fix: measure the overlap and ranges in characters via len(prev_doc.content).

# before (word unit, split_overlap=1): every chunk's _split_overlap == []
# after: overlap provenance is populated for all overlapping chunks

How did you test it?

Added test_run_split_by_dot_and_overlap_1_word_unit_split_overlap_metadata, which fails on main (empty metadata) and passes after the fix. Full test_recursive_splitter.py: 49 passed. Release note added under releasenotes/notes/.

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title
• I documented my code
• I ran pre-commit hooks and fixed any issue

---

Disclaimer: this PR was prepared with AI assistance (Claude). I reviewed the change and verified it RED→GREEN against the real component before submitting.

[vercel]

@Osamaali313 is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[CLAassistant]

All committers have signed the CLA.

[sjrl]

we use restructed text markdown for our reno files so we should make sure to use double back ticks for inline code comments.

[sjrl]

Lets remove this, we don't refer to old implementations but the current state of the code.

Suggested change

	Regression: the overlap length was computed with a word/token count while
	curr_pos/split_idx_start are character offsets, so it went negative and the
	_split_overlap metadata was silently left empty for word/token units.

[sjrl]

Hey @Osamaali313 thanks for opening the PR! Please address the failing CI issues.

[sjrl]

Lets name the test such that it represents the behavior we are expecting

Suggested change

	def test_run_split_by_dot_and_overlap_1_word_unit_split_overlap_metadata():
	def test_word_unit_split_populates_split_overlap_metadata():

[sjrl]

This is a bit wordy. Lets shorten to

Suggested change

	# curr_pos and split_idx_start are character offsets, so the overlap and
	# range must be measured in characters too. Using self._chunk_length()
	# here would mix units for word/token splitting (it returns a word/token
	# count), making overlap_length negative and silently dropping the
	# _split_overlap metadata.
	# curr_pos and split_idx_start are character offsets, so measure the
	# overlap and range in characters too (not via _chunk_length, which returns a word/token count).