Description
When the no-TOC fallback structure generation (process_no_toc → generate_toc_init/generate_toc_continue) emits multiple sibling nodes that all land on the same physical page, each sibling ends up with start_index == end_index equal to that same page. Since node text is sliced by whole page rather than by a finer offset within the page, every sibling — and their shared parent — receives the identical source text. generate_node_summary then prompts on that identical text with no reference to siblings/parent, so it produces identical (or near-identical) summaries for a parent and all of its children.
Root cause
_get_text_of_pages (pageindex/index/utils.py:339-344) slices content by whole page (start_index/end_index), not by any finer text offset within a page.post_processing (pageindex/index/utils.py:274-293) assigns start_index/end_index per node from consecutive items' physical_index — when several sibling items are all on the same page, they all get identical bounds.add_node_text (pageindex/index/utils.py:347-357) then returns byte-identical text for the parent and every affected child.generate_node_summary (pageindex/index/utils.py:192-200) summarizes purely from node['text'] with no awareness of sibling/parent overlap, so identical input → identical (or near-identical) output.
There's no de-duplication or narrower text-offset step anywhere in this path to prevent it.
Reproduction
31-page PDF with no embedded table of contents, indexed via a downstream project (OpenKB) using IndexConfig(if_add_node_text=True, if_add_node_summary=True, if_add_doc_description=True). A "Software Installation" section and its four numbered sub-items (1.1–1.4), all on the same physical page, all received the exact same generated summary paragraph verbatim — repeated 4+ times in the final tree/rendered output.
Suggested fix
Either (a) slice node text by a finer offset within a page (not just whole-page granularity) so siblings on the same page get distinct text, or (b) detect when sibling nodes have identical/near-identical source text and skip/merge redundant summarization instead of re-prompting the LLM on the same input repeatedly.
Description
When the no-TOC fallback structure generation (
process_no_toc→generate_toc_init/generate_toc_continue) emits multiple sibling nodes that all land on the same physical page, each sibling ends up withstart_index == end_indexequal to that same page. Since node text is sliced by whole page rather than by a finer offset within the page, every sibling — and their shared parent — receives the identical source text.generate_node_summarythen prompts on that identical text with no reference to siblings/parent, so it produces identical (or near-identical) summaries for a parent and all of its children.Root cause
_get_text_of_pages(pageindex/index/utils.py:339-344) slices content by whole page (start_index/end_index), not by any finer text offset within a page.post_processing(pageindex/index/utils.py:274-293) assignsstart_index/end_indexper node from consecutive items'physical_index— when several sibling items are all on the same page, they all get identical bounds.add_node_text(pageindex/index/utils.py:347-357) then returns byte-identical text for the parent and every affected child.generate_node_summary(pageindex/index/utils.py:192-200) summarizes purely fromnode['text']with no awareness of sibling/parent overlap, so identical input → identical (or near-identical) output.There's no de-duplication or narrower text-offset step anywhere in this path to prevent it.
Reproduction
31-page PDF with no embedded table of contents, indexed via a downstream project (OpenKB) using
IndexConfig(if_add_node_text=True, if_add_node_summary=True, if_add_doc_description=True). A "Software Installation" section and its four numbered sub-items (1.1–1.4), all on the same physical page, all received the exact same generated summary paragraph verbatim — repeated 4+ times in the final tree/rendered output.Suggested fix
Either (a) slice node text by a finer offset within a page (not just whole-page granularity) so siblings on the same page get distinct text, or (b) detect when sibling nodes have identical/near-identical source text and skip/merge redundant summarization instead of re-prompting the LLM on the same input repeatedly.