perf: batch spaCy processing for PDF text

[ssingh231219]

Summary

Batch spaCy processing during PDFMiner-based PDF text classification.

FAST PDF partitioning currently processes each extracted text block separately. A block may invoke spaCy multiple times through sentence tokenization, word tokenization, and POS tagging.

This change preprocesses the unique text blocks from each PDF page using nlp.pipe() and reuses the resulting Doc objects in the existing tokenizer functions.

The existing element_from_text() classification logic and output order remain unchanged.

---

Key Changes

• Add batch_process_texts() and use it page by page during PDFMiner processing.
• Deduplicate text before sending it to nlp.pipe().
• Preserve lowercase POS processing for uppercase text.
• Reuse preprocessed documents in sentence, word, and POS tokenization.
• Keep batching page-local to bound memory usage.
• Preserve existing PDF coordinates, metadata, links, element order, and classification rules.
• Add regression tests and a reproducible performance benchmark.

---

Performance Benchmark

Synthetic Classification Benchmark

1,000 unique NLP-heavy elements, three iterations, batch size 256:

Mode	Median
Sequential	5.091 s
Batched	0.536 s
Speedup	9.49×

Both modes produced the same category/text fingerprint for the generated benchmark corpus.

Reproduce with:

uv run --no-sync ./scripts/performance/benchmark_text_classification.py \
  --count 1000 \
  --batch-size 256 \
  --iterations 3

Local Real PDF Validation

A model-warm, cache-cleared FAST benchmark on a 24-page PDF produced:

Version	Median	Elements
origin/main	2.279 s	201
This branch	1.496 s	201
Improvement	34.4% faster	Identical

The page/type/text/order fingerprint was identical.

---

Compatibility

• No text-classification rules were added, removed, or reordered. The existing element_from_text() function remains responsible for headers, footers, lists, addresses, narrative text, titles, and uncategorized text.
• No multiprocessing is enabled because its startup and IPC overhead made page-sized batches slower.

---

Verification & Testing

• make check: passed
• Tokenizer tests: 8 passed
• FAST PDF tests: 46 passed
• Synthetic category/text fingerprint parity: passed
• Real-PDF page/type/text/order fingerprint parity: passed

[cubic-dev-ai]

1 issue found across 5 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="unstructured/nlp/tokenize.py">

<violation number="1" location="unstructured/nlp/tokenize.py:196">
P2: A text-dense page can now hold every parsed spaCy Doc for all unique text blocks in memory simultaneously for the whole page. Before this change each `_process` call built one Doc that was consumed and discarded, so peak memory was roughly a single Doc; now the `docs` dict accumulates all of them until the context exits. The `batch_size` argument only chunks `nlp.pipe` internally and does not bound retained memory, so the PR's 'page-bounded / safe for large PDFs' memory goal isn't actually enforced by this implementation. Consider retaining and releasing Docs in chunks (still reusing them across the three tokenizer steps per chunk) so peak memory remains bounded for dense pages, and document that `batch_size` controls pipeline chunking, not per-page retention.</violation>
</file>

<sub>Shadow auto-approve: would not auto-approve because issues were found.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: A text-dense page can now hold every parsed spaCy Doc for all unique text blocks in memory simultaneously for the whole page. Before this change each _process call built one Doc that was consumed and discarded, so peak memory was roughly a single Doc; now the docs dict accumulates all of them until the context exits. The batch_size argument only chunks nlp.pipe internally and does not bound retained memory, so the PR's 'page-bounded / safe for large PDFs' memory goal isn't actually enforced by this implementation. Consider retaining and releasing Docs in chunks (still reusing them across the three tokenizer steps per chunk) so peak memory remains bounded for dense pages, and document that batch_size controls pipeline chunking, not per-page retention.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At unstructured/nlp/tokenize.py, line 196:

<comment>A text-dense page can now hold every parsed spaCy Doc for all unique text blocks in memory simultaneously for the whole page. Before this change each `_process` call built one Doc that was consumed and discarded, so peak memory was roughly a single Doc; now the `docs` dict accumulates all of them until the context exits. The `batch_size` argument only chunks `nlp.pipe` internally and does not bound retained memory, so the PR's 'page-bounded / safe for large PDFs' memory goal isn't actually enforced by this implementation. Consider retaining and releasing Docs in chunks (still reusing them across the three tokenizer steps per chunk) so peak memory remains bounded for dense pages, and document that `batch_size` controls pipeline chunking, not per-page retention.</comment>

<file context>
@@ -162,9 +168,55 @@ def _process(text: str) -> spacy.tokens.Doc:
+
+    nlp = _get_nlp()
+    prepared_inputs = tuple(_prepare_text(text, nlp) for text in pipeline_inputs)
+    docs = dict(
+        zip(
+            pipeline_inputs,
</file context>

[ssingh231219]

Thanks, good catch. PDFMiner records are now processed in chunks of BATCH_SIZE, with a separate batch_process_texts() context per chunk, so the documents are released between chunks.