feat: doc-converter + CCR — PDF/DOCX/XLSX/PPTX reading and large doc token efficiency

[stealthwhizz]

Summary

• Doc-converter — gitagent can now read PDF, DOCX, XLSX, PPTX files. Converts to clean markdown automatically on read(). Previously returned [Binary file].
• CCR (Chunked Reversible Retrieval) — documents above 8K tokens return a lightweight outline first. Agent fetches only the sections it needs via read_doc_section, preventing large docs from flooding context on every turn.
• Benchmark script — test-live-benchmark.mjs compares two successful runs: full markdown in prompt (no CCR) vs outline + fetch (with CCR). Supports Anthropic, OpenAI, and Lyzr.

What changed

File	Change
src/tools/doc-converter.ts	New — PDF/DOCX/XLSX/PPTX → markdown, 50MB file size guard, explicit errors
src/tools/doc-store.ts	New — CCR chunk store, read_doc_section with has_refs, LRU soft cap (10MB)
src/tools/read.ts	Modified — doc-converter + CCR pipeline wired in, mode="outline" dry-run
src/tools/shared.ts	Modified — mode="outline" added to read schema
src/tools/index.ts	Modified — read_doc_section tool registered with has_refs response
test-live-benchmark.mjs	New — live before/after benchmark, multi-provider

API surface (locked)

read(path, mode="outline") → outline + token estimates (dry-run, no content loaded)
read_doc_section(doc_id, section_id) → { content, tokens, has_refs: [section_id, ...] }

has_refs lets the agent decide follow-up fetches in one hop without re-parsing content.

Known limitations (v2)

• Cross-section dependency resolution — agent's responsibility for now
• Streaming / incremental conversion for very large PDFs
• Multi-user persistence across sessions
• SmartCrusher, CodeCompressor, CacheAligner — tracked in issue feat: doc-converter + CCR — PDF/DOCX/XLSX/PPTX reading and large doc token efficiency #65

Closes #65

[shreyas-lyzr]

Solid implementation — the doc-converter + CCR pipeline is well-structured and matches the agreed spec. Four issues below that need fixing before merge: one is a runtime crash, one is a key-collision bug, one is a section-ID false-positive, and one is a path mismatch that will silently break read_doc_section on relative paths. Everything else is in good shape.

[Copilot]

Pull request overview

This PR adds a document-reading and context-cost optimization pipeline to gitagent: binary office/PDF docs can be converted to readable markdown on read(), and large documents can be served via CCR (outline-first + section fetch) to avoid repeatedly flooding prompt context across turns.

Changes:

• Added a doc converter for PDF/DOCX/XLSX/PPTX (plus HTML) that outputs cleaned markdown with a 50MB guard and caching.
• Added CCR chunking + an in-session DocStore, and registered a read_doc_section retrieval tool.
• Added multiple token benchmark/proof scripts and introduced JSON/code/system-prompt “compression” utilities.

Reviewed changes

Copilot reviewed 17 out of 18 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
src/tools/doc-converter.ts	New document conversion + cleanup pipeline to markdown.
src/tools/doc-store.ts	New CCR chunk store with outline generation, section fetch, and LRU eviction.
src/tools/read.ts	Wires conversion + CCR into read(), adds caching and optional compression passes.
src/tools/shared.ts	Adds mode="outline" option to the read tool schema.
src/tools/index.ts	Registers read_doc_section tool and wires DocStore/CostTracker into read.
src/sdk.ts	Instantiates DocStore and passes it into builtin tool creation for sessions.
src/loader.ts	Builds system prompt via CacheAligner to improve KV-cache efficiency.
src/tools/cli.ts	Compresses JSON CLI output via SmartCrusher before returning to the model.
src/cost-tracker.ts	Tracks doc-conversion “token savings” at the session level.
src/compression/smart-crusher.ts	New JSON compression utility (SmartCrusher).
src/compression/code-compressor.ts	New source-code noise stripping utility (CodeCompressor).
src/compression/cache-aligner.ts	New system-prompt prefix alignment utility (CacheAligner).
src/compression/index.ts	Barrel exports for the compression utilities.
test-live-benchmark.mjs	New live benchmark script (provider-aware) comparing CCR vs no-CCR.
test-token-proof.mjs	New “meeting demo” token proof script for CCR savings.
test-compression.mjs	New benchmark covering SmartCrusher/CodeCompressor/doc-converter/CacheAligner.
package.json	Adds new dependencies for conversion + compression features.
package-lock.json	Lockfile updates for the newly added dependencies.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[shreyas-lyzr]

All four issues from the previous review are resolved:

1. PDFParse import — pdf-parse v2.4.5 exports PDFParse as a named class, so the import and constructor call are correct for the pinned version. The previous concern was based on v1 behavior; no longer applies.

2. Cache key collision — fixed with a djb2 hash appended to the path. /tmp/a/b.pdf and /tmp/a_b.pdf now produce distinct keys.

3. has_refs false positives — fixed by requiring bracket syntax [sN] and filtering against actual chunk IDs from the store.

4. Path mismatch in read_doc_section — fixed by using resolve(config.dir, path) for altPath and trying both the raw and resolved paths.

Implementation matches the agreed spec. Good to merge.