Add anydoc file input processing

Cargo.toml

@@ -4,7 +4,7 @@ description = "A minimalist command-line utility to pipe documents from a file o
 repository = "https://github.com/VimCommando/espipe"
 homepage = "https://github.com/VimCommando/espipe"
 documentation = "https://docs.rs/crate/espipe"
-version = "0.4.0"
+version = "0.5.0"
 edition = "2024"
 rust-version = "1.88"
 license = "Apache-2.0"
@@ -19,12 +19,14 @@ include = [
     "README.md",
     "src/**/*.rs",
     "tests/**/*.rs",
+    "tests/fixtures/anydoc/*",
     "tests/fixtures/*.toon",
     "tests/fixtures/*.ndjson",
     "tests/fixtures/*.ndjson.gz",
 ]
 
 [dependencies]
+anydoc = "0.1.8"
 base64 = "0.22.1"
 clap = { version = "^4.6.1", features = ["derive"] }
 csv = "^1.4.0"

README.md

@@ -65,6 +65,9 @@ To build and publish a multi-platform Docker Hub image:
 - `.json` files
 - `.csv` files
 - `.csv.gz` files
+- local Markdown and text files
+- local PDF, Word, PowerPoint, Excel, OpenDocument, RTF, and EPUB files
+- local recursive glob patterns for file documents
 - `stdin` as NDJSON
 
 It writes records to:
@@ -78,11 +81,10 @@ When writing to Elasticsearch, `espipe` batches documents into groups of 5,000 r
 ## CLI Reference
 
 ```bash
-Usage: espipe [OPTIONS] <INPUT> <OUTPUT>
+Usage: espipe [OPTIONS] <PATHS>...
 
 Arguments:
-  <INPUT>   The input URI to read docs from
-  <OUTPUT>  The output URI to send docs to
+  <PATHS>...  Input URI(s) followed by the output URI
 
 Options:
   -k, --insecure                     Ignore certificate validation
@@ -91,6 +93,7 @@ Options:
   -p, --password <PASSWORD>          Password for basic authentication
   -q, --quiet                        Quiet mode, don't print runtime summary
   -z, --uncompressed                 Disable request body gzip compression
+      --content <CONTENT>            Content subfield name for file imports [default: body]
       --action <ACTION>              Bulk action for Elasticsearch outputs [default: create] [possible values: create, index, update]
       --batch-size <BATCH_SIZE>      Documents per Elasticsearch bulk request [default: 5000]
       --max-requests <MAX_REQUESTS>  Maximum concurrent Elasticsearch bulk requests [default: 16]
@@ -99,7 +102,8 @@ Options:
 
 ## Input And Output
 
-Both positional arguments are parsed as URI-like strings.
+All positional arguments are parsed as URI-like strings; one or more input URIs
+are followed by the output URI.
 
 ### Supported input forms
 
@@ -125,8 +129,24 @@ Both positional arguments are parsed as URI-like strings.
   Reads CSV from a `file://` URI.
 - `file:///absolute/path/to/file.csv.gz`
   Reads gzip-compressed CSV from a `file://` URI.
+- `path/to/file.pdf`
+  Converts a local PDF to Markdown and imports it as one file document.
+- `path/to/file.docx`
+  Converts a local Word document to Markdown and imports it as one file document.
+- `'docs/**/*.pdf'`
+  Recursively finds local PDFs and converts each one to a file document.
+- `path/to/file.pdf path/to/file.xlsx output.ndjson`
+  Imports multiple local file inputs in deterministic path order.
 
-HTTPS input URIs are supported for unauthenticated remote `.csv`, `.ndjson`, and `.json` sources. URLs without a supported file extension can still be accepted when the response `Content-Type` maps to CSV or NDJSON-oriented JSON input.
+HTTP and HTTPS input URIs are supported for unauthenticated remote `.csv`, `.ndjson`, and `.json` sources. URLs without a supported file extension can still be accepted when the response `Content-Type` maps to CSV or NDJSON-oriented JSON input.
+
+### AnyDoc local documents
+
+Local files with these extensions are converted to GitHub-Flavored Markdown through anydoc before entering the existing file-document pipeline:
+
+`.doc`, `.docx`, `.docm`, `.odt`, `.pdf`, `.ppt`, `.pps`, `.pot`, `.pptx`, `.pptm`, `.ppsx`, `.ppsm`, `.rtf`, `.epub`, `.xls`, `.xlsx`, `.xlsm`, `.xlsb`, `.ods`, and `.odp`.
+
+Converted content is stored in `content.body` by default. Use `--content markdown` to store it in `content.markdown`. Multi-file and glob-resolved local inputs add an `origin` object with URI components (`scheme`, `authority`, `path`, `query`, `fragment`) and `filename`; remote CSV, NDJSON, and Toon inputs preserve the same components from their source URI. Anydoc conversion remains local-only. Scanned or image-only PDFs require OCR outside espipe and are not converted.
 
 ### Supported output forms
 
@@ -163,6 +183,10 @@ The first row must be a header row. Each subsequent row is converted into a JSON
 
 CSV values are emitted as JSON strings. `espipe` does not infer numeric, boolean, or date types from CSV input.
 
+### File-document input
+
+Markdown, text, YAML, and anydoc-converted files are emitted as JSON documents through the existing file-document pipeline. Markdown frontmatter remains available under `content.*`, and converted non-text files expose their generated Markdown under the configured content field. Existing file discovery supports shell-expanded paths, multiple local input positionals, and quoted recursive glob patterns.
+
 ### Bulk actions
 
 `espipe` supports three Elasticsearch bulk actions:
@@ -281,6 +305,18 @@ cat docs.ndjson | espipe - http://localhost:9200/my-index
 espipe users.csv output.ndjson
 ```
 
+### Ingest local PDFs recursively
+
+```bash
+espipe '**/*.pdf' http://localhost:9200/documents
+```
+
+### Ingest multiple local document formats
+
+```bash
+espipe '**/*.pdf' '**/*.docx' '**/*.xlsx' output.ndjson
+```
+
 ### Read and write gzip-compressed files
 
 ```bash

openspec/changes/add-anydoc-input/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-08-12

openspec/changes/add-anydoc-input/design.md

@@ -0,0 +1,90 @@
+## Context
+
+Local file imports already resolve concrete paths and recursive glob patterns in `src/input.rs`. The `Input::FileDocuments` variant reads each path lazily, constructs a JSON object, and sends it through the same `Box<RawValue>` output path used by NDJSON, CSV, and other inputs. Today, recognized text formats have specialized readers and all other file-document paths fall back to UTF-8 text, which rejects PDFs and office containers.
+
+The `anydoc` crate provides a Rust-native conversion API for PDF, Word, PowerPoint, Excel, OpenDocument, RTF, and EPUB inputs. Its Markdown output is the appropriate intermediate representation because the existing file-document implementation already defines the desired content field, Markdown frontmatter behavior, and origin metadata.
+
+## Goals / Non-Goals
+
+**Goals:**
+
+- Convert supported local anydoc formats into the existing file-document JSON shape.
+- Preserve the existing output dispatch and behavior for Markdown, text, YAML, JSON, NDJSON, Toon, CSV, stdin, and HTTPS inputs.
+- Support direct files, shell-expanded file lists, and existing recursive glob patterns without changing discovery semantics.
+- Keep conversion errors path-specific and visible through the existing stderr error path.
+- Cover representative document formats and mixed file collections with tests.
+
+**Non-Goals:**
+
+- OCR for scanned or image-only PDFs.
+- Remote HTTPS anydoc inputs.
+- A new `--extensions` option; multiple extension patterns can be passed as existing input positionals.
+- Extraction of embedded assets or source-specific metadata beyond the Markdown produced by anydoc and the shared `origin` metadata.
+- Content-based conversion of unknown-extension files, which could change the existing unknown UTF-8 text behavior.
+
+## Decisions
+
+### Use the `anydoc` crate directly
+
+Add `anydoc` as a normal dependency and call its Rust API rather than spawning the anydoc CLI. This avoids subprocess lifecycle and temporary-file concerns and keeps conversion inside the existing input processor. The dependency is compatible with the repository's Rust 1.88 baseline.
+
+Alternatives considered:
+
+- **Invoke the anydoc CLI:** rejected because it adds an external executable/runtime dependency and makes error handling and streaming less direct.
+- **Add format-specific parsers to espipe:** rejected because it duplicates the purpose of anydoc and expands the maintenance surface.
+
+### Gate conversion by recognized non-CSV extension
+
+In `read_file_documents`, check `anydoc::Format::from_path(path)` after the existing specialized readers. If it identifies a supported format other than `Format::Csv`, convert the file with `anydoc::to_markdown(path)`. Keep existing readers ahead of this branch so CSV and all current text/document formats retain their behavior.
+
+The extension gate intentionally avoids running content detection on every unknown file. Unknown valid UTF-8 files remain text documents, while recognized anydoc extensions opt into conversion. Mislabeled-format support can be considered separately if needed.
+
+### Reuse the Markdown document builder
+
+Refactor the existing Markdown reader so it has a helper that accepts `(path, markdown_text, content_field, include_origin)`. The current Markdown path reads the source text and calls this helper; the anydoc path converts the source and calls the same helper.
+
+This preserves:
+
+- `content.<field>` placement and `--content` behavior.
+- YAML frontmatter extraction and content-field conflict checks.
+- A single `origin` metadata object for multi-file and glob-resolved local imports, containing URI components (`scheme`, `authority`, `path`, `query`, `fragment`) and `filename`. Remote HTTP and HTTPS CSV, NDJSON, and Toon imports use the same shape from their source URI.
+- One serialized `Box<RawValue>` per output document.
+
+The resulting flow is:
+
+```text
+path/glob → existing path resolution → format-specific input branch
+                                      ├─ Markdown → Markdown document builder
+                                      ├─ anydoc format → anydoc Markdown → same builder
+                                      └─ existing text/JSON/YAML/etc. readers
+              → Box<RawValue> → existing output
+```
+
+### Keep discovery unchanged
+
+No new traversal or extension-selection option is needed. Existing commands such as `espipe '**/*.pdf' output.ndjson` work because unknown local extensions already enter file-document mode, and multiple types can be supplied as separate patterns. A future `--extensions pdf,xls,doc` option should be designed as a discovery/filtering feature rather than coupled to the converter.
+
+### Preserve lazy conversion
+
+Convert files when `read_file_document_line` reaches them, matching the current lazy file-document behavior. This avoids loading an entire collection before output begins and keeps anydoc conversion scoped to the selected file. Conversion failures use the existing input error path and include the source path.
+
+### Preserve anydoc error wording
+
+Use anydoc's conversion error wording as-is, adding only the source path context needed to identify the failing input. This avoids introducing a second error taxonomy while preserving the underlying reason for failures. Error normalization can be added later if callers need a stable espipe-specific code or prefix.
+
+## Risks / Trade-offs
+
+- **Dependency and binary growth** → Accept the direct dependency for broad format coverage; document the build impact and verify the normal test/build path.
+- **Full-file memory use** → Keep the existing one-file-at-a-time document model; anydoc already returns a Markdown `String`, and the resulting raw document is released before the next file is read.
+- **Scanned PDFs fail without OCR** → Surface anydoc's unsupported conversion error with the source path; do not claim OCR support.
+- **Failures can occur after earlier documents were sent** → Retain existing lazy input semantics and error reporting; changing ingestion to preflight every file is outside this change.
+- **Generated Markdown may contain frontmatter-like syntax** → Run it through the established Markdown builder for consistent document semantics; add a fixture-based regression test for representative anydoc output.
+- **Unknown-extension detection remains extension-gated** → Preserve current unknown UTF-8 behavior; consider content-based detection only as a separate, explicitly scoped change.
+
+## Migration Plan
+
+No data migration is required. Add the dependency and input branch, then release normally. Existing commands and output document shapes remain compatible. Rollback consists of removing the anydoc dependency, branch, tests, and change artifacts; no stored documents require migration.
+
+## Open Questions
+
+None. The initial regression suite will use one representative PDF and one Office/OpenDocument fixture; coverage of every supported format remains the responsibility of the anydoc crate.

openspec/changes/add-anydoc-input/proposal.md

@@ -0,0 +1,29 @@
+## Why
+
+`espipe` currently treats local file documents as text, so PDFs and office formats cannot be ingested even though the existing glob-based file import pipeline already supports recursive document collections. The Rust `anydoc` library can convert these formats to GitHub-Flavored Markdown, allowing non-text ingestion without changing the downstream document or output model.
+
+## What Changes
+
+- Add `anydoc` as a local file input preprocessor for supported non-text formats, including PDF, Word, PowerPoint, Excel, OpenDocument, RTF, and EPUB files.
+- Convert each supported file to Markdown before constructing the existing file document, preserving `content.<field>`, Markdown handling, and the conditional `origin` metadata.
+- Allow existing concrete file lists and recursive glob patterns such as `**/*.pdf` to ingest converted documents.
+- Preserve existing CSV, JSON, NDJSON, Toon, Markdown, YAML, text, stdin, and HTTP/HTTPS behavior while attaching source URI metadata to remote streaming inputs.
+- Report conversion failures with the source path and keep remote non-streaming anydoc inputs out of scope.
+- Do not add `--extensions` in this change; multiple extension patterns can already be supplied as separate local inputs. A future extension-filter option can build on the existing discovery layer.
+
+## Capabilities
+
+### New Capabilities
+
+- `anydoc-input`: Convert supported local non-text documents to Markdown within the input pipeline.
+
+### Modified Capabilities
+
+- `file-document-import`: Supported anydoc formats are imported as converted Markdown documents instead of being rejected as binary files.
+
+## Impact
+
+- Affects `src/input.rs` only in the ingestion path and adds the `anydoc` crate dependency.
+- Adds fixtures and unit/integration coverage for representative converted formats, mixed globs, metadata preservation, and conversion errors.
+- Increases dependency graph, compile time, and binary size because anydoc includes parsers for office containers and PDFs.
+- Image-only/scanned PDFs remain unsupported because conversion does not provide OCR.

openspec/changes/add-anydoc-input/specs/anydoc-input/spec.md

@@ -0,0 +1,91 @@
+## ADDED Requirements
+
+### Requirement: Supported local non-text files are converted through anydoc
+
+The system SHALL use the Rust `anydoc` processor for local regular files with these extensions: `.doc`, `.docx`, `.docm`, `.odt`, `.pdf`, `.ppt`, `.pps`, `.pot`, `.pptx`, `.pptm`, `.ppsx`, `.ppsm`, `.rtf`, `.epub`, `.xls`, `.xlsx`, `.xlsm`, `.xlsb`, `.ods`, and `.odp`. The processor SHALL convert each file to GitHub-Flavored Markdown before file-document construction.
+
+#### Scenario: A PDF file is imported
+
+- **WHEN** the user runs `espipe` with a local text-based `.pdf` input
+- **THEN** the system converts the PDF to Markdown through anydoc
+- **AND** emits one JSON document containing the converted Markdown in the configured content field
+
+#### Scenario: An office document is imported
+
+- **WHEN** the user runs `espipe` with a local supported Word, PowerPoint, Excel, or OpenDocument input
+- **THEN** the system converts the file to Markdown through anydoc
+- **AND** emits one JSON document for the source file
+
+#### Scenario: A supported file is imported from a mixed local collection
+
+- **WHEN** file-document input resolves supported anydoc files together with Markdown or text files
+- **THEN** each anydoc file is converted at its position in the deterministic file order
+- **AND** existing Markdown and text files continue through their existing readers
+
+### Requirement: Anydoc conversion preserves the existing file-document shape
+
+The system SHALL construct converted documents using the existing file-document semantics after Markdown conversion. It SHALL store converted Markdown under `content.<field_name>`, apply the configured `--content` value, and add one `origin` metadata object for multi-file or glob-resolved local imports. The object SHALL contain `scheme`, `authority`, `path`, `query`, `fragment`, and `filename` fields. It SHALL NOT emit the legacy `file.path` or `file.name` metadata.
+
+#### Scenario: Default content field is used for converted Markdown
+
+- **WHEN** a supported anydoc file is imported without `--content`
+- **THEN** the converted Markdown is stored in `content.body`
+- **AND** the document does not expose the original binary bytes as a field
+
+#### Scenario: Custom content field is used for converted Markdown
+
+- **WHEN** a supported anydoc file is imported with `--content markdown`
+- **THEN** the converted Markdown is stored in `content.markdown`
+- **AND** the system does not add `content.body` solely because the source was converted
+
+#### Scenario: Converted files participate in multi-file origin metadata
+
+- **WHEN** anydoc files are imported together with another file-document input
+- **THEN** each converted document includes an `origin` object
+- **AND** its URI components identify the original local file
+
+#### Scenario: Glob-resolved converted files include complete origin metadata
+
+- **WHEN** anydoc files are imported through a local glob pattern
+- **THEN** each converted document includes `origin.scheme` equal to `file`
+- **AND** `origin.path` identifies the containing directory
+- **AND** `origin.filename` identifies the original local file
+- **AND** absent `origin.authority`, `origin.query`, and `origin.fragment` fields are omitted
+
+#### Scenario: Remote inputs preserve URI origin metadata
+
+- **WHEN** an HTTP or HTTPS CSV, NDJSON, or Toon input is imported
+- **THEN** each emitted document includes `origin.scheme`, `origin.authority`, `origin.path`, `origin.query`, `origin.fragment`, and `origin.filename`
+- **AND** the values reflect the source URI rather than the temporary download file
+
+### Requirement: Existing local discovery mechanisms support anydoc files
+
+The system SHALL process supported anydoc files supplied as direct local paths, shell-expanded file lists, or existing local glob patterns. It SHALL not require a separate subprocess or remote service for local conversion.
+
+#### Scenario: A quoted recursive PDF glob is imported
+
+- **WHEN** the user runs `espipe '**/*.pdf' output.ndjson`
+- **THEN** the existing glob resolver finds matching regular files
+- **AND** each matching PDF is converted and emitted as one file document
+
+#### Scenario: Multiple extension patterns are supplied
+
+- **WHEN** the user supplies separate local input patterns such as `**/*.pdf`, `**/*.xls`, and `**/*.doc`
+- **THEN** the system combines and de-duplicates the resolved paths using existing file discovery rules
+- **AND** converts each supported path according to its extension
+
+### Requirement: Anydoc conversion failures identify the source file
+
+The system SHALL report anydoc conversion failures through the existing file-input error path, including the source path and the underlying conversion reason when available. It SHALL not emit a synthetic document for a file that anydoc cannot convert.
+
+#### Scenario: An unsupported document is encountered
+
+- **WHEN** anydoc reports that a supported-extension file is encrypted, malformed, unsupported, or exceeds a conversion limit
+- **THEN** ingestion fails with a diagnostic identifying the source path
+- **AND** the diagnostic is written to stderr
+
+#### Scenario: An image-only PDF is encountered
+
+- **WHEN** anydoc cannot extract meaningful text from a scanned or image-only PDF
+- **THEN** ingestion fails with a path-specific unsupported-conversion diagnostic
+- **AND** the system does not claim to perform OCR