Add streaming JSON split input

[VimCommando]

Summary

• add --split <JSON_POINTER> for streaming root or nested JSON arrays and objects
• process bounded batches in parallel without preserving source order
• preserve object keys as id and arbitrary-precision JSON numbers
• document behavior and add unit, file-output, and Elasticsearch coverage

Testing

• cargo fmt --all -- --check
• cargo test (146 passed, 5 ignored)
• cargo clippy (0 errors; 13 existing warnings)
• openspec validate add-streaming-json-split --strict
• openspec validate json-split-input --type spec --strict

Closes #14

[Copilot]

Pull request overview

Adds opt-in --split support for streaming root or nested JSON arrays/objects through existing output pipelines.

Changes:

• Adds JSON Pointer navigation and bounded parallel batch processing.
• Preserves map keys as string id fields and arbitrary-precision numbers.
• Adds CLI, Elasticsearch, fixture, documentation, and OpenSpec coverage.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
tests/fixtures/split_root_map.json	Updated as part of this pull request.
tests/fixtures/split_nested_array.json	Updated as part of this pull request.
tests/file_output.rs	Updated as part of this pull request.
tests/elasticsearch.rs	Updated as part of this pull request.
src/main.rs	Updated as part of this pull request.
src/json_split.rs	Updated as part of this pull request.
src/input.rs	Updated as part of this pull request.
README.md	Updated as part of this pull request.
openspec/specs/json-split-input/spec.md	Updated as part of this pull request.
openspec/changes/add-streaming-json-split/tasks.md	Updated as part of this pull request.
openspec/changes/add-streaming-json-split/specs/json-split-input/spec.md	Updated as part of this pull request.
openspec/changes/add-streaming-json-split/proposal.md	Updated as part of this pull request.
openspec/changes/add-streaming-json-split/design.md	Updated as part of this pull request.
openspec/changes/add-streaming-json-split/.openspec.yaml	Updated as part of this pull request.
examples/steam-games/readme.md	Updated as part of this pull request.
CHANGELOG.md	Updated as part of this pull request.
Cargo.toml	Updated as part of this pull request.

Suppressed comments (2)

src/json_split.rs:439

• A malformed array element fails in next_element before PendingDocument::Array is created, so the reported error has the source/path and parser location but no zero-based element index. The split contract promises child-context diagnostics; wrap this error with the current index before propagating it.

        while let Some(raw) = sequence.next_element::<Box<RawValue>>()? {

src/json_split.rs:425

• If a selected object property is syntactically malformed, next_value returns directly and the outer context only includes the source and split path; the property key is lost. This violates the child-context error contract and makes large catalogues hard to diagnose (the transform-time shape errors do include the key). Wrap this deserialization error with object property '{key}' while retaining the parse location.

            let raw = map.next_value::<Box<RawValue>>()?;

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

src/json_split.rs:332

• Cancellation is not observed while this navigator consumes unmatched values with IgnoredAny (and the array branch has the same gap). If the output receiver is dropped after a selected batch has been emitted, the parser can continue reading a potentially huge trailing/preceding sibling—or block on stdin—until it reaches EOF, because no worker is sending another result to set cancelled. Propagate consumer disconnect into the parser or use cancellation-aware skipped-value traversal so split ingestion stops promptly after output failure.

            } else {
                map.next_value::<IgnoredAny>()?;

src/json_split.rs:364

• This skipped-array path has the same cancellation gap as the object branch: IgnoredAny is consumed without checking whether the output receiver has disconnected. After a downstream failure, a large remainder of the wrapper/array can still be parsed before the parser notices cancellation, defeating prompt shutdown and potentially blocking on stdin. Use the same cancellation-aware handoff/traversal fix here as for skipped object members.

            } else {
                sequence.next_element::<IgnoredAny>()?.is_some()

[Copilot]

🟢 Approval recommended

No unresolved review issues were identified.

Review details

• Files reviewed: 17/17 changed files
• Comments generated: 0 new
• Review effort level: Lite