feat(connectors): add Meilisearch source connector#3404
Open
countradooku wants to merge 9 commits into
Open
Conversation
Meilisearch write operations are task-based, so the connector uses the official Rust SDK for health checks, index creation, document writes, and task completion while preserving Iggy connector lifecycle semantics. The sink injects a stable iggy_id primary key when payloads omit the configured primary key, keeping repeated deliveries idempotent across stream/topic partitions. Constraint: User requested the official Meilisearch SDK instead of direct REST calls Constraint: Connector runtime tests live under core/integration/tests/connectors, not connector-local tests Rejected: Reuse the Elasticsearch bulk client shape directly | Meilisearch writes complete through asynchronous tasks rather than bulk item responses Rejected: Keep connector-local live tests | inconsistent with existing connector integration layout Confidence: high Scope-risk: moderate Directive: Keep Meilisearch task waiting enabled by default unless delivery semantics are intentionally relaxed Tested: cargo test -p iggy_connector_meilisearch_sink Tested: cargo clippy -p iggy_connector_meilisearch_sink --all-targets -- -D warnings Tested: cargo build -p iggy_connector_meilisearch_sink Tested: cargo test -p integration --test mod --no-run Tested: env CARGO_BIN_EXE_iggy-server=... CARGO_BIN_EXE_iggy-connectors=... cargo test -p integration --test mod -- meilisearch_sink_indexes_json_messages --nocapture Not-tested: Full unfiltered integration test suite
Meilisearch already had a sink connector, so this adds the matching source connector and test fixture structure to support bidirectional connector coverage. The source uses the official Meilisearch SDK for health checks and search polling, preserving the connector runtime's state-based offset persistence pattern. Constraint: Use the existing connector SDK Source lifecycle and persisted ConnectorState format Constraint: Use the official meilisearch-sdk rather than hand-written search HTTP calls Rejected: Keep Meilisearch source fixture code inside sink.rs | source and sink test setup should mirror the Elasticsearch container/source/sink split Confidence: high Scope-risk: moderate Directive: Keep Meilisearch source polling semantics tied to stable search ordering when indexes mutate during runtime Tested: cargo check -p iggy_connector_meilisearch_source Tested: cargo test -p iggy_connector_meilisearch_source Tested: cargo check -p integration Not-tested: Full Docker-backed Meilisearch integration test runtime
countradooku
changed the title
The PR branch needed to absorb upstream connector and workspace updates so GitHub can merge it cleanly. The resolution keeps upstream dependency bumps and connector list entries while preserving the new Meilisearch source connector workspace membership and documentation. Constraint: PR targets apache/iggy master and must be mergeable there Rejected: Rebase the branch | a merge commit preserves the already-published PR branch without rewriting history Confidence: high Scope-risk: moderate Directive: Keep upstream connector additions and Meilisearch source additions together in shared connector listings Tested: cargo check -p iggy_connector_meilisearch_source Tested: cargo test -p iggy_connector_meilisearch_source Tested: cargo check -p integration Tested: git diff --check Not-tested: Full Docker-backed connector integration suite
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #3404 +/- ##
=============================================
- Coverage 74.67% 46.42% -28.25%
Complexity 943 943
=============================================
Files 1228 1228
Lines 120529 105966 -14563
Branches 97266 82732 -14534
=============================================
- Hits 90002 49193 -40809
- Misses 27588 54087 +26499
+ Partials 2939 2686 -253
🚀 New features to boost your workflow:
|
The pre-merge TOML formatter expects long inline dependency feature lists to be expanded. Formatting the Meilisearch SDK dependency keeps the branch aligned with the existing CI taplo check after pulling the latest branch update. Constraint: Only the Meilisearch branch Cargo.toml formatting drift is in scope Confidence: high Scope-risk: narrow Tested: ./scripts/ci/taplo.sh --check Cargo.toml
Cargo refused to run locked checks because the workspace manifest resolution no longer matched Cargo.lock after the latest Meilisearch branch update. Regenerating the lockfile removes the stale duplicate untrusted 0.7.1 resolution and lets --locked checks run without mutating the lockfile. Constraint: Preserve the Meilisearch-only scope and update only the lockfile drift required by Cargo Confidence: high Scope-risk: narrow Tested: cargo check -p iggy_connector_meilisearch_source Tested: cargo check --locked -p iggy_connector_meilisearch_source
The integration harness starts the connectors runtime before the test body indexes Meilisearch documents. Treating a missing Meilisearch index as an empty poll keeps the source connector retryable during startup instead of surfacing a transient configuration error. The source fixture also disables metadata wrapping so the test receives the raw documents it asserts on. Constraint: Keep the fix scoped to Meilisearch source behavior and its test fixture Rejected: Sleeping in the test before starting the runtime | the harness owns runtime startup order Confidence: high Scope-risk: narrow Tested: cargo check --locked -p iggy_connector_meilisearch_source Tested: env CARGO_BIN_EXE_iggy-server=/Users/radudiaconu/Desktop/Code/Rust/iggy-meilisearch/target/debug/iggy-server CARGO_BIN_EXE_iggy-connectors=/Users/radudiaconu/Desktop/Code/Rust/iggy-meilisearch/target/debug/iggy-connectors cargo test -p integration --test mod connectors::meilisearch::meilisearch_source::meilisearch_source_produces_index_documents -- --nocapture
The previous Python BDD job failed before test execution because Docker Hub metadata requests timed out while resolving base images. This empty commit retriggers the PR checks without changing the Meilisearch connector diff. Constraint: No code changes are needed for an external Docker registry timeout Rejected: Modify BDD or Docker configuration | failure happened before tests and outside the branch scope Confidence: high Scope-risk: narrow Tested: Not run; empty commit for CI rerun only
Contributor
Author
|
/ready |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
meilisearch-sdkcontainer.rs,sink.rs, andsource.rs, then add source runtime coverageValidation
cargo check -p iggy_connector_meilisearch_sourcecargo test -p iggy_connector_meilisearch_sourcecargo check -p integrationcargo clippyhook passedNotes