Skip to content

feat(connectors): store source state on an HTTP state server - #3940

Open
mmodzelewski wants to merge 1 commit into
masterfrom
http-connectors-state
Open

feat(connectors): store source state on an HTTP state server#3940
mmodzelewski wants to merge 1 commit into
masterfrom
http-connectors-state

Conversation

@mmodzelewski

Copy link
Copy Markdown
Member

Source connectors checkpoint through the file state backend, which
writes one file per source to local disk. That ties a source's
cursor to the node it happens to run on, so replacing the node or
failing over to another runtime instance rewinds the source.

Add a second backend selected by state.storage. The HTTP provider
stores each source at {url}/source_{key} on any HTTP-speaking
store, so cursors outlive the process. Concurrency is optimistic:
reads remember the returned ETag, writes are conditional
(If-Match, or If-None-Match: * before the first write), and each
logical save carries a stable Idempotency-Key so a server that
committed a write but lost the response replays that outcome
instead of failing the retry with a spurious 412.

State failures are now classified in the SDK. 5xx, timeouts and
connect failures retry with backoff and end as TransientState, so
the batch is Nacked and the plugin re-polls. Version conflicts and
revoked authorization are PermanentState and latch the provider,
failing later saves without touching the network: another writer
took over, and retrying would only mask the original error. A
classified failure while loading an enabled source's state aborts
startup instead of minting a failed plugin, because treating it as
"no state" would silently rewind the source.

The file backend stays the default and its unclassified load
errors keep the per-connector failure path, so existing configs
parse and behave as before.

Source connectors checkpoint through the file state backend, which
writes one file per source to local disk. That ties a source's
cursor to the node it happens to run on, so replacing the node or
failing over to another runtime instance rewinds the source.

Add a second backend selected by state.storage. The HTTP provider
stores each source at {url}/source_{key} on any HTTP-speaking
store, so cursors outlive the process. Concurrency is optimistic:
reads remember the returned ETag, writes are conditional
(If-Match, or If-None-Match: * before the first write), and each
logical save carries a stable Idempotency-Key so a server that
committed a write but lost the response replays that outcome
instead of failing the retry with a spurious 412.

State failures are now classified in the SDK. 5xx, timeouts and
connect failures retry with backoff and end as TransientState, so
the batch is Nacked and the plugin re-polls. Version conflicts and
revoked authorization are PermanentState and latch the provider,
failing later saves without touching the network: another writer
took over, and retrying would only mask the original error. A
classified failure while loading an enabled source's state aborts
startup instead of minting a failed plugin, because treating it as
"no state" would silently rewind the source.

The file backend stays the default and its unclassified load
errors keep the per-connector failure path, so existing configs
parse and behave as before.
@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Aug 20, 2026
@codecov

codecov Bot commented Aug 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.04236% with 119 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.91%. Comparing base (c0c7493) to head (0274642).

Files with missing lines Patch % Lines
core/connectors/runtime/src/state/http.rs 90.76% 53 Missing and 21 partials ⚠️
core/connectors/runtime/src/source.rs 63.88% 12 Missing and 1 partial ⚠️
core/connectors/runtime/src/configs/runtime.rs 89.77% 9 Missing ⚠️
core/connectors/runtime/src/manager/source.rs 33.33% 6 Missing and 2 partials ⚠️
core/connectors/runtime/src/state.rs 89.04% 5 Missing and 3 partials ⚠️
core/connectors/runtime/src/stream.rs 92.30% 0 Missing and 5 partials ⚠️
core/connectors/runtime/src/error.rs 0.00% 1 Missing ⚠️
core/connectors/runtime/src/main.rs 66.66% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             master    #3940       +/-   ##
=============================================
- Coverage     83.87%   68.91%   -14.96%     
  Complexity     1358     1358               
=============================================
  Files          1212     1213        +1     
  Lines        166843   141845    -24998     
  Branches     134306   109430    -24876     
=============================================
- Hits         139937    97757    -42180     
- Misses        23266    40254    +16988     
- Partials       3640     3834      +194     
Components Coverage Δ
Rust Core 65.20% <89.04%> (-19.45%) ⬇️
Java SDK 66.67% <ø> (ø)
C# SDK 75.00% <ø> (-1.55%) ⬇️
Python SDK 90.13% <ø> (ø)
PHP SDK 84.48% <ø> (ø)
Node SDK 95.87% <ø> (+0.02%) ⬆️
Go SDK 68.32% <ø> (ø)
Files with missing lines Coverage Δ
core/connectors/runtime/src/api/auth.rs 100.00% <100.00%> (ø)
core/connectors/runtime/src/api/source.rs 74.60% <ø> (-0.14%) ⬇️
core/connectors/runtime/src/context.rs 81.64% <100.00%> (-10.76%) ⬇️
core/connectors/sdk/src/lib.rs 57.14% <ø> (ø)
core/connectors/runtime/src/error.rs 26.66% <0.00%> (-1.91%) ⬇️
core/connectors/runtime/src/main.rs 85.76% <66.66%> (-0.06%) ⬇️
core/connectors/runtime/src/stream.rs 90.09% <92.30%> (+14.75%) ⬆️
core/connectors/runtime/src/manager/source.rs 91.97% <33.33%> (-1.61%) ⬇️
core/connectors/runtime/src/state.rs 86.44% <89.04%> (+0.73%) ⬆️
core/connectors/runtime/src/configs/runtime.rs 77.02% <89.77%> (+12.15%) ⬆️
... and 2 more

... and 381 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-review PR is waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant